From 9abeaa3c2bb2d7d2a18919ceccf666f492476efc Mon Sep 17 00:00:00 2001
From: Tony-Y <11532812+Tony-Y@users.noreply.github.com>
Date: Mon, 22 Jan 2024 21:05:58 +0900
Subject: [PATCH] Update README.md

Add how to start the LR schedule after the end of the warmup
---
 README.md | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/README.md b/README.md
index 4f28552..5cd589f 100644
--- a/README.md
+++ b/README.md
@@ -68,6 +68,21 @@ for epoch in range(1,num_epochs+1):
             lr_scheduler2.step()
 ```
 
+If you want to start the learning rate schedule after the end of the linear warmup, delay it by the warmup period:
+```python
+warmup_period = 2000
+num_steps = len(dataloader) * num_epochs - warmup_period
+lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_steps)
+warmup_scheduler = warmup.LinearWarmup(optimizer, warmup_period)
+for epoch in range(1,num_epochs+1):
+    for batch in dataloader:
+        ...
+        optimizer.step()
+        with warmup_scheduler.dampening():
+            if warmup_scheduler.last_step + 1 >= warmup_period:
+                lr_scheduler.step()
+```
+
 #### Approach 2
 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Tony-Y/colab-notebooks/blob/master/PyTorch_Warmup_Approach2_chaining.ipynb)