Cifar100 Finetune NAN

[gaow0007]

When I reproduce cifar100 finetuning result on ViT. I notice that you comment this line.

I comment / uncomment this line. Both lead to NAN loss within 1 epoch.

May I ask you to provide your experimental log of cifar100 finetuning.

[chaoyanghe]

When I reproduce cifar100 finetuning result on ViT. I notice that you comment this line.

I comment / uncomment this line. Both lead to NAN loss within 1 epoch.

May I ask you to provide your experimental log of cifar100 finetuning.

Sure. I found a log here:
cifar100-pipetransformer.txt

[chaoyanghe]

 ____   _               _____                            __                                      _ 
|  _ \ (_) _ __    ___ |_   _| _ __   __ _  _ __   ___  / _|  ___   _ __  _ __ ___    ___  _ __ | |
| |_) || || '_ \  / _ \  | |  | '__| / _` || '_ \ / __|| |_  / _ \ | '__|| '_ ` _ \  / _ \| '__|| |
|  __/ | || |_) ||  __/  | |  | |   | (_| || | | |\__ \|  _|| (_) || |   | | | | | ||  __/| |   |_|
|_|    |_|| .__/  \___|  |_|  |_|    \__,_||_| |_||___/|_|   \___/ |_|   |_| |_| |_| \___||_|   (_)
          |_|                                                                                      

90550 2021-02-03,01:58:52.358 - {auto_dp.py (322)} - _build_broad_cast_message(): 
################################ Congratulations! To train faster, PipeTransformer has automatically transformed to:
################################ Epoch: 2 
################################ Number of frozen layers: 7 
################################ Pipe length: 4/8 
################################ Newly added ranks: [1, 9] 

[chaoyanghe]

When transformation happens, there would be log like this one. You can check around about the loss

[gaow0007]

Thanks for your reply.