discuss whether it worked or didn't work

[lucidrains]

share positive or negative results here

please leave the batch size and other hyperparameters

please also leave the learning rate scheduler, if one was used

[lucidrains]

https://crumbly.medium.com/small-scale-home-evaluation-of-googles-new-optimizer-lion-77115ba8a1a4

[lucidrains]

#4 (comment)

[anhnami]

Didn't work for speech enhancement tasks when I ran a sweep with Random search and ASHA: optimizer=["Adam", "Adan", "Lion"] and lr is log uniform from 1e-5 to 1e-2. Note that Lion's trials were killed early in the experiments by ASHA. Here is the top performed runs:

I will do an extra exp without ASHA.

[lucidrains]

@anhnami please do, thank you

[lucidrains]

try with higher batch sizes too, maybe at least 16 or 32. seems like it is batch size sensitive technique

[xiangning-chen]

Thanks for testing the Lion optimizer! We observe that at the very beginning, Lion may perform worse than Adam but it will catch up soon, there is another example: https://crumbly.medium.com/small-scale-home-evaluation-of-googles-new-optimizer-lion-77115ba8a1a4.

[anhnami]

Multiple runs with different seeds on a real validation set, hp_metric is predicted MOS, Lion didn't perform as good as Adam for this dense prediction task.

Similar results are also observed for other metrics:

I can increase the batch size, but I would need to reduce the sequence length to fit into a 3090; at the end of the day, the model will see a similar amount of data.

[lucidrains]

@anhnami thank you!

[lucidrains]

https://twitter.com/alexjc/status/1626586985975660546

[lucidrains]

lucidrains/denoising-diffusion-pytorch#170 (comment)

[lucidrains]

this doesn't look great https://github.com/sinpcw/showcase-optimizer

[crazydonkey200]

Thanks for sharing the comparisons! I took a look at this repo and found that it applied weight decay to AdamW but not Lion, so I just added a pull request to fix that for fair comparison.

[lucidrains]

oh good catch! hope they update the results

personally, i think it is shaping up to be an adamw replacement for sufficiently large batch size training, based on what i'm hearing! congrats!

[lucidrains]

negative result from RL https://twitter.com/kaixhin/status/1626772629796564992

[lucidrains]

slightly worse than Adam https://twitter.com/kyo_takano/status/1627147339143200768

[lucidrains]

better than Adam at extreme batch sizes (16k) at open clip mlfoundations/open_clip#432

[mlw214]

I'm currently investigating Lion for an object detection problem I'm working on. Unfortunately, I'm running into a lot of problems finding a good learning rate that works throughout the whole run. Seems for whatever reason it's prone to NaNing with our model/dataset when scaled up to what we used for training production models. Currently in the process of confirming if some other changes are the cause. I'll post updates here.

For reference, here's the model and a brief description of the dataset (which is proprietary):
Backbone: ConvNeXt-XL
FPN: Heavily modified BiFPN
Head: Dynamic RCNN
Current optimizer: MADGRAD
Dataset: can't say too much, but a significant portion of the objects are quite small, and for one of the classes there are a lot of confounders present. It's also fairly large.

[xiangning-chen]

May I know what kind of augmentations you are using? I tested with randaug and mixup, and Lion works pretty good.
Another observation is that Lion usually leads to larger weight norm, not sure whether this affects the fine-tuning process.
I can try object detection task on my end as well.

[mlw214]

We use Trivial Augment (re-implemented using Albumentations for object detection support) with horizontal flipping, along with Mosaic and Mixup augmentations (both with p=0.5 of being applied to a given image). I'll explore in more depth some of the model watching tools Weights and Biases provides. Currently, we're tracking gradients, but not weights. I'll expand it to track weights as well to try to see if any difference between AdamW and Lion pop up.

[mlw214]

Just kicked off the following experiment:

• ConvNeXt-XL backbone
• FasterRCNN detection head
• HFlip augmentation only

If the above doesn't work I'll start experimenting with the backbone and FPN.

[mlw214]

Update: the above experiment failed. I then tried another experiment based on the above but also removed all empty images from training. That greatly helped with stability (got my longest run yet), but towards the end it still threw a NaN. Gonna swap out our custom BiFPN with the original FPN and see what happens.

[mlw214]

Alright, wasn't able to get it to work with a ConvNeXt backbone at all. However, switching to a ResNet 50 did work. I still plan to compare it against AdamW and MADGRAD, but it's a low priority at the moment since it didn't work with our production architecture, which is sad.

[lucidrains]

positive result from someone really prominent in the text-to-image field. large batch sizes (1024), learning rate / 10, weight decay kept the same

[Fred-Erik]

Tried Lion with a proprietary OCR model, roughly similar to PP-OCRv3 (https://arxiv.org/abs/2206.03001). Compared to AdamW I got much slower convergence to a lower accuracy with learning rate / 3, weight decay * 3. With the same learning rate and weight decay I get NaN results. Batch size 2048.

[lucidrains]

@xiangning-chen once (if) you can resolve some of these issues, it would be nice for you to update the paper with some of these details (say, learning rate cooldown), to increase the chances of success for those trying lion

[Fred-Erik]

Did some more experiments. wd=9.0 converges much slower, so weight decay * 30 seems to be too much. But also dividing the ending learning rate of the cosine learning rate schedule does work! So with lr / 10, wd * 10, end lr / 10, Lion is better than best settings for AdamW (green is Lion):

Thank you for your help!

[lucidrains]

@Fred-Erik thank you!

[xiangning-chen]

@Fred-Erik Thank you so much!

[Fred-Erik]

Actually just did a new observation: post-training quantization consistently does not work with a Lion-optimized network, where it does work with an AdamW-trained network (n=4). I tried dynamic and static quantization (using ONNXRuntime: https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html) and both have the same issue of nonsensical output. An analysis of the difference in weights and activations per layer reveals there is no one specific layer to blame, but lots of layers have a much greater difference between the float32 output and int8 when trained with Lion than when trained with AdamW. So for a version of my model that needs quantization, I'm reverting back to AdamW.

[lucidrains]

negative result for LM https://twitter.com/VHellendoorn/status/1630737104975085568

edit: now positive result! https://twitter.com/VHellendoorn/status/1631349009473478656?s=20

[xiangning-chen]

https://twitter.com/VHellendoorn/status/1631097679693463554?s=20

[lucidrains]

https://twitter.com/VHellendoorn/status/1631097679693463554?s=20

! 🤞

[xiangning-chen]

positive now: https://twitter.com/VHellendoorn/status/1631349009473478656?s=20

[lucidrains]

@xiangning-chen amazing! thank you! even if Lion is only useful for language model training, that is a huge step forward

[lucidrains]

i place a lot of weight on Vincent's results, things are looking upwards!

[lucidrains]

positive result, 3x faster for vision transformer fine tuning https://twitter.com/Haoxiang__Wang/status/1631355469439590412?s=20

[lucidrains]

yet another positive result for training LLM from a really good researcher, however he also told me fine tuning was not as good (albeit the based model was trained with Adam, not sure if that makes any difference)

[xiangning-chen]

Thanks for the update! I was wondering, when fine-tuning, do they load the Adam optimizer state from pre-training?
Previously we find that this is helpful. However, as the LLM is pre-trained by Adam, Lion is unable to utilize the optimizer state. Therefore, to ensure a fair comparison, we disable this feature for both optimizers.

[lucidrains]

@xiangning-chen oh yes, that's a good point, i'll ask him about that next time we chat

[anhnami]

Saw this, could be relevant to Lion: https://openreview.net/pdf?id=a65YK0cqH8g

[Kingsleyandher]

Hello, I used megatron to train GPT-2 with 16B parameter number, but when the learning rate increased to 3e-6, the gradient overflow led to the failure of convergence. May I ask why?

My hyperparameter：
Adamw: lr=1.5e-4 min-lr=1e-5 wd=0.01
Lion: lr=1.5e-5 min-lr=1e-6 wd=0.1 beta1=0.95 beta2-0.98

[Kingsleyandher]

beta2=0.98

[Kingsleyandher]

We used the same configuration to achieve results consistent with Adamw as baseline on the 4B parameter model

[Kingsleyandher]

micro batch size: 4
global batic size: 32
lr_decay_style: cosine
warmup steps: 3200

[xiangning-chen]

Hi, thanks for trying out!

Are you referring to when the learning rate increases to 3e-6 in the warmup phase, the model training explodes?
Have you encountered a similar issue with convergence while employing AdamW to train the 16B model as well?