MISH as ActivationFunction

[CalogeroZarbo]

Hi @lucidrains
I was wondering if it could make sense to you if I create a pull request where the user can choose between GLUE or MISH as the activation function.
The explanation of MISH can be found here:

• https://medium.com/@lessw/meet-mish-new-state-of-the-art-ai-activation-function-the-successor-to-relu-846a6d93471f

The GitHub is here:

• https://github.com/digantamisra98/Mish

And the discussion can be found here:

• https://forums.fast.ai/t/meet-mish-new-activation-function-possible-successor-to-relu/53299/315

Here there is a little benchmark:

If I'm not mistaken there is only one place in reformer_pytorch library where you define GLUE_ in the FeedForward layer, I could add a parameter to the constructor as a flag.

Let me know what would you think about it.

Thank you,
Cal

[lucidrains]

@CalogeroZarbo I really like Mish too! I used GELU mainly to be consistent with BERT, but I added the feature where you can pass in the Module class you'd like to be instantiated as the activation function. https://github.com/lucidrains/reformer-pytorch#customizing-feedforward

[CalogeroZarbo]

@lucidrains I just lost that part! Thank you.
Btw I'm training using RangersLars with DeepSpeed and FP16 Apex optimization on an Encoder-Decoder Reformer architecture and it's working very well.

FYI: https://github.com/mgrankin/over9000

I'll close the issue since it's not needed :)

Cheers!
Cal

[lucidrains]

@CalogeroZarbo very cool, I'm a big fan of Ranger and Lookahead too :)