Fix glu activation

[thomasw21]

Test concerning glu was bypassed due to missing "--no-bias-gelu-fusion" which bypasses the whole activation path. Check https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/main/megatron/model/transformer.py#L101

The reason I came up with that, is glu activations divides the last dimension by two. which would not be possible in current MLP mechanism.

In fact https://arxiv.org/abs/2002.05202 changes the hidden size as instead of two matrices, you have three (2 are merged together in the implementation)

cc @jaketae

[stas00]

I will try to read the paper soon, but until then I defer to Jake on the proposed functionality change.

I made one proposal on the code changes, the rest looks good.

Thank you, Thomas!

[jaketae]

Thank you for fixing this! I realize that GLU unit tests only perform checks on individual tensors and their computed values, not the whole training pipeline, so this fix was necessary. Looks good to me!

[stas00]

Hmm, I think we got a new pytorch and apex needs a manual rebuild and making a new image:

ImportError: /usr/local/lib/python3.8/dist-packages/amp_C.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZNK2at6Tensor7optionsEv

but test_training_all_3_glu test now fails with this PR:

stderr: ValueError: if glu-activation is used, please set --no-bias-gelu-fusion

I pushed a fix and re-run the test suite - good to merge now.