Does using Dropout layers, even if the probability is 0, have a performance penalty?

[epwalsh]

❓ The question

Ideally there should be no performance penalty for dropout layers when the dropout probability is 0. But if there is, we should bypass dropout when p=0.0.

I'm currently testing this here: https://wandb.ai/ai2-llm/dropout-benchmarks

There will be 4 runs:

1. 1.2b-bf16-no-dropout: uses a patched branch that bypasses all calls to dropout (except inside of the scaled_dot_product_attention function which we have no control over). This model is compiled using the default settings.
2. 1.2b-bf16-zero-dropout: uses the usual implementation without any code changes were we still call the Dropout modules even though the dropout probability is set to 0. This model is compiled using the default settings.
3. 1.2b-bf16-no-compile-no-dropout: same as 1.2b-bf16-no-dropout except this model is NOT compiled.
4. 1.2b-bf16-no-compile-zero-dropout: same as 1.2b-bf16-zero-dropout except this model is NOT compiled.

[epwalsh]

Results

While there is some noise, there do not appear to be any significant differences between bypassing dropout and leaving it in.