Bug about Dropout CUDA Kernel

[MARD1NO]

In Dropout.cu (https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/Dropout.cu#L35)

In 56 lines， you set bool gridxvec_loop_state = 0。

When the gridxvec_loop_state is 0, you generate 4 rand numbers。and in other case，you want to use the last 2 rand values：

    if ((VEC == 4) || (gridxvec_loop_state == 0)) {
      rand = curand_uniform4(&state);
    } else {
      // sets up the last two values we generated last iteration to be used this iteration.
      rand.x = rand.z;
      rand.y = rand.w;
      gridxvec_loop_state ^= 1;
    }

Since you set loop_state as 0, it enter the if branch and generate 4 random numbers，you didn't change the loop_state so you never enter the else branch and cannot use the last 2 rand values.

In my opinion, it should be like this:

    if ((VEC == 4) || (gridxvec_loop_state == 0)) {
      rand = curand_uniform4(&state);
      gridxvec_loop_state ^= 1;
    } else {
      // sets up the last two values we generated last iteration to be used this iteration.
      rand.x = rand.z;
      rand.y = rand.w;
      gridxvec_loop_state ^= 1;
    }

cc @albanD @mruberry @jbschlosser @walterddr @ngimel

[eqy]

CC'ing @mcarilli who might be interested

[eqy]

CC @ngimel

[ngimel]

We'll accept a fix for this, but Dropout with vec_size=2 has probably never been used (and even if it is, the bug is not catastrophic, we just don't advance random state enough, but that's fine).

[mcarilli]

I think you're right about the bug and the proposed fix, good catch.