AssertionError: hidden layer never emitted an output with multi-gpu training

[reactivetype]

I tried your library with a WideResnet40-2 model and used layer_index=-2.

The lightning example works fine for single-gpu but i got the error with multiple GPUs.

[lucidrains]

@reactivetype Hi Rindra! It works on my two GPUs for some reason, but I introduced a change here fa1e338 to see if I could make a stab at the problem without being able to reproduce it. Could you let me know if that helped?

[lucidrains]

@reactivetype is this multiple GPUs on one machine or distributed?

[reactivetype]

@lucidrains on one machine. I used 'dp' mode in lightning. i upgrade to 0.1.4 but still get the issue

[lucidrains]

@reactivetype that's weird, are you just running the script I supplied in the examples?

[lucidrains]

can you try using ddp?

[QinbinLi]

Hi @lucidrains ,

I got the same error when using nn.DataParallel(learner) running with 2 GPUs.

[ranliu98]

Hi @lucidrains ,
thanks for the really nice implementation. I got the same issue using 2 GPUs with "dp".