Noise shape dropout

[rodgzilla]

Reproducing the specific behavior of the classifier dropout of the original OpenAI implementation of the article. The details of the this patch can be found in issue #11.

[thomwolf]

Thanks Grégory, that looks nice! Did you try to benchmark this new version?

[rodgzilla]

I'm running these experiments right now and I'll post the result here and in the issue thread.

[rodgzilla]

Here is the result of the experiments:

Seed 42

device cuda n_gpu 1
Encoding dataset...
Loading weights...                                                              
running epoch 0
Logging                                                                         
1 187 4.358 7.397 91.18 84.49                                                   
running epoch 1
Logging                                                                         
2 374 0.807 8.412 99.20 90.37                                                   
running epoch 2
Logging                                                                         
3 561 0.000 20.528 100.00 90.11                                                 
ROCStories Valid Accuracy: 90.37                                                
ROCStories Test Accuracy:  87.17

Seed 43

device cuda n_gpu 1
Encoding dataset...
Loading weights...                                                              
running epoch 0
Logging                                                                         
1 187 1.390 8.253 96.52 89.30                                                   
running epoch 1
Logging                                                                         
2 374 0.098 13.438 99.73 90.91                                                  
running epoch 2
Logging                                                                         
3 561 0.000 16.577 100.00 91.18                                                 
ROCStories Valid Accuracy: 91.18                                                
ROCStories Test Accuracy:  87.17

Seed 44

device cuda n_gpu 1
Encoding dataset...
Loading weights...                                                              
running epoch 0
Logging                                                                         
1 187 3.236 7.552 91.18 83.69                                                   
running epoch 1
Logging                                                                         
2 374 1.036 12.025 98.66 86.36                                                  
running epoch 2
Logging                                                                         
3 561 0.055 17.220 99.73 86.90                                                  
ROCStories Valid Accuracy: 86.90                                                
ROCStories Test Accuracy:  84.66