feat: add dropout in MASTER #349
Conversation
charlesmindee
added
type: enhancement
Codecov Report
@@ Coverage Diff @@
## main #349 +/- ##
==========================================
+ Coverage 95.40% 96.06% +0.65%
==========================================
Files 83 83
Lines 3396 3404 +8
==========================================
+ Hits 3240 3270 +30
+ Misses 156 134 -22
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
| @@ -239,6 +255,8 @@ def call( | |||
| x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32)) | |||
| x += self.pos_encoding[:, :seq_len, :] | |||
|
|
|||
| x = self.dropout(x, **kwargs) | |||
There was a problem hiding this comment.
Mmmh, we already have dropout in the DecoderLayer, are you positive we're supposed to put one here?
There was a problem hiding this comment.
https://www.tensorflow.org/text/tutorials/transformer#decoder Here they put another one here, but we can remove it
There was a problem hiding this comment.
In pytorch it is also used here after the postional_embedding: https://pytorch.org/tutorials/beginner/transformer_tutorial.html#define-the-model
This PR adds dropout in both tf & pytorch MASTER models
Any feedback is welcome!