Skip to content

Commit

Permalink
[EN] Possible error in lecture 2.2 - Kaiming initialization (#778)
Browse files Browse the repository at this point in the history
From "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification",  Kaiming He et al. 2015.
Equations (10) and (14).
  • Loading branch information
gschiano committed May 4, 2021
1 parent 0760143 commit d967d82
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion docs/en/week02/02-2.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ Essentially, regularization tries to tell the system to minimize the cost functi

### Weight initialisation

The weights need to be initialised at random, however, they shouldn't be too large or too small such that output is roughly of the same variance as that of input. There are various weight initialisation tricks built into PyTorch. One of the tricks that works well for deep models is Kaiming initialisation where the variance of the weights is inversely proportional to square root of number of inputs.
The weights need to be initialised at random, however, they shouldn't be too large or too small such that output is roughly of the same variance as that of input. There are various weight initialisation tricks built into PyTorch. One of the tricks that works well for deep models is Kaiming initialisation where the standard deviation of the weights is inversely proportional to square root of number of inputs.


### Use dropout
Expand Down

0 comments on commit d967d82

Please sign in to comment.