Making Conv2D unbiased by default, and adding Bias2D module #494
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #100
Motivation: For most vision based networks these days, convolution layers are followed by batchnorm2d. Best practice is to always set
bias=False
in pytorch conv layers, since batchnorm2d will effectively nullify the bias with the normalization and also contains a bias of its own.This PR removes bias from Conv2D, meaning dfdx is moving to you have to explicitly specify that its a biased conv by adding a
Bias2D
layer following:Another change introduced because of this is that biases are now initialized to zero always, instead of the random uniform they were before. This is what tensorflow does with biases, so its still a standard initialization