Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Making Conv2D unbiased by default, and adding Bias2D module #494

Merged
merged 2 commits into from
Feb 26, 2023

Conversation

coreylowman
Copy link
Owner

Resolves #100

Motivation: For most vision based networks these days, convolution layers are followed by batchnorm2d. Best practice is to always set bias=False in pytorch conv layers, since batchnorm2d will effectively nullify the bias with the normalization and also contains a bias of its own.

This PR removes bias from Conv2D, meaning dfdx is moving to you have to explicitly specify that its a biased conv by adding a Bias2D layer following:

type BiasedConv = (Conv2D<3, 5, 3>, Bias2D<5>);

Another change introduced because of this is that biases are now initialized to zero always, instead of the random uniform they were before. This is what tensorflow does with biases, so its still a standard initialization

@coreylowman coreylowman merged commit 8412ba1 into main Feb 26, 2023
@coreylowman coreylowman deleted the unbiased-conv branch February 26, 2023 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add options to not use bias in Linear (and Conv in future)
1 participant