-
-
Notifications
You must be signed in to change notification settings - Fork 121
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BatchNorm and Dropout #19
Comments
Yeah, that's a great idea. |
BatchNorm is going to be tricky because of its "statistics update step". The current thinking behind this with Zygote is to do something along the lines of https://gist.github.com/staticfloat/a509b1e1cb1fb556028779722c2531e6 |
Now that Flux's normalization interface has been re-worked and GPU batchnorm moved from CUDA.jl -> NNlibCUDA, perhaps we should revisit this. The only reason https://github.com/FluxML/Flux.jl/tree/master/src/cuda exists at all now is to accommodate a non-standard implementation of batchnorm, so getting rid of that would be great. |
I was looking into porting the functional form of normalization layers here, but I'm not sure how to handle the |
The concern has been raised earlier and is fixed by FluxML/Flux.jl#1509 |
I don't think there's any reason these |
Does it make sense to put functional forms of BatchNorm and Dropout into NNlib so that other packages could simply import them from here?
The text was updated successfully, but these errors were encountered: