Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LearnableSqueezer #94

Open
wants to merge 18 commits into
base: master
Choose a base branch
from
Open

LearnableSqueezer #94

wants to merge 18 commits into from

Conversation

grizzuti
Copy link
Collaborator

@grizzuti grizzuti commented Oct 6, 2023

Added a "learnable squeezer" layer, typically used in invertible U-nets (see Etmann, et al., 2020, https://arxiv.org/abs/2005.05220).

Some other very minor changes, the most important of which is having removed the type "InvertibleLayer" and changed it to "InvertibleNetwork". Didn't really see a need for having a separate "InvertibleLayer" type.

@codecov
Copy link

codecov bot commented Oct 6, 2023

Codecov Report

Attention: 24 lines in your changes are missing coverage. Please review.

Files Coverage Δ
src/InvertibleNetworks.jl 60.00% <ø> (ø)
src/conditional_layers/conditional_layer_glow.jl 88.63% <ø> (ø)
src/conditional_layers/conditional_layer_hint.jl 99.18% <ø> (ø)
...itional_layers/conditional_layer_residual_block.jl 100.00% <ø> (ø)
src/layers/invertible_layer_actnorm.jl 90.90% <ø> (ø)
src/layers/invertible_layer_basic.jl 93.33% <ø> (ø)
src/layers/invertible_layer_conv1x1.jl 89.47% <100.00%> (ø)
src/layers/invertible_layer_glow.jl 96.34% <ø> (ø)
src/layers/invertible_layer_hyperbolic.jl 86.20% <ø> (ø)
src/layers/invertible_layer_irim.jl 98.11% <ø> (ø)
... and 12 more

📢 Thoughts on this report? Let us know!.

src/layers/invertible_layer_conv1x1.jl Show resolved Hide resolved
src/utils/parameter.jl Outdated Show resolved Hide resolved
src/utils/parameter.jl Outdated Show resolved Hide resolved

# Gradient test (parameters)
T = Float64
C = LearnableSqueezer(k[1:N]...) |> device; C.stencil_pars.data = InvertibleNetworks.CUDA.cu(C.stencil_pars.data)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those explicit calls to CUDA will break ci. Need to use device here even if not as optimal

Copy link
Collaborator Author

@grizzuti grizzuti Oct 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One issue that I have with |> device is the automatic conversion to Float32; here I was using Float64 to avoid numerical cancellation spoiling the gradient test. Anyway, I will rewrite the test so that we are more consistent with the rest of the tests.

Related to this but perhaps for later, I think it might be a good idea to make every InvertibleNetwork subtype parametric. It might make these conversions and include half-precision nets more easily. Unless there are more important reasons not to do that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related to this but perhaps for later, I think it might be a good idea to make every InvertibleNetwork subtype parametric.

Yes completely agree!

@rafaelorozco
Copy link
Collaborator

@grizzuti in my experience sometimes the random seed for the gradient test can be finicky so I have set the tests to rerun a few times before calling it a failure. I think this particularily relevant if the test fails on a single Julia version. I will try this on this branch and if that 1.6 version passes we should merge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants