Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjusted ActNorm to work as described in the paper #167

Merged
merged 3 commits into from
Oct 4, 2023
Merged

Conversation

LarsKue
Copy link
Collaborator

@LarsKue LarsKue commented Sep 26, 2023

This is a follow-up PR from the discussion sparked in #165.

This PR involves:

  • Initializing the ActNorm Parameters to unit size in all dimensions except the channel dimension
  • Drawing the mean and std over the batch dimension, as well as any dimensions beyond the channel dimension
  • Checking the batch for correct shape before overwriting the parameters
  • Adjusting the tests to reflect these changes

For free with these changes comes:

  • A fix for silent device moves in the initialization method
  • An additional check for an initialization batch that has zero standard deviation in any channel

@LarsKue
Copy link
Collaborator Author

LarsKue commented Sep 26, 2023

Please do not yet merge, there appears to be an error with the log jacobian in another test. EDIT: Resolved.

Copy link
Collaborator

@fdraxler fdraxler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

Copy link
Collaborator

@RussellALA RussellALA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, I think you can merge

@LarsKue LarsKue merged commit 6912465 into master Oct 4, 2023
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants