Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bnb/fp output #41

Merged
merged 20 commits into from
Jun 7, 2022
Merged

Bnb/fp output #41

merged 20 commits into from
Jun 7, 2022

Conversation

bnb32
Copy link
Collaborator

@bnb32 bnb32 commented Jun 1, 2022

Basic netcdf output handling for forward pass results. Added some more parallelization for feature loading and normalization. Added new model configs for full production models. Forward pass test for fp -> netcdf -> fp included and working.

Copy link
Member

@grantbuster grantbuster left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, but our priority needs to be the flattened output h5 data because that is our desired final product.

sup3r/pipeline/forward_pass.py Outdated Show resolved Hide resolved
sup3r/pipeline/forward_pass.py Outdated Show resolved Hide resolved
logger.info('Combining forward pass output for '
f'{cls.file_info_logging(file_paths)}')
@classmethod
def write_output(cls, data, features, bottom_left_corner,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At some point we need to save meta data about the GAN used to generate this output.

sup3r/pipeline/forward_pass.py Outdated Show resolved Hide resolved
sup3r/pipeline/forward_pass.py Outdated Show resolved Hide resolved
sup3r/preprocessing/data_handling.py Show resolved Hide resolved
@grantbuster
Copy link
Member

@bnb32, the meta data interpolation still fails. This test should be a bare minimum acceptance criteria (feel free to add to the test suite): test_meta_interp.py.txt

Also, can we make sure the preprocessing / post_processing modules both either have underscores or do not? Thats going to make importing a nightmare.

@bnb32
Copy link
Collaborator Author

bnb32 commented Jun 6, 2022

Lol yeah I realized the underscore annoyance as well. Good test will add that thanks.

@bnb32 bnb32 merged commit a2bb8c3 into main Jun 7, 2022
@bnb32 bnb32 deleted the bnb/fp_output branch June 7, 2022 14:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants