Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change default export format to compressed hdf5 #2812

Merged
merged 1 commit into from Feb 20, 2024

Conversation

k-dominik
Copy link
Contributor

@k-dominik k-dominik commented Feb 15, 2024

most ilastik outputs compress very well - compressed hdf5 should be made the default format.
Furthermore the default format can now be customized in the config.

accompanying volumina request to change the default order: ilastik/volumina#313

CC: @oanegros (#2796 (comment))

most ilastik outputs compress very well - compressed hdf5 should be made
the default format.
Furthermore the default format can now be customized in the config.
Copy link

codecov bot commented Feb 15, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (e813293) 55.75% compared to head (c3e9dfa) 55.77%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2812      +/-   ##
==========================================
+ Coverage   55.75%   55.77%   +0.02%     
==========================================
  Files         530      530              
  Lines       62173    62173              
  Branches     8498     8498              
==========================================
+ Hits        34663    34679      +16     
+ Misses      25764    25751      -13     
+ Partials     1746     1743       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@k-dominik k-dominik marked this pull request as ready for review February 16, 2024 08:45
@k-dominik
Copy link
Contributor Author

this change has @akreshuk's blessing (pending me checking whether our compressed data would open in MoBiE and our plugin)

Copy link
Contributor

@btbest btbest left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree it's a sensible change, and looks fine on my system 👍 25%-ish file size reduction even on a tiny test image.

@k-dominik
Copy link
Contributor Author

The size reduction can be quite pronounced, depending on the data. My ballpark was usually that predictions will be half the size of their uncompressed counterparts.

It might be worth mentioning that this comes at a cost, too. I checked loading a random image into RAM completely, and the compressed version would take roughly 25% longer to load.

In the meantime I confirmed that

  • our Fiji plugin
  • the default fiji h5 reader
  • MoBiE
    can read the compressed files.

@k-dominik k-dominik merged commit acb6e14 into ilastik:main Feb 20, 2024
16 checks passed
@k-dominik k-dominik deleted the change-default-export-format branch February 20, 2024 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants