New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add compression to netcdf decomp files #1407
Comments
You can do that, but it has a huge affect on performance and is only available with netcdf-4 in serial. |
I mean compress the decomp file. The decomp file would have to be in netCDF-4, but decomp files are always written sequentially. Not sure that it would have a noticeable affect on performance, but I can test that. |
@wkliao the format that the PIO library now outputs is also understood by the PIO library. So if you can convert to that format, you will no longer need your own conversion format. Also I would like to see an ncdump -h of your file... |
The purpose of my dat2nc.c is also to reduce the size of decomp files. |
OK, maybe I should convert to this format. ;-) Here's what I see in your README:
For comparison, here's what I have now (but so far unused by anyone):
|
In my performance test I am seeing a decomp file of >1GB. Wow, that's big.
I can turn on compression so that when a netcdff-4 file is used for a netcdf decomp file, it can be automatically compressed.
The text was updated successfully, but these errors were encountered: