Make the analysis storage chunk equal to checkpoint interval

[Lnaden]

Self-explanatory. Some variables are not chunked to the checkpoint_interval because they are not accessed in bulk usually (online free energy, timestamp).

This is an optimization for read/write.

Fixes #860

[jchodera]

Some variables are not chunked to the checkpoint_interval because they are not accessed in bulk usually (online free energy, timestamp).

I'd chunk anything that is written each iteration to the same interval. Otherwise, we won't get the speed benefit from not having to enlarge the file each iteration.

[Lnaden]

I'd chunk anything that is written each iteration to the same interval

Fair. The only one I would need to change then is the timestamp chunk since its written each interval. Online analysis is on its own interval, so I'll need to think about that a bit more.

[Lnaden]

This is currently throwing errors due to the following problem:

Unidata/netcdf4-python#566

Given that this has been open now for a year and a half, I'm not sure if it will ever be fixed.

[jchodera]

The code is here.

I guess we're stuck with chunksizes of 1 for now, but it would be worth contributing what must be the trivial change to fix this back to Unidata.

[jchodera]

It looks like you just need to modify line 1081
from

            if(var->dim[d]->len > 0 && chunksizes[d] > var->dim[d]->len)

to

            if(!var->dim[d]->unlimited && var->dim[d]->len > 0 && chunksizes[d] > var->dim[d]->len)

Maybe we should try to fix it for them?

[Lnaden]

@jchodera I had just written that up on the parent issue Unidata/netcdf-c#299 then came back here and saw the post. I'll see what they say about any pitfalls in chunking 2 variables on the same dimension differently with existing data.

[jchodera]

@Lnaden has proposed a PR fix in Unidata/netcdf-c#760

[jchodera]

@Lnaden : Can you revive this PR for the sams branch when able? Your updated NetCDF version is now live on conda.

[Lnaden]

Superseded by #912, closing