Can 2.8.0 still create netCDF3 Classic ?

[jypeter]

@dnadeau4 and @doutriaux1 If I run the script at the end of #98 , I expect to get netCDF3 Classic because I have specified:

# Write NetCDF3 Classic
value = 0
cdms2.setNetcdfShuffleFlag(value) ## where value is either 0 or 1
cdms2.setNetcdfDeflateFlag(value) ## where value is either 0 or 1
cdms2.setNetcdfDeflateLevelFlag(value) ## where value is a integer between 0 and 9 included

If I run the script with my old 2.1.0, I do get NetCDF3 Classic

> ncdump -k t2m_1851.extracted.daily.nc
classic

But If I use 2.8.0, I get netCDF-4 classic

> ncdump -k t2m_1853.extracted.daily.nc
netCDF-4 classic model

Note that if I comment out the cdms2.set lines in the script and run it with 2.1.0, I also get netCDF-4 (as expected)

> ncdump -k t2m_1851.extracted.daily.nc
netCDF-4 classic model

Is this a bug in 2.8.0? Or is there a new flag you have to use if you want netCDF3?

[doutriaux1]

it's possible something changed in Netcdf 4.4 should be an easy fix. @dnadeau4 we should look into a new default (NC3 classic?) looks like people are typing this all the time.

[dnadeau4]

https://github.com/UV-CDAT/cdms/blob/master/Lib/dataset.py#L183

[jypeter]

OK, so the message I get when I don't set anything is the following

>>> g.write(t2m)
/home/share/unix_files/cdat/miniconda2/envs/cdatm14/lib/python2.7/site-packages/cdms2/dataset.py:1884: Warning: Files are written with compression and shuffling
You can query different values of compression using the functions:
cdms2.getNetcdfShuffleFlag() returning 1 if shuffling is enabled, 0 otherwise
cdms2.getNetcdfDeflateFlag() returning 1 if deflate is used, 0 otherwise
cdms2.getNetcdfDeflateLevelFlag() returning the level of compression for the deflate method

If you want to turn that off or set different values of compression use the functions:
value = 0
cdms2.setNetcdfShuffleFlag(value) ## where value is either 0 or 1
cdms2.setNetcdfDeflateFlag(value) ## where value is either 0 or 1
cdms2.setNetcdfDeflateLevelFlag(value) ## where value is a integer between 0 and 9 included

Turning all values to 0 will produce NetCDF3 Classic files
To Force NetCDF4 output with classic format and no compressing use:
cdms2.setNetcdf4Flag(1)
NetCDF4 file with no shuffling or deflate and noclassic will be open for parallel i/o
  "for parallel i/o", Warning)
<cdms2.fvariable.FileVariable object at 0x2b3e8d205550>
>>> g.close()

I have experimented with different options' combinations:

• If I only use cdms2.setNetcdf4Flag(0), I still get the full warning (why not), and still get a NetCDF4 output (netCDF-4 classic model)
• If I do set all four options to zero, I get indeed NetCDF3 output (64-bit offset). That's NetCDF3 with large file support when before we only got NetCDF3 classic (classic), so that's a nice improvement

So I think that the following information line should be after all four options rather than only after the first 2 options
Turning all values to 0 will produce NetCDF3 Classic files

I wonder if the last line displayed in the warning is not some kind of bug, or left-over, because it seems odd:
"for parallel i/o", Warning)

This is a very useful warning, because I never remember all the options I should use, so I wonder if you could use some indentation to make it more readable, and display the warning without starting with /home/share/unix_files/cdat/miniconda2/envs/cdatm14/lib/python2.7/site-packages/cdms2/dataset.py:1884:

Should you use returns instead of returning?

Should you use set the compression level instead of set different values of compression?

If I take what's above and several extra changes, I get the following text, which I find slightly better (but I may be wrong)

Warning: Files are written in NetCDF4 format, with compression and shuffling
  You can query the current shuffling and compression settings with:
    cdms2.getNetcdfShuffleFlag() ## returns 1 if shuffling is enabled, 0 otherwise
    cdms2.getNetcdfDeflateFlag() ## returns 1 if compression is used, 0 otherwise
    cdms2.getNetcdfDeflateLevelFlag() ## returns the compression level
    cdms2.getNetcdf4Flag() ## returns NEED_SOME_INFORMATION_HERE

  You can turn off shuffling and set the compression level with:
    cdms2.setNetcdfShuffleFlag(value) ## where value is either 0 or 1
    cdms2.setNetcdfDeflateFlag(value) ## where value is either 0 or 1
    cdms2.setNetcdfDeflateLevelFlag(value) ## where value is an integer between 0 and 9 included
  To Force NetCDF4 output with classic format and no compression use:
    cdms2.setNetcdf4Flag(1)  # where value is either 0 or 1

  Turning ALL values to 0 will produce NetCDF3 Classic files

  NetCDF4 file with no shuffling or deflate and noclassic will be open for parallel i/o

That said, I have other questions... I'm not sure I understand fully what setNetcdf4Flag does. And when you write NetCDF4 file with no shuffling or deflate and noclassic will be open for parallel i/o, it seems equivalent to having

cdms2.setNetcdfShuffleFlag(0)
cdms2.setNetcdfDeflateFlag(0)
cdms2.setNetcdfDeflateLevelFlag(0)
cdms2.setNetcdf4Flag(0)

and you get NetCDF3 and not the advertised NetCDF4!

My last question is: when do you disable the printing of the warning for the first write operation? It is not printed when you explicitely set all the options to 0, but it is printed when you set all options to 1! It is also still printed when I use different options combinatations

[jypeter]

Hmmm, I've browsed quickly cdms/Lib/dataset.py and it seems there is a function that could also help the user force netcdf3 output, except that it is missing the cdms2.setNetcdf4Flag(0) option!

def useNetcdf3():
    """ Turns off (0) NetCDF flags for shuffle/cuDa/deflatelevel
    Output files are generated as NetCDF3 Classic after that
    """
    setNetcdfShuffleFlag(0)
    setNetcdfDeflateFlag(0)
    setNetcdfDeflateLevelFlag(0)

[dnadeau4]

@jypeter Thanks! This is great! I will work carefully on it tomorrow, today I have to release CMOR 3.2.2 which will take all day.