Append mode

[wachsylon]

Hi *,
first, thanks for your great work. I hope this is the right place for my questions.
With CMOR3, the cmor_write call changes in the C-version. Now, no file_suffix can be transferred as a parameter. How does CMOR then find the chunk file in append-mode in CMOR3? Could you please update the doc for cmor_write? It would be a great help, thanks a lot.

Another problem concerning the append-mode happens in CMOR-2.9.2 and some earlier versions, however I could not test it in CMOR3 yet: An error occurred with variables having a hybrid vertical axis. The auxiliary variable ps0 needs to be written with the parameter store_with in this case. However, the time axis is at least wrong or an error occurs when applying CMOR in append mode for these variables with hybrid vertical axes.
Are you aware of or can you reproduce the problem i tried to describe? ^^

The append mode is really helpful in the operational work and we would like to use it also for these kind of variables. So i am grateful for your help.

I work at the DKRZ and develop a CDO operator which uses the CMOR library.
So best regards from Germany,
Fabian Wachsmann

[dnadeau4]

I am looking at file_suffix code and hopefully, I can get that to work soon. I don't remember changing that code, so it must be a something I introduced.

I am not aware of the problem with append-mode. I will try to create a test to reproduce this. If you can provide some of your code using the cmor.write() function that will help me investigate it.

Thanks!

[wachsylon]

After registering all variables and axes I am looping over all time steps to write data of each variable however the error also occurs when not looping.
cmor_write(vars[i].cmor_varID, vars[i].data, vars[i].datatype, chunk_files[i], 1, &time_val, time_bndsp, NULL);
The chunk_files[i] parameter is gone in CMOR3. If a zfactor is needed for this variable, I call cmor_write a second time for that time step:
if ( vars[i].zfactor_id > 0 ) cmor_write(vars[i].zfactor_id, vars[ps_index].data, vars[ps_index].datatype, chunk_files[i], 1, &time_val, time_bndsp, &vars[i].cmor_varID);

This works well for all variables (for now) but not in append mode for variables with zfactors. In append mode, the first time_val of the variable which is to append on the chunk is lost in the output - either an Error occurs if the time_val does not fit in the time series or a warning occurs.
Thanks for the quick answer.

[dnadeau4]

I will get to it this week, and let you know by Friday. Thanks for the code....

[wachsylon]

Hi Denis,
may I draw your attention to this issue again? I still struggle with the APPEND_MODE. In cmor.c we have

        ierr = mkstemp(outname);
        unlink(outname);

and

/* -------------------------------------------------------------------- */
/*      Decides NetCDF mode                                             */
/* -------------------------------------------------------------------- */
        ncid = cmor_validateFilename(outname, var_id);

In validateFilename it is:

        fperr = fopen(outname, "r");
        if (fperr == NULL) {

/* -------------------------------------------------------------------- */
/*      ok it does not exists... we will open as new                    */
/* -------------------------------------------------------------------- */
            ierr = nc_create(outname, NC_CLOBBER | cmode, &ncid);

        } else {                /*ok it was there already */

But how can it be there already if mkstemp is used? And how can I use NetCDF APPEND_MODE then? How can I pass the chunk to append on?
Best regards,
Fabi

[dnadeau4]

It seems that this functionality was stripped when converting to 3.0.0. I will work on it today and will create a nightly version for you to try.

[dnadeau4]

I added back this functionality, you can give it a try using the following conda file.

https://anaconda.org/PCMDI/cmor/files?version=2017.09.22.3.2.8&channel=nightly

You can install the latest version using this command.

conda install -c conda-forge -c pcmdi/label/nightly cmor==2017.09.22.3.2.8

I also created a test for it.

https://github.com/PCMDI/cmor/blob/master/Test/test_python_appending.py

[dnadeau4]

@wachsylon did you give it a try? Thanks!

[wachsylon]

Thanks for the enhancement! Sorry for the late answer.
It worked in my tests, also for hybrid zaxis.

[mauzey1]

@dnadeau4 It looks like @wachsylon was satisfied by these changes. Should we close this issue.

In the API documentation, https://cmor.llnl.gov/mydoc_cmor3_api/, there is this sentence for the file_suffix parameter of cmor_write.

CMOR2 will be smart enough to figure out if a suffix was used when creating that file.

Should that be CMOR3 instead?

[dnadeau4]

@mauzey1 you can close it. Charles removed me from the PCMDI group due to the private repository authentication.