Shapes not aligned rrror when appending to a FileVariable in Python 3

[zshaheen]

This issue is only present in the Python 3 version of the code, not the Python 2 version.
When doing something like the example below, which concatenates data to a file, we get errors using Python 3.

out_file = cdms2.open(fnm, 'w')
for month_file_nm in monthly_files:
    month_file = cdms2.open(month_file_nm)
    data = month_file['FLNT']
   
    out_file.write(data)

The error is this:

Traceback (most recent call last):
  File "cdms_timeseries.py", line 69, in <module>
    out_file.write(data)
  File "/export/shaheen2/anaconda2/envs/acme_diags_env_dev_py3/lib/python3.6/site-packages/cdms2/dataset.py", line 2068, in write
    v[index:index + len(vec1)] = var.astype(v.dtype)
  File "/export/shaheen2/anaconda2/envs/acme_diags_env_dev_py3/lib/python3.6/site-packages/cdms2/fvariable.py", line 113, in __setitem__
    self._obj_.setitem(*(index, numpy.ma.filled(value)))
ValueError: shapes are not aligned

[dnadeau4]

@zshaheen Do you have a path to your files?

[zshaheen]

The files are located here: /p/user_pub/work/E3SM/1_0/piControl/1deg_atm_60-30km_ocean/atmos/129x256/model-output/mon/ens1/v1. You can access this dir from aims4, acme1, or any other machine that has the correct drive mounted.

[zshaheen]

@dnadeau4 I wonder if using 'a' instead of 'w' would work. Since we are appending to the file, shouldn't we use 'a' instead?

[doutriaux1]

@zshaheen we need w the first time because the file does not exists. If i remember correctly you are not closing the file in between calls to write. Beside it works for 2.0. Seems like a 3.0 issue.

[zshaheen]

Just going off on a small tangent, but when you use 'a', it gives the following error:

CDMS system error: No such file or directory
CDMS I/O error: Opening file /export/shaheen2/e3sm_diags_timeseries/cdat_climo_results/20180129.DECKv1b_piControl.ne30_oEC.edison_11_climo.nc

But does this mean that it still writes to the file?

Because I use 'a' as the default option for when I'm creating the climo using CDAT. I'd hope it's not messing up on something.

[dnadeau4]

@zshaheen this issue is about setitem which change from py2 to py3. It use to call setslice and CDMS now enter in the wrong function of Cdunif. I just have to remember how I fixed it last year for Transient Variables and do the same with File Variables.

[dnadeau4]

8209099

still need a test...

[dnadeau4]

@zshaheen can you send me the files for the test?

[zshaheen]

@dnadeau4 Wouldn't any files work? I would think so.

[dnadeau4]

I think so, but I wanted you to write the test to make sure it works the way you want it.

[dnadeau4]

@zshaheen Can you close this if it works for you?

[dnadeau4]

@zshaheen can you write test for this issue?