cmor.axis is not handling unicode string values correctly in Python

[mauzey1]

This is a followup of #611.

Passing axis values of unicode string type to cmor.axis will cause CMOR to store only the first characters of the axis values.

Example:

#Values passed to CMOR
sector_values = [
                  'barents_opening', 
                  'bering_strait',
                  'windward_passage', 
                  'davis_strait', 
                  'denmark_strait', 
                  'drake_passage',
                  'english_channel', 
                  'faroe_scotland_channel', 
                  'florida_bahamas_strait',
                  'fram_strait', 
                  'gibraltar_strait', 
                  'iceland_faroe_channel', 
                  'indonesian_throughflow', 
                  'mozambique_channel', 
                  'pacific_equatorial_undercurrent', 
                  'taiwan_luzon_straits'
                ]

sector_string = np.array(sector_values, dtype='U')
oline_axis_id = cmor.axis(table_entry='oline', units='kg s-1', length=len(sector_string), coord_vals=sector_string)

...
#What gets stored in the NetCDF file
 sector =
  "b",
  "b",
  "w",
  "d",
  "d",
  "d",
  "e",
  "f",
  "f",
  "f",
  "g",
  "i",
  "i",
  "m",
  "p",
  "t" ;

This can be fixed by converting the unicode strings to zero-terminated byte strings, which is type 'S' in numpy.

sector_string = np.array(sector_values, dtype='S')

Shouldn't this conversion happen in CMOR? Or should we change the code and documentation to say that only strings of type 'S' are accepted by cmor.axis?

[durack1]

@mauzey1 would it be easy to implement an input and output length check or similar, just to protect against this issue occurring again? I think the software should deal with type issues so a user can be oblivious about s types. @taylor13 agree?

[taylor13]

yes I agree, assuming @mauzey1 also agrees.

[durack1]

@mauzey1 I also think we should (so CMOR should) be able to handle unicode strings. Often I will rewrite strings to unicode, to ensure that they view correctly on html pages for e.g., so we should really support such a web standard format

[mauzey1]

I am trying to apply the solution that I suggested to @markcollier in #611 (comment) inside of the axis function in pywrapper.py.

cmor/Lib/pywrapper.py

Lines 432 to 440 in 16c7c35

	if data_type == 'S' or data_type == 'U':
	if data_type == 'U':
	coord_vals = coord_vals.astype('S')
	data_type = 'c'
	cbnds = 0
	for s in coord_vals:
	# print 'testing:',s,len(s)
	if len(s) > cbnds:
	cbnds = len(s)

If an array with unicode strings is passed to cmor.axis, then those strings will be converted to zero-terminated byte arrays.

[durack1]

@mauzey1 you know this code far better than me, would it benefit code maintenance to fix the fact that unicode strings aren't being dealt with correctly? This seems like a kludge fix that might squash this particular bug, but it may appear elsewhere