simulationStartTime not being read in

[darincomeau]

The following error is appearing, causing some (but not all) time series tasks to not run:

File "/global/project/projectdirs/m3412/dcomeau/mpas_analysis_repos/develop/MPAS-Analysis/mpas_analysis/shared/timekeeping/utility.py", line 64, in get_simulation_start_time
    simulationStartTime = ''.join(simulationStartTime.astype('U')).strip()
TypeError: sequence item 19: expected str instance, MaskedConstant found

The analysis is run off current develop (hash 3e700c2).

The simulations on which this occurs were branched off master on May 29, 2020 (E3SM hash be8d6eadf, PR E3SM-Project/E3SM#3526). The mpas-source hash is bdf772aa; the latest ocean/develop merge into e3sm/develop for this hash is (MPAS-Dev/MPAS-Model@9073caf).

Perhaps overkill information, but I've been trying to track down recent changes that could point to the error.

tagging @stephenprice who has also seen this error.

[milenaveneziani]

I am not seeing this error using the same mpas-analysis hash, but probably a very different E3SM code (a version Luke has with GM/Redi changes).

[darincomeau]

Time series that do NOT run:
Antarctic melt time series, temperature anomaly trends vs depth, OHC time series, sea ice area/volume

Time series that DO run:
Antarctic region time series, transport time series

[darincomeau]

@milenaveneziani I haven't seen it either before these runs off a pretty recent E3SM hash. I'm hoping the narrowed time frame will identify the source, but I've had no luck myself.

[milenaveneziani]

Don't know if it helps, but here is the last mpas-source hash that says 'Merge PR#.. into ocean/develop' in the branch I am using:
78ddd84 Merge PR #531 'ocean/ARM_update' into ocean/develop
(there are other commits after this, but those are GM/Redi related)

[darincomeau]

Yeah looks like in mine the latest two ocean/develop PRs are MPAS-Dev/MPAS-Model#545 and MPAS-Dev/MPAS-Model#496

mpas-source git logg:

*   bdf772aa (HEAD, origin/e3sm/testing, origin/e3sm/develop) Merge branch 'ocean/develop' into e3sm/develop
|\
| *   80d2f30d (origin/ocean/develop) Merge PR #496 'philipwjones/ocean/gpumesh' into ocean/develop
| |\
| | * f074a89b White space, vim folds.
| | * 8e0a329c COMPASS: remove block test from nightly regression
| | * c8fee12f eliminated the mesh structure in favor of public module variables
| | * 6fc0133d Added new ocean mesh structure that is replicated on GPU
| |/
* |   9073cafe Merge branch 'origin/ocean/develop' into e3sm/develop
|\ \
| |/
| *   9b19e527 Merge PR #545 'xylar/ocean/compass_revert_conda_mpi' into ocean/develop

[xylar]

@darincomeau and @milenaveneziani, this is almost certainly related to the recent switch from PIO1 to PIO2 (scorpio-classic to scorpio). Strings seem to be getting null characters \x00 when read in by netcdf4 (and maybe xarray, though maybe not). I'll look into this but here's a related issue for the paraview extractor:
MPAS-Dev/MPAS-Tools#317

[xylar]

I can probably fix this next week but probably not today.

[xylar]

This code is using netcdf4, which confirms my suspicion that xarray may be fine with these strings with extra NULL characters but netcdf4 is not:
https://github.com/MPAS-Dev/MPAS-Analysis/blob/develop/mpas_analysis/shared/timekeeping/utility.py#L61-L65

I think the easiest would be to modify this code to use xarray instead. This would likely involve:
https://github.com/MPAS-Dev/MPAS-Analysis/blob/develop/mpas_analysis/shared/io/utility.py#L368-L392
One could find a call to decode_strings and use that as an example to modify the get_simulation_start_time in a similar way.

[xylar]

Could you see if #705 fixes this?

[darincomeau]

Thanks @xylar for looking into this (but no more until next week!). With #705 , I'm getting a different error:

Traceback (most recent call last):
  File "/global/project/projectdirs/m3412/dcomeau/mpas_analysis_repos/develop/MPAS-Analysis/mpas_analysis/__main__.py", line 373, in add_task_and_subtasks
    analysisTask.setup_and_check()
  File "/global/project/projectdirs/m3412/dcomeau/mpas_analysis_repos/develop/MPAS-Analysis/mpas_analysis/shared/time_series/mpas_time_series_task.py", line 203, in setup_and_check
    firstYearInputFiles = self.historyStreams.readpath(
  File "/global/project/projectdirs/m3412/dcomeau/mpas_analysis_repos/develop/MPAS-Analysis/mpas_analysis/shared/io/namelist_streams_interface.py", line 444, in readpath
    startDate = string_to_datetime(startDate)
  File "/global/project/projectdirs/m3412/dcomeau/mpas_analysis_repos/develop/MPAS-Analysis/mpas_analysis/shared/timekeeping/utility.py", line 113, in string_to_datetime
    _parse_date_string(dateString, isInterval=False)
  File "/global/project/projectdirs/m3412/dcomeau/mpas_analysis_repos/develop/MPAS-Analysis/mpas_analysis/shared/timekeeping/utility.py", line 462, in _parse_date_string
    = [int(sub) for sub in hms.split(':')]
  File "/global/project/projectdirs/m3412/dcomeau/mpas_analysis_repos/develop/MPAS-Analysis/mpas_analysis/shared/timekeeping/utility.py", line 462, in <listcomp>
    = [int(sub) for sub in hms.split(':')]
ValueError: invalid literal for int() with base 10: '00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

[xylar]

@darincomeau, same string issue (related to PIO2) manifesting itself somewhere else. Could you point me to your config file and I'll try to get things running?

[xylar]

@darincomeau, I'm getting back to this now. Hopefully, I'll be able to fix it tonight.