Skip to content

Work around xarray StringDType serialization error#1086

Merged
spencerkclark merged 2 commits into
mainfrom
fix/evaluator-power-spectrum
Apr 24, 2026
Merged

Work around xarray StringDType serialization error#1086
spencerkclark merged 2 commits into
mainfrom
fix/evaluator-power-spectrum

Conversation

@spencerkclark
Copy link
Copy Markdown
Member

@spencerkclark spencerkclark commented Apr 24, 2026

The evaluator integration test on beaker failed with the following error:

Details

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/conda/lib/python3.11/site-packages/fme/ace/evaluator.py", line 9, 
in <module>
    main(args.yaml_config, override_dotlist=args.override)
  File "/opt/conda/lib/python3.11/site-packages/fme/ace/inference/evaluator.py",
line 289, in main
    return run_evaluator_from_config(config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fme/ace/inference/evaluator.py",
line 459, in run_evaluator_from_config
    aggregator.flush_diagnostics()
  File "/opt/conda/lib/python3.11/site-packages/torch/utils/_contextlib.py", 
line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File 
"/opt/conda/lib/python3.11/site-packages/fme/ace/aggregator/inference/main.py", 
line 573, in flush_diagnostics
    write_reduced_diagnostics(
  File "/opt/conda/lib/python3.11/site-packages/fme/core/diagnostics.py", line 
60, in write_reduced_diagnostics
    to_netcdf_via_inter_filesystem_copy(ds, filename)
  File "/opt/conda/lib/python3.11/site-packages/fme/core/cloud.py", line 38, in 
to_netcdf_via_inter_filesystem_copy
    ds.to_netcdf(source)
  File "/opt/conda/lib/python3.11/site-packages/xarray/core/dataset.py", line 
2123, in to_netcdf
    return to_netcdf(  # type: ignore[return-value]  # mypy cannot resolve the 
overloads:(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/xarray/backends/writers.py", 
line 441, in to_netcdf
    dump_to_store(
  File "/opt/conda/lib/python3.11/site-packages/xarray/backends/writers.py", 
line 491, in dump_to_store
    store.store(variables, attrs, check_encoding, writer, 
unlimited_dims=unlimited_dims)
  File "/opt/conda/lib/python3.11/site-packages/xarray/backends/common.py", line
533, in store
    self.set_variables(
  File "/opt/conda/lib/python3.11/site-packages/xarray/backends/common.py", line
571, in set_variables
    target, source = self.prepare_variable(
                     ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", 
line 633, in prepare_variable
    datatype = _get_datatype(
               ^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", 
line 165, in _get_datatype
    return _nc4_dtype(var)
           ^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", 
line 186, in _nc4_dtype
    raise ValueError(f"unsupported dtype for netCDF4 variable: {var.dtype}")
ValueError: unsupported dtype for netCDF4 variable: StringDType(na_object=nan)

This should be fixed in the next release of xarray (pydata/xarray#11218), but for the time being we can work around it with the code in this PR.

Changes:

  • Modifies the way we assign "source" coordinate values to the dataset constructed in PairedSphericalPowerSpectrumAggregator.get_dataset such that they are of a unicode dtype rather than a string dtype.

  • Tests added

Copy link
Copy Markdown
Contributor

@Arcomano1234 Arcomano1234 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for quickly spotting this.

@spencerkclark spencerkclark enabled auto-merge (squash) April 24, 2026 21:15
@spencerkclark spencerkclark disabled auto-merge April 24, 2026 21:16
@spencerkclark spencerkclark enabled auto-merge (squash) April 24, 2026 21:16
@spencerkclark spencerkclark merged commit 910f9e8 into main Apr 24, 2026
7 checks passed
@spencerkclark spencerkclark deleted the fix/evaluator-power-spectrum branch April 24, 2026 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants