TypeError: Object of type Shuffle is not JSON serializable

[kthyng]

Hi! I have some model output I am trying to serve with xpublish (and xpublish-host and kerchunk). This model output has compression, and when I try to access it through the server I get the following error, though I am able to just open the model output fine (pointing to the kerchunked file):

TypeError: Object of type Shuffle is not JSON serializable

Indeed, a "Shuffle" object is in the metadata, for example here is what it shows in the xpublish code for an example variable:

'Uwind_eastward/.zarray': {'compressor': {'id': 'zlib', 'level': 3}, 'filters': [Shuffle(elementsize=4)], 'chunks': [1, 362, 794], 'dtype': '<f4', 'fill_value': 'NaN', 'order': 'C', 'shape': [1, 362, 794], 'zarr_format': 2}

I'm not sure why the compressor items remained a dictionary but the filter item was converted to an object, which is what seems to cause the issue. In the metadata coming out of kerchunk the filter item looked like

"filters":[{"elementsize":8,"id":"shuffle"}]

(I'm not sure if that change in "elementsize" is important as I go back and forth between two cases I'm testing, but I can track it down if it seems relevant.)

Does anyone follow this? I am not sure how to set up an actual working example for this workflow yet.

[kthyng]

For reference, I decompressed the file with

ncks -L 0 nwgoa_1999-02-02.nc nwgoa_1999-02-02_decompressed.nc

and I do not hit any error anymore. Is there an issue with serving compressed model output?

[kthyng]

Update: I removed the filters encoding in my plugin for serving the model output and that seems to have fixed the issue. I've been trying to resist reset all of the encoding with ds.reset_encoding() so I did:

    for varname in ds.variables:
        del ds[varname].encoding["filters"]

I'll close this for now as I seem to have sidestepped the issue but will open it if needed. Thanks. 🦆

[kthyng]

Unfortunately of course that didn't work. Still not sure how to handle having filters necessary in the model output metadata.