Add columns_with_iterables parameter to esm_datastore

[dougiesquire]

This PR adds an optional columns_with_iterables parameter to the intake_esm.esm_datastore API that specifies the columns to convert with ast.literal_eval. This enables intake yaml descriptions of intake-esm catalogs with multi-variable assets that work by default. See #587 for context/motivation.

Note, I also explored using intake dataset transforms, but I don't think this provides quite the functionality we need.

Fixes #587

Checklist

• Unit tests for the changes exist
• Tests pass on CI
• Documentation reflects the changes where applicable

[mgrover1]

I like this approach, especially with the error catching here. Would you be willing to add an example as well? So the docs reflect these new changes?

[mgrover1]

Also a test here would be helpful

[dougiesquire]

Thanks @mgrover1. Tests and docs added. Note, I had to pin netcdf4<1.6.0 to get the tests to pass and docs to build. This is due to a change in 1.6.1 that seems to be causing issues all over the place, see e.g. Unidata/netcdf4-python#1192

[mgrover1]

Just a few minor changes - I appreciate the contribution @dougiesquire !! Great work!

[andersy005]

thank you for this addition, @dougiesquire & @mgrover1!

[dcherian]

Should this info be stored in the JSON instead? It's a property of the catalog, so the information can be specified at write-time instead of at read-time.

[andersy005]

Should this info be stored in the JSON instead? It's a property of the catalog, so the information can be specified at write-time instead of at read-time.

i'm definitely in favor of supporting this when it's defined in the JSON and adding it to the spec: https://github.com/intake/intake-esm/blob/main/docs/source/reference/esm-catalog-spec.md

[mgrover1]

Should this info be stored in the JSON instead? It's a property of the catalog, so the information can be specified at write-time instead of at read-time.

i'm definitely in favor of supporting this when it's defined in the JSON and adding it to the spec: https://github.com/intake/intake-esm/blob/main/docs/source/reference/esm-catalog-spec.md

Agreed! I think allowing both here makes sense.