Community meeting - July 7th at 1 PM Eastern

[abkfenris]

It's almost that time again for another community meeting. We had a good discussion about Dask last time that lead to some ideas to play with.

I've personally been pondering if there is any metadata we want to be able to pass from a dataset provider to a router without necessarily opening a dataset, and pondering if we can abstract catalog support into a common plugin after seeing what @xaviernogueira did with catalog_to_xpublish, but if others have ideas to discuss, feel free to share them.

If you don't have an invite, reach out and we can add you.

[xaviernogueira]

I too was thinking about making a common STAC or Intake plugin. Since my work was on a project with limited hours, I went straight to build what they wanted, but I do think a pure plugin could be a better setup. The pattern of needing a string name for a dataset seems to complicate things, but I'm sure we could agree on a workaround.

I'd be happy to build a plugin with anyone who wants to collaborate as an outside-of-work project!

[abkfenris]

I haven't spent as much time with STAC or Intake catalogs, but I wonder if we could make a few assumptions:

• Dataset IDs will be unique in a catalog (I think STAC is explicit about Item and Collection IDs being unique)
• Catalog names will be unique, even if there are many sub catalogs (I think STAC is also explicit about sub Catalog IDs being unique)

Then maybe could flatten the structure of the catalogs:

• /catalog/{catalog_type}/{catalog_name}/hierarchy - existing catalog structure mapping to catalog datasets
• /catalog/{catalog_type}/{catalog_name}/datasets/{dataset_id}/ and then load all dataset routers here?

We may be able to flatten things further if we don't need to specify STAC/Intake catalog_type.

Then we could have a common CatalogRouter that provides those routes and have different types of CatalogProvider plugins, similar to the relationship between DatasetRouters and DatasetProviders.

Note: this doesn't cover how do we advertise routes and create a new/enhance an existing catalog.

[abkfenris]

Attendees:

• Alex Kerney / Gulf of Maine Research Institute / @abkfenris
• Joe Hamman / Earthmover / @jhamman
• Xavier Nogueira / LimnoTech/ @xaviernogueira
• Matthew Iannucci / RPS Group / @mpiannucci

Agenda and Notes:

• Dask
  • Experimental Dask client/cluster plugins Experimenting with Dask integration xpublish#208
  • Hard to make the worker environments match the server
  • How do we asyncronously compute a xarray Dask graph?
    • We should be able to await client.submit(da.sel(time=0).data)
    • https://distributed.dask.org/en/stable/asynchronous.html
  • Do we want to require Dask?
    • General consensus is No
    • Non async requests don't block with multiple threads, where as async requests block
• Share catalog plugin
  • Collections can share item IDs
  • Have we tried the :path parameter type? E.g. /foo/{p:path}/zarr
  • /catalogs/{catalog}/{p:path}/_routes_/zarr
  • TODO: open a ticket on fast api discussions?
• Dataset metadata
  • /datasets/{dataset_id}/files...
  • We don't want to get too far from being able to access xarray datasets directly, but there are circumstances when it would be nice to be able to be able to direct users directly to storage instead of hitting Xpublish and having it say translate zarr-to-zarr
  • Maybe ds.encoding['source'] might have some of that info
  • Wouldn't want to try to direct users to local file paths
  • Same data may be stored in multiple places (local, S3, GCS, OSN...)
• Zarr chunks
  • Encoding chunks do not match inferred chunks xpublish#207
  • something to try: ds.reset_encoding() before serving these datasets

[kthyng]

@abkfenris Hi can I be added to this meeting for the future? I understand they are monthly — is there a calendar somewhere I can add myself to?

[abkfenris]

Hi @kthyng , @jhamman manages the meeting invite, I'll give him a ping.