Xpublish community meeting - April 7th at 1 Eastern

[abkfenris]

It's been a few months since we last spoke (#138), and I know that there has been a lot of cool stuff built by folks since then, so it'd be cool to see what folks have been up to.

@rsignell-usgs somehow found that 10 am Eastern on a Friday worked for a lot of folks last time, so how about we try to meet at 10 Eastern on April 7th?

If that doesn't work for a bunch of folks we can find another time.

That's apparently a holiday in some places, so how about people add their availability to When2Meet and we can decide on a time by Tuesday of next week.

Looking at folks availability, lets go with Friday the 7th at 1 Eastern.

I’ve sent an invite directly to most folks who responded to When2Meet, but for anyone else who would be interested in joining tomorrow: https://gmri-org.zoom.us/j/86069490606?pwd=MXdRaEk2UjdxaWFXcE1IMC9XNEx2dz09
and a I drafted a few things to chat about https://hackmd.io/@LIrAVhP6Rzm1d_vhTynHZQ/ryx7rvx5-h

[jonblower]

Great idea, although April 7th is a public holiday in the UK (Good Friday)

[abkfenris]

I did not expect that I'd find there are multiple dates for Good Friday when I opened the Wikipedia page or that there would be that big of a page on just how to calculate the date.

I guess I'll fire up a when2meet.

[abkfenris]

Updated the initial post with a When2Meet (though I'm leaving Friday the 7th in in case there is a majority who can meet then).

[abkfenris]

I'm going to call it now (so that folks don't need to worry about if we are going to meet tomorrow), despite the holiday, that's when most folks are available.

Lets meet on Friday the 7th at 1 Eastern.

[abkfenris]

I’ve sent an invite directly to most folks who responded to When2Meet, but for anyone else who would be interested in joining tomorrow: https://gmri-org.zoom.us/j/86069490606?pwd=MXdRaEk2UjdxaWFXcE1IMC9XNEx2dz09
and a I drafted a few things to chat about https://hackmd.io/@LIrAVhP6Rzm1d_vhTynHZQ/ryx7rvx5-h

[abkfenris]

Attendees:

• Alex Kerney / Gulf of Maine Research Institute / @abkfenris
• Xavier Nogueira / LimnoTech / @xaviernogueira
• Joe Hamman / Earthmover / @jhamman
• Kyle Wilcox / Axiom Data Science / @kwilcox
• Josh Rhoades / Axiom Data Science
• Shane St Savage / Axiom Data Science
• Jonathan Joyce / RPS Group / @jonmjoyce
• Rich Signell / USGS / @rsignell-usgs
• Anthony Aufdenkampe / LimnoTech / @aufdenkampe
• Micah Wengren / NOAA IOOS / @mwengren
• Matthew Iannucci / RPS / @mpiannucci

Agenda and Notes:

• Previous
  • 2022-12-09 Xpublish & ZarrDAP meeting notes
• What have folks been up to

  • Alex

    • Plugins
    • Working on documentation updates
      • Documentation updates xpublish#159
      • Eventually working towards the Getting Started/User Guide/API Ref/Contributing structure that other Pydata projects have
    • Answering some lingering issues with plugin based solutions, I'll probably convert those to discussions. Once I'm happy with docs updates and we have a new release, I'll start closing some issues that are now not really relevant

  • Rich

    • Set up Xpublish Slack channel on ESIP

  • Kyle

    • Have a bunch of FastAPI services already
    • Intake plugin - https://github.com/axiom-data-science/xpublish-intake
    • A host project so that they can standardize their deployments - https://github.com/axiom-data-science/xpublish-host
    • Deployment?
      • Not at scale yet
      • Dask clusters on the backend
      • THREDDS and OpenDAP isn't keeping up
      • Docker image with volume mounts
      • On prem

  • Anthony

    • Limnotech for USGS National Hydrological Geospatial Fabric (NHGF, hydrofabric) team
      • Primarily working on PyGeoAPI OGC EDR deployment and async optimization
      • Also tasked with Xpublish OpenDAP endpoints for same datasets
        • Plan to start with Alex's https://github.com/gulfofmaine/xpublish-opendap
        • Alex thinking about looking at ZarrDAP to get relevant
        • Rich will touch base with the ZarrDAP folks
        • Axiom also needs OpenDAP
    • USGS moving a lot of things into the cloud/Zarr

  • Matt

    • Trying to replicate what THREDDS can do for model data
    • Kerchunk harvesting from NODD and then served with Xpublish to serve aggregations
    • XREDS - XaRray Environmental Data Server
    • Serving direct from cloud storage
    • Working on
      • WMS
        • Trying to standardize for different grids
        • Both it and EDR rely on some knowledge of the grids
      • Subsetting
      • Productionizing, only really funded for a prototype, how to scale out access '45 tiles at the same time'
      • Want to be able to deal with FVCOM and ROMS grids
      • Figure out how to make Dask deal with the subsetting
    • Kyle: a higher level interface for wrapping a dataset
• Plugin Discussion
  • Alex's vision is that a server plugin can provide capabilities to other plugins
• A Github org for Xpublish
  • Do we want multiple orgs: 'xpublish-community' and 'xpublish-experiments' or similar?
  • Can we move/abstract the zarr plugin to a new repo or different default prefix (i.e. /zarr)
• Next release - 0.3
  • Plugin support
  • What do we need to make a release?
    • I think if we make a tag then create a Github release this workflow will run: https://github.com/xarray-contrib/xpublish/blob/main/.github/workflows/pypipublish.yaml
      • I'm not sure if it will run successfully if I don't have access to the secrets (and will they make the leap to a new org?)
      • Should run when a Github release is created, and should already have secrets tied to Joe's account
  • Get zarr prefix
  • Release from the new org

---

Lost people when Alex's free Zoom account cut the call. About half the folks logged back on.

---

• How often should we meet?
  • Monthly, first friday of the month, Joe will set up a Google Meet
• Micah: Code sprint?
  • Find some time to virtually collaborate?
  • Post-ESIP summer meeting? Week of July 24 - 28.
• Caching
  • Redis with cachey, multiple operators could work on the same key
  • Maybe get rid of cachey
  • Dask is using cachey under the hood, but isn't
• Catalogs
  • How should endpoints be advertised
• Dask
  • Kyle and Matt have experimented with a few different cluster setups. Some issues with duplicate downloading of chunks

Action Items:

• Alex: Continue working on docs
• Joe to set up the xpublish-community org and move Xpublish repo there Planning to move this repository to the xpublish-community organization xpublish#181
  • Alex will set up community resources (discussions)
  • Matt will work on getting permission and moving RPS developed routers to the community org
• Alex to set up xpublish-expirments org
• Add /zarr prefix to the Zarr router #182
• Create a 0.3.0 release #183
• Alex or others: Create threads discussing caching, dask, and catalogs in new discussions https://github.com/xpublish-community/community/discussions
• Aiming for monthly meetings, going with the first Friday of the month at 1 Eastern. Monthly community meeting - First Friday at 1 PM Eastern #5

[aufdenkampe]

@abkfenris, thanks for moving forward on the first few action items by creating these two new orgs and moving repos into them:

• https://github.com/xpublish-community
• https://github.com/xpublish-experiments

And thanks for inviting me to xpublish-experiments!

I noticed that most members of the two orgs have their Organization Visibility set to private. Could each of you go to those links and make your visibility public? That will make us look like more of a community, and only individuals can change their visibility.

• https://github.com/orgs/xpublish-community/people (UPDATED)
• https://github.com/orgs/xpublish-experiments/people

[abkfenris]

Good call on the community management. The first visibility link should be https://github.com/orgs/xpublish-community/people

[jonmjoyce]

I'm only seeing Kyle and Alex in the xpublish-community. There's no option to change my visibility. I do see a list of people in xpublish-experiments so I wonder if the two are configured differently, or perhaps I'm not a member of xpublish-community yet.

[abkfenris]

@jonmjoyce I just invited you to xpublish-community.