Is ZarrDAP an high-performance alternative to Pydap?

[aufdenkampe]

@rsignell-usgs, @abuddenb, @markccapece,

I was really happy to discover that you all publicly released this repo on GitHub recently, as we have been asked to explore and implement a Python-based, cloud-ready alternative to a THREDDS data server for sharing xarray-compatable object stores (i.e. netCDF & cf-compliant Zarr files).

From what I can tell, ZarrDAP does exactly that, acting as a DAP server "lite". Is that right?

Do you see this as an asynchronous, higher-performant alternative to Pydap?

What are your future development plans for ZarrDAP?

[rsignell-usgs]

@aufdenkampe , I don't know what the capabilities of ZarrDAP, but I'd sure like to know!
Maybe we just need to try deploying it and see for ourselves!

[J-NOAA]

Chad Whitney recently gave a talk about ZarrDAP at NOAA EDMW 2022

Starting at 1:13:48 of https://drive.google.com/file/d/1QZRDLH8G1N8ootB3Og_ofbpeCLJUd8ov/view

[aufdenkampe]

@J-NOAA, thanks for that tip and link! I just requested access to that Google Drive folder, which is currently private.

[rsignell-usgs]

I was granted access to the video, and just watched it. At the point of the NOAA EDMW 2022, it sounded like Chad was just starting his exploration of ZarrDAP, but mentioned they did have it running on EKS (as well as on prem HPC and locally on laptops). Is there a public facing endpoint?

If not, maybe we could set one up on the ESIP AWS account so folks could test it...

I've emailed Chad to see if he has any additional progress or insights to report here.

[ocefpaf]

Chad Whitney recently gave a talk about ZarrDAP at NOAA EDMW 2022

Starting at 1:13:48 of https://drive.google.com/file/d/1QZRDLH8G1N8ootB3Og_ofbpeCLJUd8ov/view

That video is closed to non-NOAA folks. Any chance of making it available to a larger audience?

[rsignell-usgs]

@aufdenkampe , I just found that that Chad Whitney has left NOAA as well. 😢

So it seems we won't be getting insights from former developers any time soon (and likely never).

Probably the best way forward is just to try installing and trying it out?

@abkfenris, @dblodgett-usgs

[J-NOAA]

Rich, when did Chad leave? I can still see him in the NOAA system. I know the manager of the zarrdap project. Maybe I should reach out to him?

[rsignell-usgs]

@J-NOAA I don't know, I was just told that he was a contractor and no longer with NCEI.

[abuddenb]

@rsignell-usgs @J-NOAA Chad is definitely still a contractor with NCEI; he's on my team. We're trying to fit zarrdap development into the schedule. We're not yet sure whether the upcoming holidays will help or hinder this effort.

[rsignell-usgs]

@abuddenb well that's some good news! Phew!

[aufdenkampe]

@abuddenb, that's great to know that you are planning on having Chad Whitney work on ZarrDAP in coming months.

My team at LimnoTech was recently tasked by @dblodgett-usgs to implement a Python OpenDAP server for Water Data Labs. Just a couple of weeks ago, we decided to try to move forward with ZarrDAP over Pydap. We hope to get started in December. So, as your plans solidify, please loop me into any conversations so that we can help each other out!

BTW, what's Chad Whitney;s GitHub handle?

[abkfenris]

I'd also be interested in any conversations. There are a few of us in the IOOS data management space that are exploring what possibilities are out there for a next generation of data server. A few months ago I implemented an OpenDAP router for Xpublish during the IOOS Code Sprint along with a bunch of other new Xpublish routers that our group built that week.

[cwgu]

Looking forward to having some time to schedule work with Zarrdap again. My schedule has been pretty full of late.

[abkfenris]

@mpiannucci might also be interested in any discussion.

[rsignell-usgs]

Would it make sense to have a little meet & greet webinar where different folks could show what they've done and talk about their plans? Give a thumbs up if you would like to attend such a meeting.

[rsignell-usgs]

Would 10am ET Friday Dec 9 work for folks? Hoping to get lucky, if not we can doodle.

[abkfenris]

That works for me.

[mpiannucci]

Good for me as well

[J-NOAA]

Works for me.On Dec 2, 2022, at 2:01 PM, Matthew Iannucci ***@***.***> wrote: Good for me as well —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[aufdenkampe]

Fri 12/9 @ 10 am ET works for me too! Seems like you got lucky @rsignell-usgs!

[rsignell-usgs]

Those of you who are interested please send your email address to rsignell@usgs.gov and I'll send a Teams calendar invite.

[J-NOAA]

@rsignell-usgs I sent you email but have not received our meeting notice yet.

[abkfenris]

Meeting notes are over on the Xpublish repo: xpublish-community/xpublish#138