Community meeting - September 1st at 1PM Eastern

[abkfenris]

It's about that time again. This time a few folks from the Tiled Project (repo) are planning on joining us, so that we can discuss what we can learn from each others projects, and how we may be able to collaborate.

As usual, if you don't have an invite, reply and we can get that figured out.

[kthyng]

I was planning to come but my daughter is home sick and I won't be able to meaningfully engage. I see the interesting conversation linked above. Where can I find the notes from the meeting?

[abkfenris]

We keep the running meeting notes in HackMD, and the somewhat cleaned up version gets added here afterwards. Check ESIP Slack.

[kthyng]

Thank you!

[abkfenris]

Note on the notes, I did an especially poor job taking notes this month. Sorry folks! Thanks to those who stepped in to clarify when I only got some shorthand down.

Attendees:

• Alex Kerney / Gulf of Maine Research Institute / @abkfenris
• Jonathan Joyce / RPS Ocean Science / @jonmjoyce
• Matthew Iannucci / RPS Ocean Science / @mpiannucci
• Dan Allan / Brookhaven National Laboratory / @danielballan
• Garrett Bischof / Brookhaven National Lab - NSLS2 / @gwbischof
• Padraic Shafer / Brookhaven National Lab - NSLS2 / @padraic-shafer
• Xavier Nogueira / LimnoTech / @xaviernogueira

Agenda and Notes:

• Meet and greet with Tiled: Learning from xpublish bluesky/tiled#523 (comment)
  • Indivudal intros
  • Xpublish intro
  • Tiled intro
    • Pain point
      • IO was baked too deeply into scientific code
        • Made a Intake like library before Intake
        • Realized that a service was an interesting thing
          • Needed to bring data to more than just Python users
    • Started by working within intake
      • Intake had a prototype HTTP server, intake-server
      • Client-side xarrays pull chunks from server-side xarrays -- cool idea!
      • Contributed 40-45 PRs to intake over a year or two to try to develop this into something more production
      • Found that it was too Python focused to be language agnostic
    • Don't have a lot of existing services that exist in the space
    • Centralized on structures
      • Array
      • Table (arrow-like)
      • Awkward
      • Containers...
    • Three serving modes
      • File based, a read-only view (walk a directory) tiled serve directory files/
      • As the source of truth, clients write data into it tiled serve catalog catalog.db --writable-storage data/
      • Proxying from another service with custom Python code
      • Some combination of the above tiled server config config.yml
    • Focused on data access, slicing, not general-purpose data processing endpoints
      • Concerned about resource requirements for processing
      • Put it in a separate microservice?
      • Concerned about frequently-evolving or growing list of processing needs from our diverse user groups