Developer team meeting notes

[eileenrmartin]

Here, we'll have someone from each developer team meeting take high-level notes to indicate discussion topics and make any project decisions clear.

[eileenrmartin]

Oct. 19, 2022

• intros (Derrick Chambers, Peiyao Li, Ahmad, Hafiz, Frantisek, Ge Jin, Adesh, Nicholas, Donglin Zhu, Aaron Matt Mendoza, Brad Lipovsky, Parker Sprinkle)

• new contributors (aaronjgirard, JacksonZhu90, danesnick, bradlipovsky, spri902)

• check out dascore.org for info on becoming a developer

• using discussions and projects for planning

• future presentations may be recorded - Jin to create shared folder for videos with link in the dev team meetings - be sure to focus on quality for outward facing material

• Hafiz Issah discussed his experience contributing the TDMS format (new dev. experience):
  - needed additional permission to create a new branch in DASCore
  - noted utility of help page for developers (can submit pull request to edit this page to improve it for new users)
  - used PyTest to create test cases to ensure code was working properly
  - had to create small synthetic data file for testing

• Note opensourcecourse.dev page has useful tools for new developers getting started (potential course at Mines)

• Discussion on whether to include visualization in DASCore or other DASDAE packages
  - Keep DASCore with minimal dependencies (waterfall plot and wiggle plot)
  - separate visualization package with more detailed visualizations with fancier features (3D visualization, multiresolution plot)
  - Examples of packages to wrap: PyVista, Plotly, Bokeh

• To maintain: list of DASCore dependencies, and require group discussion for adding additional dependencies

• To do (Eileen, Jin, Derrick, Nicholas, Aaron) before next meeting: propose a governance model, reference at https://opensource.guide/leadership-and-governance/

• Note: Parker interested in adding Sintella format - will add to project for tracking work

• Did overview of upcoming timeline and goals (including summer '23 workshop)

• Discussed needs for interoperability with Madagascar and Obspy (will do through separate packages, but I/O format can output appropriate formats for Madagascar and Obspy)

• Note: Aaron may be involved in a madagascar workshop next year for those who are interested

• Next time: assigning tasks and overview of prioritization of I/O, visualization needs

• Next time: presentation on governance model proposal

• Next time: outline scope of interfaces with Madagascar/Obspy in DASCore vs other package, including discussion of license compatability

[eileenrmartin]

Agenda for Nov. 2, 2022 developer team meeting:

1. presentation on governance model proposal (summary of Nov. 1 discussion of Ge Jin, Eileen Martin, Derrick Chambers, Nicholas Danes, Aaron Girard)
2. outline scope of interfaces with Madagascar/Obspy in DASCore vs other package, including discussion of license compatibility
3. assigning tasks and overview of prioritization of I/O, visualization needs

[eileenrmartin]

Also forgot to add item 0. if there is anyone on the call who is joining for the first time, we'll do a quick-format introduction of the group (shorter than our first meeting)

[aaronjgirard]

DASDAE team check-in meeting November 2

Attendees: Eileen Martin, Ge Jin, Derrick Chambers, Aaron Girard, Frantisek Stanek, Ahmad Tourei, Ana Garcia, Brad Lipovsky, Donglin Zhu, Fu Yin, Hafiz Issah, Peiyao Li

Talking about contributing (CoC, dacore.org website) and rules enforcement. Look at contributing page to see definitions of what is DAScore and what is DASDAE (and what isn't, should be in new packages).

Adding packages should be names something else (Jin)

Derrick is currently Benevolent Dictator for Life (BDFL) but in Oct 2023 leadership will be moved to a more shared style based on who is developing and contributing. This is intended to be a rotating role. The real goal is to make sure the vision of the project is maintained, so that new merges and additions follow the desired path. (Eileen will add some language that this is not just for DAScore leadership)

One thing to add, is once you have two succesful pull requests, you can receive more permissions to become a developer. We will also keep recordings of meeting and keep a video library so that later contributors can keep up to speed.

Will have to make 2 recorded videos on how to install, write tests, etc (one for developers, and one for Docker). Eileen and Hafiz will take lead on this. They will also take some notes and make comments on things that were difficult so that Derrick could update the documentation for difficult parts.

• Frantisek also created an image that can be a dfferent (more foolproof) way of making sure everything is compatible during an install.
• Must add a disclaimer to say that videos can become outdated, but documentation pages will not be, since to see them the code needs to be compiled.

Bug reports need to have specifics to make the bug reproducible in some environment

Using Madagascar: we want to avoid making a full install. However, we can look for some specific priority operations madagascar does and import/rewrite them. Must develop a way to write RSF files so that madagascar could be used separately (external to DAScore).

ObsPy: Another package we may want to use part of or interface with. But should be done separately, like madagascar.

SEGYsak to read segy files. ObsPy can also manage this. However we may not want to deal with SEGY as a bad format. We can do a way where when you want to read a SEGY it will tell you that you need an optional dependecy (SEGYsak) and allow it to be installed or not.

For things that are in-progress and complete, we are going to have an in-review column as well. After a week or two, completed items will be archived. In items, we will use hashtags to note which items are associated with different discussions. Low- medium- ans high-priority items were discussed.

[eileenrmartin]

Ahmad has volunteered to help with making a table in the docs that gives an overview of what data formats are supported.

Derrrick and Eileen will make a list of good first project items on the planning project scrum board.

Eileen to post link to meeting recording here once that's done processing.

[eileenrmartin]

Here is the video recording link: https://mines.zoom.us/rec/share/4YER3uX8wnXKDHcVWB9bB3ENrOA9NDVSiIqxkoh4gwL2aHa40e3NDdCrhZUN8n8h.1svxru3SXMMSFw-6
Passcode: M+c#.GA6

Also, adding to the notes on attendees, Parker Sprinkle also joined.

[eileenrmartin]

As was suggested in the Nov. 2 developer team meeting, let's plan a DASCore development sprint after finals. Each person or group will choose an item to work on.

Fill out your availability by Nov. 16 at: http://whenisgood.net/1234gds
We can all see responses here: zrir9b2

We'll get food for the folks on the Mines campus, and we'll set up a couple check-in points at a few points in the day when we'll have a zoom call with remote participants too (Eileen to look into whether we can do anything regarding snacks for remote participants - probably)

[eileenrmartin]

Proposed agenda for Nov. 16 developer team check-in:
0. if there is anyone on the call who is joining for the first time, we'll do a quick group intro

1. new scrum board with a list of some good intro/starter projects https://github.com/DASDAE/dascore/projects/2
2. scheduling DASDAE code sprint following finals/AGU, so be sure to fill out your availability http://whenisgood.net/1234gds
3. AGU lunch get-together Wed. Dec. 14?
4. discuss contribution of examples of DASCore recipes https://dascore.org/recipes/overview.html
5. Any other points that folks would like to discuss tomorrow? If so, please feel free to add to this discussion.

[eileenrmartin]

Additional point: DASDAE recording month priorities

[eileenrmartin]

From Fu Yin: You can see the live meeting notes here: https://otter.ai/u/SfyX1Oe0Vt7fFKMnbExj-b1qPSs?utm_source=va_chat_link_1

Scroll back to review anything you missed.

[eileenrmartin]

A few high level points:

• A couple resources were shared:
  - on Pydantic https://pydantic-docs.helpmanual.io/
  - a list of topics for the open source course Derrick will lead in spring at https://opensourcecourse.dev/ which will take place at Mines on Wednesdays 3-4 (Eileen to be in charge of classroom recording so lectures can hopefully be shared)

• DASDAE developer team lunch on Wednesday Dec. 14 at the AGU meeting. RSVP to the calendar invite, or email Eileen to RSVP.

• DASDAE code sprint will be throughout the day Dec. 20. Choose something from the scrum board (prioritization of I/O formats for DAS global recording month), or propose your own project. Free food for those who are on the Mines campus. We'll have a hybrid setup with a couple of short check-in calls throughout the day so collaborators who aren't at Mines can connect, too.

• Decided that we shouldn't provide an output writer for most of the interrogator specific data formats. RSF will have an output (allows interfaces with Madagascar for array processing), and miniDAS for DAS RCN white paper format will have an output (for use in global DAS recording month data conversions prior to PubDAS posting), and maybe possibly SEG-Y (for general interface with other seismic array processing software)

• Recording link to be posted here after zoom cloud processing is complete.

[eileenrmartin]

Here is the zoom recording link:
https://mines.zoom.us/rec/share/iiJQBsXAgCHpRaVQ2nEx0Kxr7_C_KspRgd17Tlntzi_UGw4cJCzoGA4UdHk9TzkF.3QRJawNxTzOQIdLD
Passcode: t?hCVL&8

[eileenrmartin]

It looks like the timing between my dentist appointment and today's DASDAE/DASCore office hours is a little too tight for me to make it out to Golden in time. So, if you'd like to meet about anything between 11 am - noon today, send me an email at eileenrmartin@mines.edu and we can set up a zoom link to chat. Sorry about this!

[eileenrmartin]

Proposed agenda topics for tomorrow's team check-in:

1. Scrum board updates
2. Finalize software citation (co-authorship) policies (see #79 ) and advice on automation of DOI assignment when releasing?
3. Announcements of upcoming events: Dec. 14 from 11:45 am - 1 pm CT team AGU lunch at Showroom Food Hall in Chicago, Dec. 20 from 8:30 am - 4 pm MT DASDAE code sprint in Green Center room 259 and with a few online check-ins throughout the day
4. To do: each person can mark an item on the scrum board (or add a new item) with your name (or names if working in a team) for the task they plan to work on during the code sprint

What other items need to be discussed?

[eileenrmartin]

Recording from today's discussion:
https://mines.zoom.us/rec/share/8i_NofAIvOAK7RTYrVJu2PW6X2KZa1oHldoGW_Ate7Of5suVY6X_zBbQczHgbLIP.igCccYnOnghfHvBK
Passcode: CH70+RZ.

[d-chambers]

Notes for 6/12/2023 meeting

• Eileen: Going to restart meetings, purpose is regular updates, questions, etc.
• Goals:
  • Development plan for the code base
  • Develop office hours (maybe for now just developer meetings, but consider for future). Nicholas suggests from experience to promote a way to book meetings rather than office hours (has set up a page). Also a way for users to have a discussion/forum page, so that there a user community.
    - This will also allow people to see if others have seen bugs/problems so they don't get over reported
  • Use developer meeting for curating list of bugs etc.
  • Review the cards that are pulled and make sure they are being addressed.

Review of the tutorial:

• Use learned timing of the tutorial to better set curriculum times for future events

• Hands on went well, with multiple project options

• Feedback session was valuable and will be compiled and addressed

• EarthScope discussion: moving to AWS and tileDB formats. Derrick has opened discussion on DASCore for this implementation.

  • tileDB is likely an important tool for DASDAE (and the metadata with it)

• Need a better tutorial on DASCORE webpage. Recipes and examples will be important part of this. Some people showed interest in adding examples to the repository.

  • have some more complicated or superfluous examples be external and part of a 'gallery' so we don't add too much complexity to the DASCORE documentation build.
  • Also include a list of Python required skills needed before going into the next workshop/tutorial. This could be distributed in advance to participants can play/learn in advance. Previous users of ObsPy were well prepared/adapted for this.

• Consider what conferences/workshops people should be presenting at. (SEG/AGU/etc. EAGE will require different funds)

  • HPS workshop (Denver this fall?) will also help us develop better tutorials/examples

• For this bi-weekly meeting we should set some technical time each meeting to keep on track with current projects.

• External IO package: how to organize the files and support external files. Donglin says we need to know the structure of the data (e.g., currently it is just the binary with a simple header in the file, and can read only header rather than all data). Derrick and Donglin will set up an example. Many DTS datasets/ for example, are in different formats (Excel, CSV, etc).

• Planning Tier structure for packages:

• Tier 1: flagship packages with high standard of testing and documentation, DASCore and closely related, QC, visualization (Qtgraph as likely dependency), DASClient (e.g. Globus support, AWS support with TileDB)

• Tier 2: slightly higher standard than 3, could be application specific or more specialized processing, some testing and documentation, examples we'll build will be EQ hazards, urban geophysics and near surface package

• Tier 3: anyone (e.g. external I/O) using DASCore data structures

• How to manage: DASDAE will be a meta package, which can install selected tiers (tier 1 only by default, or maybe bare bones tier 1), there will be a csv for all of DASDAE with (for each package): name, tier (1/2/3), patch namespaces, spool namespaces, etc...

[eileenrmartin]

Notes from July 10, 2023 meeting:

• overview of recent updates

  • patch refactor branch with added support for units
  • treble refactoring for performance improvement (not yet released, to keep testing from master branch, and need testers - Shihao to help through development version)
  • more people trying to plot array setups (inventory class needed and integration with visuals - Eileen)
  • Ahmad and dDerrick including Silixa h5 format, and fixed bug in chunk function

• Overview from Yiyu Ni (UW), DAS Store cloud storage with chunked storage , with examle https://colab.research.google.com/drive/19tY6DFhOC3_eWjV7e5j-WygGw63bjodP#scrollTo=f06bdf61

• discussion of Zarr - possible suport through NoisePy, so could be promising for DASCore to support, more geared towards on prem HPC than cloud

• update from Matt Mendoza on Port Angeles WA data

  • north to south event on Cascadia subduction zone, first 9 days of recording during tremor (so don't use these for open examples early on)
  • a few examples of earthquakes (not tremor) and a nearby stream, and maybe airplanes (see https://www.flightradar24.com/airport/clm/arrivals)
  • release data through pubdas (after 1dst 9 days), then do aws open data application for tiledb support examples

• To invite: Shawn Boltz (NIOSH)

• Potentially Derrick to present at a conference this year

• Add Yiyu, Matt, Shihao for DASDAE permissions - project planning

[aaronjgirard]

Attendees: Eileen, Aaron, Shihao, Yiyu, Matt, Derrick, Ahmad & Nicholas

[aaronjgirard]

Attendance: Derrick, Shihao, Jin, Matt, Ahmad, Aaron

Lots of changes to master branch. Things should be faster. Will be a release soon.
adding new parser easier, can run tests automatically. Should make it easier to make contributions.
Matt and Shihao have pulled the master branch and it works fine, but have not tested everything.

Shihao is working on a new interrogator (Sentek). There is a different patch and spool infrastructure, so he want's to know if anyone has implemented anything like this. Derrick mentions that the first thing to check is the "contributing" section of the documentation about adding a new format. The idea will be to implement those four methods (version, etc). The TDES module and the IO module.
Can also read header information to seek rather than read all.

Derrick is trying to get in touch with Alexei to learn more about headers in data formats for HDF5. For each implementation for HDF5 each attribute needs to be handles differently.

Jin has been working on patch processing and visualization.
- patch downsampling and pre-processing functions (e.g, velocity to strain)
- standard deviation calculation (essentially making a large data set a smaller data set
- includes some date/time processing
- eventually should be merged into DASCORE

Derrick has a discussion for patch processing functions for parallel processing. He has started a discussion (#200) on github.
The open question is if anyone forsees need for more complicated parallel processing than that suggested in this discussion. It seems this is an open question and will wait until later for an answer.

Derrick has gone through some DFT processes ad Aaron is starting some testing and QC. Anyone else should look into it as well:
Go to DASCORE -> development documentation -> Notes -> Fourier transforms
(dascore.quarto.pub/dascore/notes/dft_notes..html)

[eileenrmartin]

DASDAE meeting Aug 7, 2023

Attendees: Eileen Martin, Derrick Chambers, Matt Mendoza, Ahmad Tourei, Ge Jin, Aaron Girard, Shihao Yuan

Agenda:
- update on Port Angeles data
- discussions from Git page

Port Angeles:
- Aug 17 - tap tests Aug 18 and maybe 19
- tap tests Matt Mendoza, Hafiz Issah, Halina Dingo
- possible extra disks needed
- settings changed for lower rate
- avoid first 8 days of recording for public examples
- need to decide on which portions of data and how to share
- currently first batch in /data2 with backup hard drives in Eileen's office
- Matt to send copy of map to Aaron and Shihao

Update on discussions from Github:
- almost to 0.1 version:
- need inventories, more formats
- generally ok with moving to 0.1.0
- plotting question - Shihao - on labels (for 0.1.0 to 0.1.1)
- discussioon of adding link to colab for direct open and edit (or mybinder.org or github code) - Shihao and Aaron to discuss and decide for next meeting - and link from recipes part of webpage
- manuscript led by Derrick in SRL Electronic Seismologist
- Concurrency, parallelism - map_to, put and split methods
- considering memory issues
- custom attribute extraction
- callable or extract list of strings with path
- adding processing on spools
- spool processing as classes, not methods, which allows flexibility to add extra info with the processing strategy, similar to low frequency processing currently
- module of processors
- naming so proc -> patchProc or patch_proc and then have spoolProc or spool_proc
- moving window methods
- should have skip length so not every entry shifted
- look for solutions with windowing and striding (and compare to numba or scython or pandas rolling window) - Ahmad to do tests
- new branch by Derrick on more flexible attributes (more than just time and distance, for transforms e.g.)

[eileenrmartin]

DASDAE developer team meeting Aug 24, 2023
Attendees:
Ana Garcia-Ceballos, Matt Mendoza, Ahmad Tourei, Eileen Martin, Derrick Chambers, Ge Jin, Aaron Girard, Shihao Yuan, Nicholas Danes, Tomas Snyder, Donglin Zhu
• PubDAS testing - DC
• test whether we can use DASCore I/O to read big datasets from that repository
• don't plan to support all global DAS month but target the 6-8 big ones there originally
• modified TDMS (Hafiz to be assigned), some SEGY (note issue with SEGY-SAC license, so instead use SEGYIO, Aaron to take a look), some hdf5 or prodml (we just have v 2 and 2.1, Ahmad to be assigned)
• data currently under /data2/pubdas_samples in lightwave server for those with accounts
• Eileen to send Hafiz and Derrick an example of the tdms reader from Silixa
• Eileen to propose submitting the reader examples to the repository (and possibly in future index files consistent with DASCore)
•
•
• discuss potential new additions to DASCore or other packages (e.g. 2.5D borehole DAS data simulator - PL, low frequency processing and/or rolling mean integration - AT)
• Recommendation forr 2.5D simulator - separate package with the same license, and if it is combined with other packages in the future (related) we would mark original repo/package as depracated and pointing to the new combined package
• Jin already submitted into spool processing package with low pass and downsampling
• Ahmad to do code review of spool processing package, then make a branch with additional features he has worked on and add examples
•
•
•
• dasdae.org with overview of all packages, guidelines - Aaron and Shihao to start this building on the dascore.org framework in quarto
• how to sell dasdae.org from Derrick to Mines - Eileen to verify google domain purchase rules for Mines
•
• template repository creation within the organization - Yida to help Derrick with this
•
•
• update on Port Angeles, WA data examples - AG, SY, MM
• last week Matt, Hafiz, Helena Dingo, went out for tap tests with about 50 locations focused on turning points, and data swap (got 10 TB)
• Note about 1/4 mile of fiber was along busy road / highway, part along a bridge, some evidence of mass movement - to be included in notes
• Comparing erthquake catalog that exists with the data (note - multiple folks encountering some issues with permissions on lightwave, but seem to be resolved with tunnelling and index file change, so could be good to include some notes in the notebooks)
• note: units of phase are a bit odd, but conversion done with factor in manual
• figuring out catalogs to automatically search
• Google Colab or Binder more for training purposes (Shihao and Aaron to take a look at GitHub Code Spaces if limitations are encountered)
•
•
•
•
• discuss paper draft - DC
• SRL Electronic Seismologist or Seismica (open source focused, new journal, nice overleaf template)
• contributors to list what their contribution is for deciding co-authorrship order
•
•
• multiresolution visualization - GJ
• MultiResViewer package created, ability to zoom in interactively, change colorbar, go back, etc..., and generates low-passed downsampled versions at X factor (higher number levels are more coarser)
• Testing: in progress
•
•
• update on TileDB support discussion - DC (need to get MB's email to include him in future meetings) - tabled to include in next meeting

[mbriggs134]

@d-chambers my email is matt.briggs@earthscope.org for the next meeting invite.

[eileenrmartin]

I've got your email added to the invite for the upcoming meetings @mbriggs134

[eileenrmartin]

Developer team meeting notes from Sep. 29

New people joining, start with introduction

• participants: Ahmad Tourei, Matt Mendoza, Ge Jin, Eileen Martin, Aaron Girard, Thomas Cullison (SEP), Derrick Chamber, Matt Briggs (Earthscope, data center), Tomas Snyder

Things to talk about

• what’s needed for dascore for data center
• paper draft
• possible fall coding event

Matt Briggs:

• vision is still in transition
• try to collocate compute resources with data storage
  • to save bandwidth on data transfer
• not finalize DAS format yet, dascore could be part of the pipeline for the deliverable format.
• cloud service: AMS / google cloud
• main goal is to support the researchers
• Eileen: for satellite imaging community, put a limit data and computational budget per user.
• Earthscope is open to new ideas and still learning.
• Earthscope is moving the current dataset to the cloud → AWS
• A potential model: user purchase AWS notes for processing, and access to data storage from Earthquake for free.

Derrick:

• from dascore side we need to make sure to have user-friendly interface for cloud computing.
• How to model metadata, develop metadata prototype.
• Good to set up conversation with Earthscope on metadata discussion.

Eileen

• we need to make sure our processing interface is cloud friendly, e.g., testing memory before processing.

Derrick is leading the draft paper to be submitted.

• is the section approporated?
• Ahmad made comments on the paper.

Derrick to add some task cards to projects

• more visualization
• make FileIO format more flexible

Eileen

• encourage people to submit usage issues to GitHub as discussion or bug report

Jin

• sometimes need to upgrade pytable package after dascore installation

Eileen: recent works

• recent request to be able to stack multiple shots of VSP
• how to handle timestamps if the time is relative instead of absolute
• need to support DASPUB
• tdms format issue - Matt Mendoza
• Ahmad: working on spool for real-time processing
  • issue: reading partial files

Eileen to send out a survey for code sprint time.

Matt and Thomas volunteer to test the tutorials.

[eileenrmartin]

Note: if you're watching the discussion boards and haven't been invited to the developer meetings or gotten the survey for setting a code sprint time, email me at eileenrmartin@mines.edu and I will send you these.

[eileenrmartin]

Oct. 12, 2023
Attendees: Matt Briggs, Matt Mendoza, Ahmad Tourei, Nicholas Danes, Eileen Martin

• Code sprint, exact time location
• - 8 am - noon MT Monday Oct,. 16, hybrid on Teams and GC 200E
•
• Design for attaching spatial coordinates from dataframes (see #277, Hafiz is leading this)
• - push to next time (with Hafiz)

• DASDAE at SSA (tutorial ideally, talk, or poster? multiple people including Derrick, Eileen, Ahmad, Hafiz, Matt lead a tutorial)

• Goal to finalize the DASDAE manuscript for submission before Thanksgiving (Seismica vs SRL Electronic Seismologist section)
• - discuss again next time, did consider use of preprint servers (e.g. Mines library has one) if SRL were submitted to
• - update text following code sprint if needed
•
• Assigning tasks for the code sprint and coming up with smaller projects (indicate these with tags on planning page
• - see planning page
•
• Update on error in spool update mid-transfer (from Ahmad, noted by Jin): unable to reproduce error even mid-transfer, so will need to check with Jin more for reproducing error

[eileenrmartin]

DASDAE developers meeting Oct. 26, 2023

Attendees: Eileen Martin, Derrick Chambers, Ahmad Tourei, Donglin Zhu, Matt Mendoza, Shihao Yuan, Thomas Cullison, Nicholas Danes

• Updates from the code sprint
• Stack - mostly done but no formal testing yet
• Added unit labels to colorbar
• Implementing cross-correlation - almost done need to test
• Reviewed Ariel L.'s code on signal whitening and dispersion - done
• Ariel L did one bit normalization (option in normalize)
• Added Dockerfile - done (will be good to do CI for repeat deployments, Eileen to introduce Nicholas and Rachel Willis so she can show him this)
• Ahmad added recipe on contributing
• Hafiz working on allowing dataframe to add coordinates to patch

• Paper - updates and what's needed to wrap it up
• Journal choice: SRL (electronic seismologist) and Seismica
• Decided on Seismica
• Eileen to talk to Noelle to figure out if optional publication fee/donation is allowed
• DASDAE website
• Use Quarto with DASCore as example
• basic starting point: landing page with packages list, contributors page, community governance
• Community guidelines for governance, list of package types (level of testing, style, dependencies) with links to packages and description of how to cite the package, metapackage organization (maybe later?), lots of links to get started with DASCore, team page (binary of who has contributed with name, affiliation and github handle, later add CI chronjob to check contributors to packages and list those), separate prototype packages from others that are more long-term supported
• Text and figure needs:
• need waterfall and wiggle plot next to each other (Matt)
• Derrick adding additional co-authors

• Spool processing methods to implement (e.g. get, put, and this ties in closely with low frequency processing needs)
• For classes that expect spools as an input (get) and apply this and make a new spool (put)
• Based in part on Jin's existing code for spool processing (abstraction beyond directory structure expected now)
• Might in the future have some specific types for converting to TensorFlow or PyTorch acceptible outputs
• Decision: get in by 0.2 version and change paper example
• For now: use Ahmad's example of MPI4py to break up spool into patches that then get processing (use magic command in cell), note dependency issue

• More flexible hdf5 parsing (push to next meeting)

• Discuss tileDB (next time, if Matt Briggs on)

[eileenrmartin]

Agenda and notes from Nov. 9, 2023 DASDAE developer team meeting:

Attendees: Aaron Girard, Derrick Chambers, Eileen Martin, Ge Jin, Matt Mendoza, Matt Briggs, Chad Trabant, Nicholas Danes, Thomas Cullison

1. More flexible hdf5 parsing

   • Hafiz and Derrick to work on this with solution proposed by Derrick at More flexible HDF5 support (includes h5py) #290

2. Discuss tileDB

   • will plug into the dasclients package
   • Eileen to send Matt a few days of Cascadia data - via Google Drive
   • https://gitlab.com/earthscope/geocrate-group/timeseries

3. Paper status and needs

   • several comments added there, maybe antoher waterfall plot with spectra

4. Code updates

   • RSF output need (request by Maddie Pels) - Aaron taking that on
   • keep thinking on multi interrogator experiments
   • Ahmad working on example that combines use of DASCore with MPI as simple way to get parallelism without changing DASCore itself
   • Hafiz working on dimension issue
   • need to still get addition in code so to_obspy and to_pandas (not just metadata, but full object) are in there
   • Thomas getting in touch with Ahmad on parallelism, and Derrick on to_pandas

5. Got planning board set up with permissions so planning can be publicly viewed (so those who haven't been added with permissions can interact in some way), and after people have their first successful pull request, they can get the permissions added (e.g. by Jin, Eileen or Derrick)

6. DASDAE website

   • Eileen has basic draft of content in github.com/DASDAE/DASDAE/Docs
   • OK to delete code in github.com/DASDAE/DASDAE - yes
   • OK to delete redundant information and replace with links to dasdae.org in dascore.org - yes
   • Eileen and Derrick to follow up for making page go live
   • will be moving over the team meeting discussions from dascore repo to DASDAE repo