Skip to content

MVO Seisan event database to PyMSEC format

Jump to bottom
Glenn Thompson edited this page Nov 9, 2021 · 2 revisions

Overview

The Montserrat seismic event catalog is stored as a Seisan database, and events were classified (labelled) by Montserrat Volcano Observatory (MVO) generally as:

  • teleseism (D),
  • regional (R),
  • volcano-tectonic earthquake (t),
  • hybrid earthquake (h),
  • long-period earthquake (l), and
  • rockfall (r).

But MVO event classifications are unreliable (I say this as someone who originally classified many of those events while at MVO, 1996-2004), which is the main motivation for attempting supervised learning. Since a Seisan event database (400,000 metadata files in Nordic format plus 400,000 corresponding waveform files, and response metadata & station coordinates) is awkward to work with, we convert it to a more convenient format for Python. Steps are:

  1. Convert Seisan database to ObsPy & pandas friendly formats. Trace IDs are converted to SEED-compliant NSLC format, if necessary. Data QC and optionally instrument correction is performed also. Note that the final product of this should be independent of Seisan and easy to reproduce from other data sources, e.g. CSS3.0 databases. Otherwise we would have to convert those to Seisan first. So the target output is MiniSEED files for event waveform files, CSV files for trace metrics, or Pickle files for both (if not bloated compared to Miniseed files). And PNG files for plots. These should not refer in any way to the index files discussed below.
  2. Compute statistical metrics on each waveform.

1. Convert Seisan database to ObsPy and Pandas friendly formats

This includes:

  1. Converting Seisan waveform (WAV) files to MiniSEED.
  2. Converting Seisan S-files to ObsPy Catalog objects and CSV files. S-files are in Nordic format.
  3. Converting Seisan calibration and RESP files to stationXML.

01_seisandb_get_classifications: generates the standard D, R and L main classes allowed by Seisan, but if VOLCANO.DEF exists, adds the local subclasses contained therein also. Saved as 01_seisandb_classes.csv, which can be hand-edited if needed.

02_seisandb_index: Create five indexes:

  1. 02_index_wavfile2sfile.csv: maps all WAVfiles to their corresponding S-files. This is tricky because WAV files might not have the same time as the S-file. At MVO, we allowed up to a 15-second difference in trigger time between analog and digital seismic networks, though this was likely arbitrary and largely dependent on the PRETRIGGER time window each STA/LTA associator was configured with, which differed on each network and changed through time. Compute and log the gap/overlap (in seconds) compared to previous WAV file time window and attempt to use this information to flag adjacent WAVfiles that might be the same event, generating this as a report. In the current case, we only intend to compute metrics and perform supervised learning WAV files from the digital seismic network. But if we also put WAV files from the analog seismic network in there, we could get a more useful report and also the ASN WAV filenames could help find the corresponding Sfiles for the DSN WAV files too. Also flag WAVfiles that have no corresponding S-file. These may need to be manually registered into the Seisan DB.
  2. 02_index_sfile2wavfile.csv: maps all Sfiles to their corresponding WAVfiles (maximum 2 for MVO), indicating if they exist. And whether phase arrivals, locations, magnitudes and AEF data lines exist. Once generated, check if any WAVfiles occur more than once, generating this as a report, indicating duplicate S-files, and resolve these conflicts before generating the next index. Note: Need to check if 75.19 samples per second is really 75.0 based on trying to line up continuous and triggered data.
  3. 02_index_wavfile_original_traceID.csv: an index of all traceIDs found in WAV files, detailing the earliest and latest occurrence of each traceID and the count, and the sample rate of each, and the correct band code (first character of channel) according to SEED nomenclature. Each time the sample rate changes, or the original NSLC changes but maps to the same SEED-compliant NSLC, the location code should probably change to reflect a different configuration. For example, a possible format is: YYYY-MM-DD, (count for) traceID1, (count for) traceID2, ... This would be a convenient dataframe to work with.
  4. 02_index_sfile_original_traceID.csv: an index of all traceIDs found in S-files (phase arrivals), detailing the earliest and latest occurrence of each traceID and the count, or each day they are found. These are not fully qualified NSLC trace IDs.
  5. 02_index_response_original_traceID.csv: an index of all traceIDs found in instrument response metadata files (in CAL directory), detailing the earliest and latest occurrence of each traceID and the count, or each day they are found. If multiple original traceIDs map to the same SEED-compliant trace ID, but have different instrument responses, they should perhaps be given different location codes. Somehow this needs to work with changes in the wavfile IDs.

With these indices, it should be possible to:

  • see which WAV files have not been registered,
  • see which S-files have no linked or existing WAV files,
  • see which traceIDs are most common for any given time period
  • decide what logic rules are needed to convert traceIDs from their original form in WAV files, S-files and instrument response files into SEED-compliant trace IDs in the Pickle files and CSV files.

Note: 02_index_sfile2wavfile.csv could also track the following, though this may then start to look a lot like the catalog CSV file:

  • the existence of (perhaps using bools since filenames have a fixed pattern):
  • PNG files (e.g. raw/corrected seismic/infrasound channels),
  • PICKLE files,
  • event CSV file,
  • original mainclass and subclass,
  • whether event has been reviewed/reclassified (boolean)
  • columns for each acceptable mainclass and subclass (the latter from volcano_def.csv) and a corresponding percentage probability,
  • weight (a visual judgment of signal clarity),
  • whether an event appears to be real but subclass is unknown, unclassified, or unclassifiable (or 'u' may be an acceptable subclass)
  • whether event needs to be split (boolean),
  • whether event should be deleted (boolean)

03_seisandb_traceid_mapping. translate all traceIDs in the index files. this creates a mapping table in a file called 03_traceID_mapping.csv. This file should be manually checked and can be hand edited before it is applied in later steps.

04_seisancal2stationxml: attempt to convert all Seisan calibration and RESP files in the CAL directory to stationXML. Fix each traceID (using 03_traceID_mapping.csv). Then assemble them into either a single stationXML file, or one per trace id, stored back in the CAL directory. create a new index fixedTraceID_index.csv that maps each fixed TraceID to the corresponding stationXML file.

05_seisandb_wavqc: quality check each WAV file listed in 02_index_wavfile2sfile.csv. Fix each traceID (using 03_traceID_mapping.csv) saving original traceID as Trace.Stats.originalID and compute QC metrics on each trace (record in Trace.stats.metrics). Eliminate traces with quality 0. Trace objects can optionally be saved to Pickle or Miniseed files (see which are bigger), and/or Trace.Stats can be saved to traceCSV files. Save these in RAW directory. Number of original traces and number of good traces should be added as columns to 02_index_wavfile2sfile.csv but now saved as 05_index_wavfile2sfile.csv.

06_seisandb_correct: (optional) attempt to apply instrument corrections from stationXML files to correct waveform files from RAW. Omit traces that cannot be corrected. Update the standard Miniseed metrics. Save Pickle/Miniseed+CSV files into CORRECTED directory.

07_seisandb_plot_and_browse: (optional) create plots of seismic and infrasound traces (time domain, spectra, spectrograms) and save as PNG files, with a Javascript browser. This could be applied to WAV, REA, RAW or CORRECTED directories.

08_seisandb_get_picks: extract any phase pick information from S-files, plus corresponding event location and magnitude data. Correct the trace IDs too. Create an ObsPy Catalog object from this information, and save to a Pickle file or a (set of) QuakeML file(s). Further work: A separate application could be developed to then link each event in the catalog to the corresponding Miniseed or Pickle file, to perform further analysis. For example, this could be a gateway for making PyGMT plots and perhaps a MTUQ workflow.

09_seisandb_get_AEF (only applies to MVO data): extract any AEF data, correct trace IDs to SEED-compliance, and save to CSV files. Further work: Generate similar metrics from the 10_seisandb_ampengfft and compare.

At this point, we have extracted as much information as we can from Seisan.

2. Compute metrics on each waveform

10_seisandb_ampengfft: compute various amplitude, energy and frequency metrics for each waveform event file. This updates Trace.stats.metrics and corresponding Pickle and/or CSV files. Need to add metric here to compute frequency changes.

11_seisandb_detectevents: (optional) detect events within each waveform event file. This updates Trace.stats.metrics and corresponding Pickle and/or CSV files.

File formats

The event-level CSV file could track:

  • Sfile path
  • percentages for each classification
  • weight
  • overall Stream metrics, averaged from trace metrics
  • WAV DSN path
  • WAV ASN path
  • For each trace in each WAV:
  • trace ID
  • trace quality
  • trace weight
  • trace deleted (if quality or weight are zero)
  • other trace metrics

To do

  • Test whether a pickle file is larger than a corresponding Miniseed and pickle file just containing metrics. Or can metrics all be stored in a CSV, using tracemetrics2csv and csv2tracemetrics? I could then wrap this with libseisGT.StreamWrite and libseisGT.StreamRead.
  • Test whether writing each trace to a separate Miniseed file, and loading them again, is significantly slower than saving a whole stream to miniseed and loading it again.
  • Ensure traces are being corrected.
  • Ensure ampengfft and spectrograms are (also?) computed on corrected traces

Clone this wiki locally