MVO Seisan event database to Pickle CSV files

What we will do here is go through the various codes in the MontserratML directory and reconstruct what they do.

1. 00_convert_seisandb_to_csvfiles.ipynb and 00_convert_seisandb_to_csvfiles.py: I believe 00_convert_seisandb_to_csvfiles.py is more recent, from memory and git history. main() is called which calls processSeisanYearMonth() one month at a time. This:

• calls get_sfile_list(),
• loops over each S-file in the returned list, reading it in to an Sfile object, and constructing the paths to the linked WAV-files,
• if there is a linked WAV-file with 'MVO' in the filename, this indicates it comes from the MVO Digital seismic network, so then is called on it, returning the contents in a dict called eventrow.
• parameters 'sfile', 'main class' and 'subclass' are added to this dict, and eventrow is then added to a list of dicts.
• once all S-files for the year/month have been processed, the list of dicts is turned into a DataFrame and saved as a CSV file called reawav_MVOE_YYYYMM.csv file.

More information is given at 00_convert_seisandb_to_csvfiles.py

2. 01_compute_gains.py: This appears to be an attempt to generalize 00_convert_seisandb_to_csvfiles.py to create both uncorrected and corrected pickle files and traceCSV files. Main differences are:

• WAV2picklefile() is replaced by wav2stream(), which reads (and writes to) paths elements picklefile_u and picklefile_c instead of just picklefile. These, and paths elements traceCSVfile_u and traceCSVfile_c are set in wavfile2paths().
• processWAV() is replaced by examineWAV(), which just modifies the products list and replaces WAV2picklefile() with wav2stream().
• processSeisanYearMonth() is replaced with examineSeisanYearMonth(). The same input arguments are used. But much of the code from the former is missing, suggesting it may have been edited more recently.
• main() also appears to have parts missing here that are in 00_convert_seisandb_to_csvfiles.py.

In conclusion, it seems we may wish to consider merging elements of this code into 00_convert_seisandb_to_csvfiles.py, if we want to process uncorrected and corrected waveforms within the same workflow.

3. 02_remove_duplicates.ipynb: Through some bugs in 00_convert_seisandb_to_csvfiles.py (now corrected I believe - I think that is the extra code in main() and processSeisanYearMonth() mentioned in 2 above), I ended up with large numbers of lines from other years/months in the files like reawav_MVOE_YYYYMM.csv. This code eliminates those, and rewrites rea_MVOE_YYYYMM.csv. Algorithm:

• Read each reawav_MVOE_YYYYMM.csv file into DataFrames
• concatenate these DataFrames together
• drop duplicate lines (new piece of code added 2021-11-16, see https://github.com/gthompson/kitchensinkGT/issues/18)
• eliminate lines with non-unique S-file/WAV-file path combinations (they can differ in class, and rounding errors on numeric metrics)
• loop over any lines with matching WAV-file path but different S-file names (files with MVO/DSN and SPN/ASN WAV files)

• determine the correct S-file name
• remove the line(s) with the incorrect S-file names (back to the older code)

• write to reawav_MVOE_all.csv
• loop over all years and months

• select all lines for that specific year/month, and write back to reawav_MVOE_YYYYMM.csv (in DATA/MVO)