A definitive collection of calibration and ingestion parameter data for all platforms.
Switch branches/tags
Clone or download
cdobs Merge pull request #339 from cdobs/master
Post Pioneer Asset Management Push
Latest commit 3b3c887 Dec 7, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
CE01ISSM SUNA comment clean up Dec 1, 2018
CE01ISSP Multiple ingests sheets for Endurance 9 recovered, gliders and CSPPs Nov 30, 2018
CE02SHBP Fixed the header repeating multiple times Feb 23, 2018
CE02SHSM Updates to ingest files for SUNA and other issues Nov 30, 2018
CE02SHSP Multiple ingests sheets for Endurance 9 recovered, gliders and CSPPs Nov 30, 2018
CE04OSBP Fixed the header repeating multiple times Feb 23, 2018
CE04OSPD changed the node to lowercase in filename_mask column Mar 5, 2018
CE04OSPS Fixed the header repeating multiple times Feb 23, 2018
CE04OSSM Updates to ingest files for SUNA and other issues Nov 30, 2018
CE05MOAS-GL247 file update Aug 23, 2018
CE05MOAS-GL311 Endurance glider ADCP filename_mask fix (#308) Aug 30, 2018
CE05MOAS-GL312 Endurance glider ADCP filename_mask fix (#308) Aug 30, 2018
CE05MOAS-GL319 Added CE05MOAS-GL319 R00007 Nov 29, 2018
CE05MOAS-GL320 Endurance gliders (#307) Aug 30, 2018
CE05MOAS-GL326 added Endurance glider deployment 326-00007 and recovery 327-00007 Oct 18, 2018
CE05MOAS-GL327 added Endurance glider deployment 326-00007 and recovery 327-00007 Oct 18, 2018
CE05MOAS-GL381 Endurance glider recovered ingest csvs (#306) Aug 29, 2018
CE05MOAS-GL382 ingest for D00004 Sep 10, 2018
CE05MOAS-GL383 Multiple ingests sheets for Endurance 9 recovered, gliders and CSPPs Nov 30, 2018
CE05MOAS-GL384 Endurance glider recovered ingest csvs (#306) Aug 29, 2018
CE05MOAS-GL386 added Endurance glider deployment 386-00010 Sep 24, 2018
CE06ISSM SUNA comment clean up Dec 1, 2018
CE06ISSP Multiple ingests sheets for Endurance 9 recovered, gliders and CSPPs Nov 30, 2018
CE07SHSM SUNA comment clean up Dec 1, 2018
CE07SHSP Multiple ingests sheets for Endurance 9 recovered, gliders and CSPPs Nov 30, 2018
CE09OSPM Multiple ingests sheets for Endurance 9 recovered, gliders and CSPPs Nov 30, 2018
CE09OSSM Updates for the Washington moorings Nov 30, 2018
CP01CNPM Post Pioneer Asset Management Push Nov 13, 2018
CP01CNSM Post Pioneer Asset Management Push Nov 13, 2018
CP01CNSP updated status & note columns Feb 7, 2018
CP02PMCI Post Pioneer Asset Management Push Nov 13, 2018
CP02PMCO Post Pioneer Asset Management Push Nov 13, 2018
CP02PMUI Post Pioneer Asset Management Push Nov 13, 2018
CP02PMUO Update CP02PMUO/CP02PMUO_D00012_ingest.csv Nov 19, 2018
CP03ISPM Post Pioneer Asset Management Push Nov 13, 2018
CP03ISSM Update CP03ISSM/CP03ISSM_D00009_ingest.csv Nov 14, 2018
CP03ISSP added some relevant Redmine tickets to notes (#321) Sep 19, 2018
CP04OSPM Post Pioneer Asset Management Push Nov 13, 2018
CP04OSSM Post Pioneer Asset Management Push Nov 13, 2018
CP05MOAS-A6263 updated file ADCP note Sep 24, 2018
CP05MOAS-A6264 updated file ADCP note Sep 24, 2018
CP05MOAS-AV00# Remove "version" number from template file names. Aug 5, 2015
CP05MOAS-GL335 CP recovered data available (#283) Jul 30, 2018
CP05MOAS-GL336 Post P11 glider recovery ingest csvs Nov 19, 2018
CP05MOAS-GL339 Post P11 glider recovery ingest csvs Nov 19, 2018
CP05MOAS-GL340 Pioneer raw glider data available (#190) Nov 15, 2017
CP05MOAS-GL374 Removed uframe_route column Nov 2, 2017
CP05MOAS-GL375 Removed uframe_route column Nov 2, 2017
CP05MOAS-GL376 Added CP05MOAS-GL376 R9 ingest csv Oct 9, 2018
CP05MOAS-GL379 removed ADCPA row - telemetered data not to be expected Nov 28, 2017
CP05MOAS-GL380 Post P11 glider recovery ingest csvs Nov 19, 2018
CP05MOAS-GL387 data available for ingest (#285) Aug 2, 2018
CP05MOAS-GL388 Added P11 Glider ingest csvs Nov 9, 2018
CP05MOAS-GL389 Post P11 glider recovery ingest csvs Nov 19, 2018
CP05MOAS-PG564 updates to global and Pioneer gliders (#313) Sep 10, 2018
CP05MOAS-PG583 Pioneer updates (#207) Dec 1, 2017
GA01SUMO removed space at the end of filename_mask (#330) Sep 26, 2018
GA02HYPM GA R00003 new ingest csvs (#233) Feb 2, 2018
GA03FLMA GA R00003 new ingest csvs (#233) Feb 2, 2018
GA03FLMB removed space at the end of filename_mask (#330) Sep 26, 2018
GA05MOAS-GL364 update status (#239) Feb 13, 2018
GA05MOAS-GL470 CP05MOAS and GA05MOAS updates (#185) Nov 9, 2017
GA05MOAS-GL493 Removed uframe_route column Nov 2, 2017
GA05MOAS-GL494 Removed uframe_route column Nov 2, 2017
GA05MOAS-GL495 Removed uframe_route column Nov 2, 2017
GA05MOAS-GL496 Removed uframe_route column Nov 2, 2017
GA05MOAS-GL538 Removed uframe_route column Nov 2, 2017
GA05MOAS-PG562 Removed uframe_route column Nov 2, 2017
GA05MOAS-PG563 Removed uframe_route column Nov 2, 2017
GA05MOAS-PG578 Global PG FLORT updates (#213) Dec 14, 2017
GA05MOAS-PG580 updates to global and Pioneer gliders (#313) Sep 10, 2018
GI01SUMO updated endurance deployments. Jul 23, 2018
GI02HYPM updated files Sep 20, 2018
GI03FLMA GI03FLMA D00005 (#311) Sep 5, 2018
GI03FLMB updated endurance deployments. Jul 23, 2018
GI05MOAS-GL363 updated endurance deployments. Jul 23, 2018
GI05MOAS-GL453 updated endurance deployments. Jul 23, 2018
GI05MOAS-GL469 updated endurance deployments. Jul 23, 2018
GI05MOAS-GL477 Removed uframe_route column Nov 2, 2017
GI05MOAS-GL478 Removed uframe_route column Nov 2, 2017
GI05MOAS-GL484 updated status and note columns with new information Nov 21, 2017
GI05MOAS-GL485 Removed uframe_route column Nov 2, 2017
GI05MOAS-GL486 global glider updates (#186) Nov 10, 2017
GI05MOAS-GL493 Removed uframe_route column Nov 2, 2017
GI05MOAS-GL495 Removed uframe_route column Nov 2, 2017
GI05MOAS-GL559 Removed uframe_route column Nov 2, 2017
GI05MOAS-PG528 Removed uframe_route column Nov 2, 2017
GI05MOAS-PG564 Removed uframe_route column Nov 2, 2017
GI05MOAS-PG577 Removed uframe_route column Nov 2, 2017
GI05MOAS-PG581 Removed uframe_route column Nov 2, 2017
GP02HYPM updated files Sep 20, 2018
GP03FLMA GP R00005 (#312) Sep 10, 2018
GP03FLMB GP R00005 (#312) Sep 10, 2018
GP05MOAS-GL276 GP05MOAS-GL276 updated FLORD driver (#294) Aug 20, 2018
GP05MOAS-GL361 Removed uframe_route column Nov 2, 2017
GP05MOAS-GL362 Removed uframe_route column Nov 2, 2017
GP05MOAS-GL363 GP05MOAS-GL363_D00002 update (#183) Nov 7, 2017
GP05MOAS-GL364 Removed uframe_route column Nov 2, 2017
GP05MOAS-GL365 GA R00003 new ingest csvs (#233) Feb 2, 2018
GP05MOAS-GL453 Removed uframe_route column Nov 2, 2017
GP05MOAS-GL469 GP05MOAS-GL469 new ingest csv (#301) Aug 27, 2018
GP05MOAS-GL523 Removed uframe_route column Nov 2, 2017
GP05MOAS-GL525 Removed uframe_route column Nov 2, 2017
GP05MOAS-GL537 Removed uframe_route column Nov 2, 2017
GP05MOAS-PG514 GP05MOAS-PG514-04-FLORTO000 updated status (#229) Jan 31, 2018
GP05MOAS-PG515 updates Feb 14, 2018
GP05MOAS-PG575 Global PG FLORT updates (#213) Dec 14, 2017
GP05MOAS-PG576 global glider updates (#186) Nov 10, 2017
GS01SUMO G*01SUMO: ingested MOPAK data Apr 9, 2018
GS02HYPM GS02HYPM new ingest csv (#232) Feb 2, 2018
GS03FLMA Reverting status to Expected Sep 13, 2018
GS03FLMB new GS R00003 ingest csvs (#226) Jan 26, 2018
GS05MOAS-GL484 Removed uframe_route column Nov 2, 2017
GS05MOAS-GL485 Removed uframe_route column Nov 2, 2017
GS05MOAS-GL486 Removed uframe_route column Nov 2, 2017
GS05MOAS-GL524 Removed uframe_route column Nov 2, 2017
GS05MOAS-GL560 Removed uframe_route column Nov 2, 2017
GS05MOAS-GL561 Removed uframe_route column Nov 2, 2017
GS05MOAS-PG565 global glider updates (#186) Nov 10, 2017
GS05MOAS-PG566 global glider updates (#186) Nov 10, 2017
RS01SBPD fixed filename_masks RS01SBPD (#325) Sep 21, 2018
RS01SBPS Fixed the header repeating multiple times Feb 23, 2018
RS01SLBS Fixed the header repeating multiple times Feb 23, 2018
RS01SUM1 Fixed the header repeating multiple times Feb 23, 2018
RS01SUM2 Added commas to the globs Feb 23, 2018
RS03ASHS Added commas to the globs Feb 23, 2018
RS03ASPI Revert "Redmin 4116 PCO2W and PHSEN cal updates for all moorings" Jul 20, 2015
RS03AXBS Fixed the header repeating multiple times Feb 23, 2018
RS03AXPD Removed uframe_route column Nov 2, 2017
RS03AXPS Fixed the header repeating multiple times Feb 23, 2018
RS03CCAL Added commas to the globs Feb 23, 2018
RS03ECAL Added commas to the globs Feb 23, 2018
RS03INT1 Fixed the header repeating multiple times Feb 23, 2018
RS03INT2 Fixed the header repeating multiple times Feb 23, 2018
Tracking Repository Management Aug 25, 2015
calibration Reorganization of files Apr 30, 2015
.gitignore Issue #13103 - Added usage statement to display when bad arguments ar… Jan 16, 2018
README.md Update ReadMe.md Sep 19, 2018
RELEASE_NOTES.md Merged README and RELEASE_NOTES Apr 2, 2018
cabled_drivers_list.txt Added a list of drivers for the cabled playback tool Feb 21, 2018
find_raw_cabled_files.py Minor logic fix. Cleared up some statements Apr 2, 2018
ingest_csv.py Merge updates May 19, 2017
ingest_file.py Merge branch 'rm13103' of https://github.com/renegelinas/ingestion-csvs Jan 16, 2018
ingest_missing.py Made changes based on comments Mar 29, 2018
list_missing_dates.py Made changes based on comments Mar 29, 2018
sort.py Update sort.py to overwrite files. Aug 25, 2015
validate.py Add directories May 6, 2015

README.md

ingestion-csvs

A definitive collection of calibration and ingestion parameter data for all platforms.

The repository is organized by platform. Each platform directory contains two types of data files.

  • Calibration sheets in Excel format.
  • Ingestion parameter sheets in CSV format.

Filenames adhere to the following conventions:

  • Calibration sheets are named Omaha_Cal_Info_PLATFORM_XXXXX.xlsx, where:
    • PLATFORM is the platform name (e.g. CP02PMCI).
    • XXXXX is the five digit deployment number (e.g. 00001).
  • Ingestion parameter sheets are named PLATFORM_SXXXXX_ingest.csv, where:
    • PLATFORM is the platform name.
    • S is the source of data (e.g. D for telemetered, R for recovered).
    • XXXXX is the five-digit deployment number (e.g. 00001).

A Platform Parameters Checklist Excel spreadsheet is also included in the repository to help keep track of the sheets that are available and the sheets that still need to be created.

Ingestion Parameter Sheets (CSV Files)

The ingestion parameter sheets require 4 columns of data:

Column Name Description
uframe_route The name of the ingestion queue that the EDEX application will pull the files from.
filename_mask A filename mask that points to the files to be ingested. Can include UNIX style wildcards (*, ?).
reference_designator The reference designator associated with the files to be ingested.
data_source An indication of whether the data is telemetered or recovered.

Lines can be commented out in the CSV file by placing a pound sign # at the beginning of the line, or at the beginning of any cell in the line. It's recommended to place the pound sign # in the uframe_route column. When a line in the CSV file is commented out, the ingestion script will ignore that line completely.

Extra columns, such as "notes" or "status," can be included in the CSV files. The ingestion script will ignore these columns completely. Feel free to annotate CSV files with extra columns of data as necessary, but entries in the "status" column should use a standard set of codes, using the definitions below:

Status Description
Not Deployed Instrument was not integrated onto the platform for this deployment.
Not Expected Data are not available for a known reason (e.g. instrument failed, files were corrupted, instrument was lost).
Missing No data in raw data archive, although the instrument was deployed and operational.
Expected Data are scheduled for upload to the raw data archive.
Pending Data are on the raw data archive but not yet able to be ingested (e.g. parser needs to be developed or modified).
Available Data are on the raw data archive and ready for ingestion.

Utilities

ingest_csv.py

Ingest all files described in one or more ingestion csv files.

This allows a simple way to ingest a large amount of uncabled data by simply providing the list of ingestion CSV files.

Usage: ingest_csv.py csv_glob

For example, to ingest all the uncabled data from the CE02SHSP mooring:

./ingest_csv.py CE02SHSP/*

This uses ingest_file.py to process each file glob as defined in the ingestion csv file.

ingest_file.py

Ingest a set of files for a given reference designator.

This tool is designed to test ingestion for a particular uncabled instrument stream. It can also be used to patch ingestion on production, however Large or continual ingestion should be managed by the ingestion tool currently under design by Mark Steiner.

Usage: ingest_file.py uframe_route reference_designator data_source deployment number filename_mask

Example:

./ingest_file.py Ingest.flort-dj-cspp_recovered CE02SHSP-SP002-07-FLORTJ000 telemetered 2 /omc_data/whoi/OMC/CE02SHSP/R00002/extract/*TRIP.txt

ingest_missing.py

Ingest set of cabled files that have been missed during a previous ingestions. It will scrape all missing dates and then it will perform the playback command.

written by Phil Tran

Usage: ingest_missing.py event_url particle_url server

Example: python ingest_missing.py qpid://guest/guest@uframe-test?queue=Ingest.instrument_events qpid://guest/guest@uframe-test?queue=Ingest.instrument_particles uframe-test

find_raw_cabled_files

Find files associated with the specified cabled reference designator.

written by Phil Tran

find_raw_cabled_files.py reference_designator find_raw_cabled_files.py [-l] reference_designator

The optional -l option will provide the list of files associated with the provided reference designator.

Example:

python find_raw_cabled_files.py CE04OSPS-SF01B-4B-VELPTD106
python find_raw_cabled_files.py -l CE04OSPS-SF01B-4B-VELPTD106

Monitoring

The file ingest queue can be monitored using the qpid_stat.py utility in the ooi-tools repository: