diff --git a/README.md b/README.md index 632afc8..c97ff71 100644 --- a/README.md +++ b/README.md @@ -1,39 +1,119 @@ # MIMIC IV to OMOP CDM Conversion # -### What is this repository for? ### -* Quick summary -* Version -* [Learn Markdown](https://bitbucket.org/tutorials/markdowndemo) +The project implements an ETL conversion of MIMIC IV PhysioNet dataset to OMOP CDM format. -### Who do I talk to? ### +* Version 1.0 -* Repo owner or admin -* Other community or team contact -### How to run the conversion ### +### Concepts / Phylosophy ### -* Workflows: ddl, vocabulary_refresh, staging, etl, ut, qa, unload -* It is supposed that the project root (location of this file) is current directory. +The ETL is based on the five steps. +* Create a snapshot of the source data. The snapshot data is stored in staging source tables with prefix "src_". +* Clean source data: filter out rows to be not used, format values, apply some business rules. Create intermediate tables with prefix "lk_" and postfix "clean". +* Map distinct source codes to concepts in vocabulary tables. Create intermediate tables with prefix "lk_" and postfix "concept". + * Custom mapping is implemented in custom concepts generated in vocabulary tables beforehand. +* Join cleaned data and mapped codes. Create intermediate tables with prefix "lk_" and postfix "mapped". +* Distribute mapped data by target cdm tables according to target_domain_id values. -* Run a workflow: - * with local variables: `python scripts/run_workflow.py -c conf/workflow_etl.conf` - * copy "variables" section from file.etlconf - * with global variables: `python scripts/run_workflow.py -e conf/dev.etlconf -c conf/workflow_etl.conf` -* Run explicitly named scripts (space delimited): - `python scripts/run_workflow.py -e conf/dev.etlconf -c conf/workflow_etl.conf etl/etl/cdm_drug_era.sql` -* Run in background: - `nohup python scripts/run_workflow.py -e conf/full.etlconf -c conf/workflow_etl.conf > ../out_full_etl.out &` -* Continue after an error: - `nohup python scripts/run_workflow.py -e conf/full.etlconf -c conf/workflow_etl.conf etl/etl/cdm_observation.sql etl/etl/cdm_observation_period.sql etl/etl/cdm_fact_relationship.sql etl/etl/cdm_condition_era.sql etl/etl/cdm_drug_era.sql etl/etl/cdm_dose_era.sql etl/etl/cdm_cdm_source.sql >> ../out_full_etl.out &` +Intermediate and staging CDM tables have additional working fields like unit_id. Field unit_id is composed during the ETL steps. From right to left: source table name, initial target table name abbreviation, final target table name or abbreviation. For example: unit_id = 'drug.cond.diagnoses_icd' means that the rows in this unit_id belong to Drug_exposure table, inially were prepared for Condition_occurrence table, and its original is source table diagnoses_icd. + +Vocabularies are kept in a separate dataset, and are copied as a part of the snapshot data too. + + +### How to run the conversion ### + +* The ETL process encapsulates the following workflows: ddl, vocabulary_refresh, staging, etl, ut, unload. +* The unload workflow results in creating a final OMOP CDM dataset, which can be analysed with OHDSI tools as Atlas or DQD. + +* How to run ETL end-to-end: + * update config files accordingly + * perform vocabulary_refresh steps if needed (see vocabulary_refresh/README.md) + * set the project root (location of this file) as the current directory + + ` + cd vocabulary_refresh + python vocabulary_refresh.py -s10 + python vocabulary_refresh.py -s20 + python vocabulary_refresh.py -s30 + cd ../ + python scripts/run_workflow.py -e conf/dev.etlconf -c conf/workflow_ddl.conf + python scripts/run_workflow.py -e conf/dev.etlconf -c conf/workflow_staging.conf + python scripts/run_workflow.py -e conf/dev.etlconf -c conf/workflow_etl.conf + python scripts/run_workflow.py -e conf/dev.etlconf -c conf/workflow_ut.conf + python scripts/run_workflow.py -e conf/dev.etlconf -c conf/workflow_metrics.conf + ` +* How to look at UT and Metrics reports + * see metrics dataset name in the corresponding .etlconf file + + ` + -- UT report + SELECT report_starttime, table_id, test_type, field_name + FROM metrics_dataset.report_unit_test + WHERE NOT test_passed + ; + -- Metrics - row count + SELECT * FROM metrics_dataset.me_total ORDER BY table_name; + -- Metrics - person and visit summary + SELECT + category, name, count AS row_count + FROM metrics_dataset.me_persons_visits ORDER BY category, name; + -- Metrics - Mapping rates + SELECT + table_name, concept_field, + count AS rows_mapped, + percent AS percent_mapped, + total AS rows_total + FROM metrics_dataset.me_mapping_rate + ORDER BY table_name, concept_field + ; + -- Metrics - Top 100 Mapped and Unmapped + SELECT + table_name, concept_field, category, source_value, concept_id, concept_name, + count AS row_count, + percent AS rows_percent + FROM metrics_dataset.me_tops_together + ORDER BY table_name, concept_field, category, count DESC; + ` + +* More option to run ETL parts: + * Run a workflow: + * with local variables: `python scripts/run_workflow.py -c conf/workflow_etl.conf` + * copy "variables" section from file.etlconf + * with global variables: `python scripts/run_workflow.py -e conf/dev.etlconf -c conf/workflow_etl.conf` + * Run explicitly named scripts (space delimited): + `python scripts/run_workflow.py -e conf/dev.etlconf -c conf/workflow_etl.conf etl/etl/cdm_drug_era.sql` + * Run in background: + `nohup python scripts/run_workflow.py -e conf/full.etlconf -c conf/workflow_etl.conf > ../out_full_etl.out &` + * Continue after an error: + `nohup python scripts/run_workflow.py -e conf/full.etlconf -c conf/workflow_etl.conf etl/etl/cdm_observation.sql etl/etl/cdm_observation_period.sql etl/etl/cdm_fact_relationship.sql etl/etl/cdm_condition_era.sql etl/etl/cdm_drug_era.sql etl/etl/cdm_dose_era.sql etl/etl/cdm_cdm_source.sql >> ../out_full_etl.out &` ### Change Log (latest first) ### +**2021-02-08** + +* Set version v.1.0 + +* Drug_exposure table + * pharmacy.medication is replacing particular values of prescription.drug + * source value format is changed to COALESCE(pharmacy.medication.selected, prescription.drug) || prescription.prod_strength +* Labevents mapping is replaced with new reviewed version + * vocabulary affected: mimiciv_meas_lab_loinc + * lk_meas_labevents_clean and lk_meas_labevents_mapped are changed accordingly +* Unload for Atlas + * Technical fields unit_id, load_row_id, load_table_id, trace_id are removed from Atlas devoted tables +* Delivery export script + * tables are exported to a single directory and single files. If a table is too large, it is exported to multiple files +* Bugfixes and cleanup +* Real environmental names are replaced with placeholders + + **2021-02-01** -* Waveforms POC-2 (load from folders tree and csv files) +* Waveform POC-2 is created for 4 MIMIC III Waveform files uploaded to the bucket + * iterate through the folders tree, capture metadata and load the CSVs * Bugfixes diff --git a/conf/dev.etlconf b/conf/dev.etlconf index 65566b6..0cfb9fb 100644 --- a/conf/dev.etlconf +++ b/conf/dev.etlconf @@ -3,28 +3,28 @@ "variables": { - "@source_project": "physionet-data", - "@core_dataset": "mimic_demo_core", - "@hosp_dataset": "mimic_demo_hosp", - "@icu_dataset": "mimic_demo_icu", - "@ed_dataset": "mimic_demo_ed", + "@source_project": "source_project...", + "@core_dataset": "core...", + "@hosp_dataset": "hosp...", + "@icu_dataset": "icu...", + "@ed_dataset": "ed...", - "@voc_project": "odysseus-mimic-dev", - "@voc_dataset": "vocabulary_2020_09_11", + "@voc_project": "etl_project...", + "@voc_dataset": "voc...", - "@wf_project": "odysseus-mimic-dev", - "@wf_dataset": "waveform_source_poc", + "@wf_project": "etl_project...", + "@wf_dataset": "wf...", - "@etl_project": "odysseus-mimic-dev", - "@etl_dataset": "mimiciv_demo_cdm_2021_01_20", + "@etl_project": "etl_project...", + "@etl_dataset": "etl...", - "@metrics_project": "odysseus-mimic-dev", - "@metrics_dataset": "mimiciv_demo_metrics_2021_01_20", + "@metrics_project": "etl_project...", + "@metrics_dataset": "metrics...", - "@atlas_project": "odysseus-mimic-dev", - "@atlas_dataset": "mimiciv_demo_202101_cdm_531", + "@atlas_project": "etl_project...", + "@atlas_dataset": "atlas...", - "@waveforms_csv_path": "gs://mimic_iv_to_omop/waveforms/source_data/csv" + "@waveforms_csv_path": "gs://bucket..." }, diff --git a/conf/full.etlconf b/conf/full.etlconf index b79103a..0cfb9fb 100644 --- a/conf/full.etlconf +++ b/conf/full.etlconf @@ -3,28 +3,28 @@ "variables": { - "@source_project": "physionet-data", - "@core_dataset": "mimic_core", - "@hosp_dataset": "mimic_hosp", - "@icu_dataset": "mimic_icu", - "@ed_dataset": "mimic_ed", + "@source_project": "source_project...", + "@core_dataset": "core...", + "@hosp_dataset": "hosp...", + "@icu_dataset": "icu...", + "@ed_dataset": "ed...", - "@voc_project": "odysseus-mimic-dev", - "@voc_dataset": "vocabulary_2020_09_11", + "@voc_project": "etl_project...", + "@voc_dataset": "voc...", - "@wf_project": "odysseus-mimic-dev", - "@wf_dataset": "waveform_source_poc", + "@wf_project": "etl_project...", + "@wf_dataset": "wf...", - "@etl_project": "odysseus-mimic-dev", - "@etl_dataset": "mimiciv_full_cdm_2021_01_31", + "@etl_project": "etl_project...", + "@etl_dataset": "etl...", - "@metrics_project": "odysseus-mimic-dev", - "@metrics_dataset": "mimiciv_full_metrics_2021_01_31", + "@metrics_project": "etl_project...", + "@metrics_dataset": "metrics...", - "@atlas_project": "odysseus-mimic-dev", - "@atlas_dataset": "mimiciv_full_202101_cdm_531", + "@atlas_project": "etl_project...", + "@atlas_dataset": "atlas...", - "@waveforms_csv_path": "gs://mimic_iv_to_omop/waveforms/source_data/csv" + "@waveforms_csv_path": "gs://bucket..." }, @@ -68,6 +68,12 @@ "conf": "workflow_qa.conf" }, + { + "workflow": "metrics", + "comment": "build metrics with metrics_gen scripts", + "type": "sql", + "conf": "workflow_metrics.conf" + }, { "workflow": "gen_scripts", "comment": "automation to generate similar queries for some tasks", diff --git a/custom_mapping_csv/custom_mapping_list.tsv b/custom_mapping_csv/custom_mapping_list.tsv index 5cc7f09..e1a30ac 100644 --- a/custom_mapping_csv/custom_mapping_list.tsv +++ b/custom_mapping_csv/custom_mapping_list.tsv @@ -1,6 +1,6 @@ "file_name" "source_vocabulary_id" "min_concept_id" "max_concept_id" "row_count" "target_domains" "gcpt_mimic_generated.csv" "mimiciv_mimic_generated" 2000000000 2000001000 "all(?)" -"gcpt_meas_lab_loinc.csv" "mimiciv_meas_lab_loinc" 2000001001 2000001173 173 "measurement" +"gcpt_meas_lab_loinc.csv" "mimiciv_meas_lab_loinc" 2000001001 2000001235 235 "measurement" "gcpt_obs_insurance.csv" "mimiciv_obs_insurance" 2000001301 2000001305 5 "observation, Meas Value" "gcpt_per_ethnicity.csv" "mimiciv_per_ethnicity" 2000001401 2000001408 8 "person" "gcpt_obs_marital.csv" "mimiciv_obs_marital" 2000001501 2000001507 7 "observation" diff --git a/custom_mapping_csv/gcpt_meas_lab_loinc.csv b/custom_mapping_csv/gcpt_meas_lab_loinc.csv index 3f83029..6d596aa 100644 --- a/custom_mapping_csv/gcpt_meas_lab_loinc.csv +++ b/custom_mapping_csv/gcpt_meas_lab_loinc.csv @@ -1,165 +1,236 @@ "concept_name","source_concept_id","source_vocabulary_id","source_domain_id","source_concept_class_id","standard_concept","concept_code","valid_start_date","valid_end_date","invalid_reason","target_concept_id","relationship_id","reverese_relationship_id","relationship_valid_start_date","relationship_end_date","invalid_reason_cr" -"Myelocytes",2000001001,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Myelocytes","1970-01-01","2099-12-31",,43055360,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Metamyelocytes",2000001002,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Metamyelocytes","1970-01-01","2099-12-31",,43055361,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Blasts",2000001003,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Blasts","1970-01-01","2099-12-31",,43055362,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Inpatient Hematology/Oncology Smear",2000001004,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Inpatient Hematology/Oncology Smear","1970-01-01","2099-12-31",,3018760,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD34",2000001005,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD34","1970-01-01","2099-12-31",,40763401,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Creatinine, Serum",2000001006,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Creatinine, Serum","1970-01-01","2099-12-31",,3016723,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD56",2000001007,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD56","1970-01-01","2099-12-31",,3005460,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD138",2000001008,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD138","1970-01-01","2099-12-31",,3031831,"Maps to","Mapped from","1970-01-01","2099-12-31", -"AFP, Maternal Screen",2000001009,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"AFP, Maternal Screen","1970-01-01","2099-12-31",,3009306,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Arachadonic Acid",2000001010,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Arachadonic Acid","1970-01-01","2099-12-31",,3008541,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD20",2000001011,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD20","1970-01-01","2099-12-31",,3022878,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD20 %",2000001012,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD20 %","1970-01-01","2099-12-31",,3022878,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD16/56%",2000001013,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD16/56%","1970-01-01","2099-12-31",,3015455,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Urine Creatinine",2000001014,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Urine Creatinine","1970-01-01","2099-12-31",,3017250,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD19 %",2000001015,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD19 %","1970-01-01","2099-12-31",,3016228,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD103",2000001016,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD103","1970-01-01","2099-12-31",,3018791,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Inhibitor Screen",2000001017,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Inhibitor Screen","1970-01-01","2099-12-31",,3031335,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD71",2000001018,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD71","1970-01-01","2099-12-31",,3002153,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Ventilation Rate",2000001019,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Ventilation Rate","1970-01-01","2099-12-31",,3024171,"Maps to","Mapped from","1970-01-01","2099-12-31", -"UE3, Maternal Screen",2000001020,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"UE3, Maternal Screen","1970-01-01","2099-12-31",,3025455,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Lipase, Pleural",2000001021,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Lipase, Pleural","1970-01-01","2099-12-31",,3048752,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Reptilase Time Control",2000001022,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Reptilase Time Control","1970-01-01","2099-12-31",,3011893,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Immunophenotyping",2000001023,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Immunophenotyping","1970-01-01","2099-12-31",,4276031,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD59",2000001024,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD59","1970-01-01","2099-12-31",,3043394,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD16/56 Absolute Count",2000001025,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD16/56 Absolute Count","1970-01-01","2099-12-31",,3020358,"Maps to","Mapped from","1970-01-01","2099-12-31", -"tacroFK",2000001026,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"tacroFK","1970-01-01","2099-12-31",,3026250,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Lupus Anticoagulant",2000001027,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Lupus Anticoagulant","1970-01-01","2099-12-31",,4268621,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD3 Absolute Count",2000001028,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD3 Absolute Count","1970-01-01","2099-12-31",,3011412,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD20 Absolute Count",2000001029,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD20 Absolute Count","1970-01-01","2099-12-31",,3018071,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Reticulocyte, Cellular Hemoglobin",2000001030,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Reticulocyte, Cellular Hemoglobin","1970-01-01","2099-12-31",,46234968,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Temperature",2000001031,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Temperature","1970-01-01","2099-12-31",,3020891,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD5 %",2000001032,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD5 %","1970-01-01","2099-12-31",,3020383,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Acid Phosphatase, Non-Prostatic",2000001033,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Acid Phosphatase, Non-Prostatic","1970-01-01","2099-12-31",,3009383,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Red Blood Cell Fragments",2000001034,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Red Blood Cell Fragments","1970-01-01","2099-12-31",,3019880,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD22",2000001035,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD22","1970-01-01","2099-12-31",,3022703,"Maps to","Mapped from","1970-01-01","2099-12-31", -"25-OH Vitamin D",2000001036,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"25-OH Vitamin D","1970-01-01","2099-12-31",,3020149,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Other",2000001037,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Other","1970-01-01","2099-12-31",,3033206,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Ristocetin",2000001038,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Ristocetin","1970-01-01","2099-12-31",,3018367,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD5 Absolute Count",2000001039,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD5 Absolute Count","1970-01-01","2099-12-31",,3018627,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD57",2000001040,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD57","1970-01-01","2099-12-31",,3004291,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Kappa",2000001041,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Kappa","1970-01-01","2099-12-31",,4005508,"Maps to","Mapped from","1970-01-01","2099-12-31", -"ADP",2000001042,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"ADP","1970-01-01","2099-12-31",,3014543,"Maps to","Mapped from","1970-01-01","2099-12-31", -"WBC Clumps",2000001043,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"WBC Clumps","1970-01-01","2099-12-31",,3043728,"Maps to","Mapped from","1970-01-01","2099-12-31", -"PLASMGN",2000001044,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"PLASMGN","1970-01-01","2099-12-31",,3026834,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Oval Fat Body",2000001045,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Oval Fat Body","1970-01-01","2099-12-31",,3032469,"Maps to","Mapped from","1970-01-01","2099-12-31", -"ACID PHOSPHATASE, PROSTATIC",2000001046,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"ACID PHOSPHATASE, PROSTATIC","1970-01-01","2099-12-31",,3035029,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD16",2000001047,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD16","1970-01-01","2099-12-31",,3010201,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Methotrexate",2000001048,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Methotrexate","1970-01-01","2099-12-31",,3017115,"Maps to","Mapped from","1970-01-01","2099-12-31", -"TdT",2000001049,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"TdT","1970-01-01","2099-12-31",,3039644,"Maps to","Mapped from","1970-01-01","2099-12-31", -"TDT",2000001050,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"TDT","1970-01-01","2099-12-31",,3039644,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Hematocrit, Joint Fluid",2000001051,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Hematocrit, Joint Fluid","1970-01-01","2099-12-31",,42868642,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Epithelial Casts",2000001052,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Epithelial Casts","1970-01-01","2099-12-31",,3024290,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Hematocrit, Pleural",2000001053,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Hematocrit, Pleural","1970-01-01","2099-12-31",,42868643,"Maps to","Mapped from","1970-01-01","2099-12-31", -"FMC-7",2000001054,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"FMC-7","1970-01-01","2099-12-31",,3003047,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD25",2000001055,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD25","1970-01-01","2099-12-31",,3011497,"Maps to","Mapped from","1970-01-01","2099-12-31", -"NRBC",2000001056,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"NRBC","1970-01-01","2099-12-31",,40761514,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Nucleated RBC",2000001057,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Nucleated RBC","1970-01-01","2099-12-31",,40761514,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Required O2",2000001058,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Required O2","1970-01-01","2099-12-31",,3020716,"Maps to","Mapped from","1970-01-01","2099-12-31", -"SPECIMEN TYPE",2000001059,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"SPECIMEN TYPE","1970-01-01","2099-12-31",,4081838,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD38",2000001060,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD38","1970-01-01","2099-12-31",,3019183,"Maps to","Mapped from","1970-01-01","2099-12-31", -"FetalFN",2000001061,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"FetalFN","1970-01-01","2099-12-31",,3036848,"Maps to","Mapped from","1970-01-01","2099-12-31", -"HCG, Maternal Screen",2000001062,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"HCG, Maternal Screen","1970-01-01","2099-12-31",,3038136,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD55",2000001063,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD55","1970-01-01","2099-12-31",,3007166,"Maps to","Mapped from","1970-01-01","2099-12-31", -"RBC Clumps",2000001064,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"RBC Clumps","1970-01-01","2099-12-31",,3033026,"Maps to","Mapped from","1970-01-01","2099-12-31", -"D-Dimer",2000001065,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"D-Dimer","1970-01-01","2099-12-31",,3051714,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD3 %",2000001066,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD3 %","1970-01-01","2099-12-31",,3022533,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD19 Absolute Count",2000001067,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD19 Absolute Count","1970-01-01","2099-12-31",,3010503,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Macro Prolactin",2000001068,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Macro Prolactin","1970-01-01","2099-12-31",,44816844,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Bands",2000001069,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Bands","1970-01-01","2099-12-31",,3018199,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Collagen",2000001070,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Collagen","1970-01-01","2099-12-31",,3007451,"Maps to","Mapped from","1970-01-01","2099-12-31", -"SURFACTANT ALBUMIN RATIO",2000001071,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"SURFACTANT ALBUMIN RATIO","1970-01-01","2099-12-31",,3017439,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Cellular Cast",2000001072,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Cellular Cast","1970-01-01","2099-12-31",,3042529,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Albumin, Joint Fluid",2000001073,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Albumin, Joint Fluid","1970-01-01","2099-12-31",,3025380,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Other Cell",2000001074,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Other Cell","1970-01-01","2099-12-31",,40761572,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Reducing Substances, Urine",2000001075,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Reducing Substances, Urine","1970-01-01","2099-12-31",,3035113,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Epinepherine",2000001076,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Epinepherine","1970-01-01","2099-12-31",,3003882,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Lambda",2000001077,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Lambda","1970-01-01","2099-12-31",,4007662,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Joint Crystals, Number",2000001078,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Joint Crystals, Number","1970-01-01","2099-12-31",,4057589,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Amikacin",2000001079,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Amikacin","1970-01-01","2099-12-31",,3036152,"Maps to","Mapped from","1970-01-01","2099-12-31", -"AF-AFP",2000001080,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"AF-AFP","1970-01-01","2099-12-31",,3010296,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Ventilator",2000001081,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Ventilator","1970-01-01","2099-12-31",,3004921,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Intubated",2000001082,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Intubated","1970-01-01","2099-12-31",,45884415,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Alveolar-arterial Gradient",2000001083,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Alveolar-arterial Gradient","1970-01-01","2099-12-31",,3007913,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Chloride, Whole Blood",2000001085,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Chloride, Whole Blood","1970-01-01","2099-12-31",,3018572,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Free Calcium",2000001086,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Free Calcium","1970-01-01","2099-12-31",,3021119,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Glucose",2000001087,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Glucose","1970-01-01","2099-12-31",,3000483,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Hematocrit, Calculated",2000001088,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Hematocrit, Calculated","1970-01-01","2099-12-31",,3009542,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Hemoglobin",2000001089,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Hemoglobin","1970-01-01","2099-12-31",,3000963,"Maps to","Mapped from","1970-01-01","2099-12-31", -"O2 Flow",2000001090,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"O2 Flow","1970-01-01","2099-12-31",,3005629,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Oxygen",2000001091,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Oxygen","1970-01-01","2099-12-31",,3024882,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Oxygen Saturation",2000001092,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Oxygen Saturation","1970-01-01","2099-12-31",,3013502,"Maps to","Mapped from","1970-01-01","2099-12-31", -"PEEP",2000001093,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"PEEP","1970-01-01","2099-12-31",,3022875,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Sodium, Whole Blood",2000001095,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Sodium, Whole Blood","1970-01-01","2099-12-31",,3000285,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Tidal Volume",2000001096,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Tidal Volume","1970-01-01","2099-12-31",,3012410,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Barbiturate Screen",2000001097,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Barbiturate Screen","1970-01-01","2099-12-31",,3003132,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Benzodiazepine Screen",2000001098,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Benzodiazepine Screen","1970-01-01","2099-12-31",,3031951,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CK-MB Index",2000001099,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CK-MB Index","1970-01-01","2099-12-31",,3016311,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Glomerular filtration rate/1.73 sq M.predicted [Volume Rate/Area] in Serum, Plasma or Blood by Creatinine-based formula (MDRD)",2000001100,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Estimated GFR (MDRD equation)","1970-01-01","2099-12-31",,46236952,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Iron Binding Capacity, Total",2000001101,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Iron Binding Capacity, Total","1970-01-01","2099-12-31",,3021044,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Lipase",2000001102,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Lipase","1970-01-01","2099-12-31",,3004905,"Maps to","Mapped from","1970-01-01","2099-12-31", -"NTproBNP",2000001103,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"NTproBNP","1970-01-01","2099-12-31",,3029187,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Osmolality, Measured",2000001104,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Osmolality, Measured","1970-01-01","2099-12-31",,3008295,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Protein Electrophoresis",2000001105,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Protein Electrophoresis","1970-01-01","2099-12-31",,3004588,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Tricyclic Antidepressant Screen",2000001106,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Tricyclic Antidepressant Screen","1970-01-01","2099-12-31",,3025478,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Total Protein, Pleural",2000001108,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Total Protein, Pleural","1970-01-01","2099-12-31",,3003434,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Amphetamine Screen, Urine",2000001109,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Amphetamine Screen, Urine","1970-01-01","2099-12-31",,3027944,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Benzodiazepine Screen, Urine",2000001110,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Benzodiazepine Screen, Urine","1970-01-01","2099-12-31",,3000764,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Cocaine, Urine",2000001111,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Cocaine, Urine","1970-01-01","2099-12-31",,3016879,"Maps to","Mapped from","1970-01-01","2099-12-31", -"HCG, Urine, Qualitative",2000001112,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"HCG, Urine, Qualitative","1970-01-01","2099-12-31",,3018954,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Length of Urine Collection",2000001113,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Length of Urine Collection","1970-01-01","2099-12-31",,3016750,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Methadone, Urine",2000001114,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Methadone, Urine","1970-01-01","2099-12-31",,3036180,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Opiate Screen, Urine",2000001115,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Opiate Screen, Urine","1970-01-01","2099-12-31",,3027008,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Osmolality, Urine",2000001116,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Osmolality, Urine","1970-01-01","2099-12-31",,3026782,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Urea Nitrogen, Urine",2000001117,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Urea Nitrogen, Urine","1970-01-01","2099-12-31",,3011965,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Macrophage",2000001119,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Macrophage","1970-01-01","2099-12-31",,3041635,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Monocytes",2000001120,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Monocytes","1970-01-01","2099-12-31",,3033483,"Maps to","Mapped from","1970-01-01","2099-12-31", -"RBC, Ascites",2000001122,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"RBC, Ascites","1970-01-01","2099-12-31",,3009613,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Fibrinogen, Functional",2000001123,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Fibrinogen, Functional","1970-01-01","2099-12-31",,3016407,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Granulocyte Count",2000001124,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Granulocyte Count","1970-01-01","2099-12-31",,3035715,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Reticulocyte Count, Absolute",2000001125,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Reticulocyte Count, Absolute","1970-01-01","2099-12-31",,3023520,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Reticulocyte Count, Automated",2000001126,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Reticulocyte Count, Automated","1970-01-01","2099-12-31",,3007124,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Monos",2000001128,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Monos","1970-01-01","2099-12-31",,3043387,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Blood",2000001130,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Blood","1970-01-01","2099-12-31",,3011397,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Eosinophils",2000001131,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Eosinophils","1970-01-01","2099-12-31",,3037579,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Transitional Epithelial Cells",2000001132,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Transitional Epithelial Cells","1970-01-01","2099-12-31",,3035851,"Maps to","Mapped from","1970-01-01","2099-12-31", -"eAG",2000001133,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"eAG","1970-01-01","2099-12-31",,3005131,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Glucose, CSF",2000001134,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Glucose, CSF","1970-01-01","2099-12-31",,3022548,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Total Protein, CSF",2000001135,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Total Protein, CSF","1970-01-01","2099-12-31",,3019473,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Influenza A by PCR",2000001136,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Influenza A by PCR","1970-01-01","2099-12-31",,3044938,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Influenza B by PCR",2000001137,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Influenza B by PCR","1970-01-01","2099-12-31",,3038288,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Oxycodone",2000001138,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Oxycodone","1970-01-01","2099-12-31",,3007534,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Total Nucleated Cells, Ascites",2000001139,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Total Nucleated Cells, Ascites","1970-01-01","2099-12-31",,3041408,"Maps to","Mapped from","1970-01-01","2099-12-31", -"RBC, CSF",2000001140,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"RBC, CSF","1970-01-01","2099-12-31",,3012986,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Total Nucleated Cells, CSF",2000001141,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Total Nucleated Cells, CSF","1970-01-01","2099-12-31",,3041372,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Total Nucleated Cells, Pleural",2000001142,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Total Nucleated Cells, Pleural","1970-01-01","2099-12-31",,3041079,"Maps to","Mapped from","1970-01-01","2099-12-31", -"RDW-SD",2000001143,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"RDW-SD","1970-01-01","2099-12-31",,3002888,"Maps to","Mapped from","1970-01-01","2099-12-31", -"pH",2000001144,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"pH","1970-01-01","2099-12-31",,3010421,"Maps to","Mapped from","1970-01-01","2099-12-31", -"pCO2",2000001145,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"pCO2","1970-01-01","2099-12-31",,3013290,"Maps to","Mapped from","1970-01-01","2099-12-31", -"pO2",2000001146,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"pO2","1970-01-01","2099-12-31",,3027315,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Base Excess",2000001147,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Base Excess","1970-01-01","2099-12-31",,3012501,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Albumin [Mass/volume] in Urine",2000001148,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Albumin, Urine","1970-01-01","2099-12-31",,3012516,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Creatinine [Mass/volume] in Peritoneal fluid",2000001149,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Creatinine, Ascites","1970-01-01","2099-12-31",,3016647,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Lymphocytes/100 leukocytes in Blood",2000001150,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Lymphocytes, Percent","1970-01-01","2099-12-31",,3002030,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Leukocytes [#/volume] in Blood by Automated count",2000001151,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"WBC Count","1970-01-01","2099-12-31",,3000905,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Cholesterol [Mass/volume] in Pleural fluid ",2000001152,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Cholesterol, Pleural","1970-01-01","2099-12-31",,3033364,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Hepatitis A virus IgM Ab [Units/volume] in Serum",2000001153,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Hepatitis A Virus IgM Antibody","1970-01-01","2099-12-31",,3011922,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Cytomegalovirus DNA [Units/volume] (viral load) in Unspecified specimen by Probe with signal amplification",2000001154,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Cytomegalovirus Viral Load","1970-01-01","2099-12-31",,3022489,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Hepatitis B virus core IgM Ab [Units/volume] in Serum",2000001155,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Hepatitis B Core Antibody, IgM","1970-01-01","2099-12-31",,3018806,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Cholesterol in LDL [Moles/volume] in Serum or Plasma",2000001156,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Cholesterol, LDL, Measured","1970-01-01","2099-12-31",,3001308,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Erythrocytes [#/volume] in Pleural fluid",2000001158,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"RBC, Pleural","1970-01-01","2099-12-31",,3028308,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Glucose [Mass/volume] in Pleural fluid",2000001159,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Glucose, Pleural","1970-01-01","2099-12-31",,3003403,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Lymphocytes [#/volume] in Blood",2000001160,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Absolute Lymphocyte Count","1970-01-01","2099-12-31",,3019198,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Erythrocyte shape [Morphology] in Blood",2000001161,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"RBC Morphology","1970-01-01","2099-12-31",,3014029,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Potassium [Moles/volume] in Blood",2000001163,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Potassium, Whole Blood","1970-01-01","2099-12-31",,3005456,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Immature granulocytes [#/volume] in Blood",2000001164,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Immature Granulocytes","1970-01-01","2099-12-31",,3040168,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Protein [Mass/volume] in Peritoneal fluid",2000001165,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Total Protein, Ascites","1970-01-01","2099-12-31",,3002331,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Albumin [Mass/volume] in Peritoneal fluid",2000001166,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Albumin, Ascites","1970-01-01","2099-12-31",,3026692,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Albumin [Mass/volume] in Pleural fluid",2000001167,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Albumin, Pleural","1970-01-01","2099-12-31",,3011544,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Lactate dehydrogenase [Enzymatic activity/volume] in Peritoneal fluid by Pyruvate to lactate reaction",2000001168,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Lactate Dehydrogenase, Ascites","1970-01-01","2099-12-31",,40763080,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Amylase [Enzymatic activity/volume] in Peritoneal fluid",2000001169,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Amylase, Ascites","1970-01-01","2099-12-31",,3008691,"Maps to","Mapped from","1970-01-01","2099-12-31", -"CD3+CD4+ (T4 helper) cells/CD3+CD8+ (T8 suppressor cells) cells [# Ratio] in Blood",2000001170,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD4/CD8 Ratio","1970-01-01","2099-12-31",,40757349,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Creatine [Mass/volume] in 24 hour Urine",2000001171,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"24 hr Creatinine","1970-01-01","2099-12-31",,3049226,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Erythrocytes [#/volume] in Synovial fluid",2000001172,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"RBC, Joint Fluid","1970-01-01","2099-12-31",,3010144,"Maps to","Mapped from","1970-01-01","2099-12-31", -"Clostridioides difficile toxin B tcdB gene [Presence] in Stool by NAA with probe detection",2000001173,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"C. diff PCR","1970-01-01","2099-12-31",,44816714,"Maps to","Mapped from","1970-01-01","2099-12-31", +"24 hr Creatinine|Urine|Chemistry",2000001001,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"24 hr Creatinine|Urine|Chemistry","1970-01-01","2099-12-31",,3049226,"Maps to","Mapped from","1970-01-01","2099-12-31", +"25-OH Vitamin D|Blood|Chemistry",2000001002,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"25-OH Vitamin D|Blood|Chemistry","1970-01-01","2099-12-31",,3020149,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Absolute Lymphocyte Count|Blood|Chemistry",2000001003,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Absolute Lymphocyte Count|Blood|Chemistry","1970-01-01","2099-12-31",,3019198,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Absolute Lymphocyte Count|Blood|Hematology",2000001004,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Absolute Lymphocyte Count|Blood|Hematology","1970-01-01","2099-12-31",,3019198,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Acid Phosphatase, Non-Prostatic|Blood|Chemistry",2000001005,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Acid Phosphatase, Non-Prostatic|Blood|Chemistry","1970-01-01","2099-12-31",,3009383,"Maps to","Mapped from","1970-01-01","2099-12-31", +"ADP|Blood|Hematology",2000001006,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"ADP|Blood|Hematology","1970-01-01","2099-12-31",,3014543,"Maps to","Mapped from","1970-01-01","2099-12-31", +"AFP, Maternal Screen|Blood|Chemistry",2000001007,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"AFP, Maternal Screen|Blood|Chemistry","1970-01-01","2099-12-31",,3009306,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Albumin, Ascites|Ascites|Chemistry",2000001008,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Albumin, Ascites|Ascites|Chemistry","1970-01-01","2099-12-31",,3026692,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Albumin, Joint Fluid|Joint Fluid|Chemistry",2000001009,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Albumin, Joint Fluid|Joint Fluid|Chemistry","1970-01-01","2099-12-31",,3025380,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Albumin, Pleural|Pleural|Chemistry",2000001010,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Albumin, Pleural|Pleural|Chemistry","1970-01-01","2099-12-31",,3011544,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Albumin, Urine|Urine|Chemistry",2000001011,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Albumin, Urine|Urine|Chemistry","1970-01-01","2099-12-31",,3012516,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Alveolar-arterial Gradient|Blood|Blood Gas",2000001012,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Alveolar-arterial Gradient|Blood|Blood Gas","1970-01-01","2099-12-31",,3007913,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Amikacin|Blood|Chemistry",2000001013,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Amikacin|Blood|Chemistry","1970-01-01","2099-12-31",,3036152,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Amphetamine Screen, Urine|Urine|Chemistry",2000001014,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Amphetamine Screen, Urine|Urine|Chemistry","1970-01-01","2099-12-31",,3027944,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Amylase, Ascites|Ascites|Chemistry",2000001015,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Amylase, Ascites|Ascites|Chemistry","1970-01-01","2099-12-31",,3008691,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Arachadonic Acid|Blood|Hematology",2000001016,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Arachadonic Acid|Blood|Hematology","1970-01-01","2099-12-31",,3008541,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Bands|Ascites|Hematology",2000001017,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Bands|Ascites|Hematology","1970-01-01","2099-12-31",,3017161,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Bands|Cerebrospinal Fluid|Hematology",2000001018,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Bands|Cerebrospinal Fluid|Hematology","1970-01-01","2099-12-31",,3007905,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Bands|Joint Fluid|Hematology",2000001019,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Bands|Joint Fluid|Hematology","1970-01-01","2099-12-31",,3042845,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Bands|Other Body Fluid|Hematology",2000001020,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Bands|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3017161,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Bands|Pleural|Hematology",2000001021,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Bands|Pleural|Hematology","1970-01-01","2099-12-31",,3017161,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Barbiturate Screen|Blood|Chemistry",2000001022,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Barbiturate Screen|Blood|Chemistry","1970-01-01","2099-12-31",,3003132,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Base Excess|Blood|Blood Gas",2000001023,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Base Excess|Blood|Blood Gas","1970-01-01","2099-12-31",,3012501,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Base Excess|Fluid|Blood Gas",2000001024,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Base Excess|Fluid|Blood Gas","1970-01-01","2099-12-31",,21490592,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Benzodiazepine Screen, Urine|Urine|Chemistry",2000001025,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Benzodiazepine Screen, Urine|Urine|Chemistry","1970-01-01","2099-12-31",,3000764,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Benzodiazepine Screen|Blood|Chemistry",2000001026,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Benzodiazepine Screen|Blood|Chemistry","1970-01-01","2099-12-31",,3031951,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Blasts|Ascites|Hematology",2000001027,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Blasts|Ascites|Hematology","1970-01-01","2099-12-31",,3046003,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Blasts|Cerebrospinal Fluid|Hematology",2000001028,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Blasts|Cerebrospinal Fluid|Hematology","1970-01-01","2099-12-31",,3025698,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Blasts|Joint Fluid|Hematology",2000001029,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Blasts|Joint Fluid|Hematology","1970-01-01","2099-12-31",,3028907,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Blasts|Other Body Fluid|Hematology",2000001030,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Blasts|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3007708,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Blasts|Pleural|Hematology",2000001031,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Blasts|Pleural|Hematology","1970-01-01","2099-12-31",,3028880,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Blood|Urine|Chemistry",2000001032,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Blood|Urine|Chemistry","1970-01-01","2099-12-31",,3011397,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Blood|Urine|Hematology",2000001033,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Blood|Urine|Hematology","1970-01-01","2099-12-31",,3011397,"Maps to","Mapped from","1970-01-01","2099-12-31", +"C. diff PCR|Other Body Fluid|Chemistry",2000001034,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"C. diff PCR|Other Body Fluid|Chemistry","1970-01-01","2099-12-31",,40764128,"Maps to","Mapped from","1970-01-01","2099-12-31", +"C. diff PCR|Stool|Chemistry",2000001035,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"C. diff PCR|Stool|Chemistry","1970-01-01","2099-12-31",,44816714,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD103|Blood|Hematology",2000001036,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD103|Blood|Hematology","1970-01-01","2099-12-31",,3018791,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD103|Bone Marrow|Hematology",2000001037,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD103|Bone Marrow|Hematology","1970-01-01","2099-12-31",,3029695,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD103|Other Body Fluid|Hematology",2000001038,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD103|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3034064,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD138|Blood|Hematology",2000001039,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD138|Blood|Hematology","1970-01-01","2099-12-31",,3031831,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD138|Bone Marrow|Hematology",2000001040,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD138|Bone Marrow|Hematology","1970-01-01","2099-12-31",,3031244,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD138|Other Body Fluid|Hematology",2000001041,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD138|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3033713,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD16/56 Absolute Count|Blood|Hematology",2000001042,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD16/56 Absolute Count|Blood|Hematology","1970-01-01","2099-12-31",,3020358,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD16/56%|Blood|Hematology",2000001043,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD16/56%|Blood|Hematology","1970-01-01","2099-12-31",,3015455,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD16|Blood|Hematology",2000001044,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD16|Blood|Hematology","1970-01-01","2099-12-31",,3010201,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD16|Bone Marrow|Hematology",2000001045,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD16|Bone Marrow|Hematology","1970-01-01","2099-12-31",,3000109,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD16|Other Body Fluid|Hematology",2000001046,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD16|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3022430,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD19 %|Blood|Hematology",2000001047,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD19 %|Blood|Hematology","1970-01-01","2099-12-31",,3016228,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD19 Absolute Count|Blood|Hematology",2000001048,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD19 Absolute Count|Blood|Hematology","1970-01-01","2099-12-31",,3010503,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD20 %|Blood|Hematology",2000001049,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD20 %|Blood|Hematology","1970-01-01","2099-12-31",,3022878,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD20 Absolute Count|Blood|Hematology",2000001050,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD20 Absolute Count|Blood|Hematology","1970-01-01","2099-12-31",,3018071,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD20|Blood|Hematology",2000001051,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD20|Blood|Hematology","1970-01-01","2099-12-31",,3022878,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD20|Bone Marrow|Hematology",2000001052,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD20|Bone Marrow|Hematology","1970-01-01","2099-12-31",,3013075,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD20|Other Body Fluid|Hematology",2000001053,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD20|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,40760530,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD22|Blood|Hematology",2000001054,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD22|Blood|Hematology","1970-01-01","2099-12-31",,3022703,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD22|Bone Marrow|Hematology",2000001055,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD22|Bone Marrow|Hematology","1970-01-01","2099-12-31",,3027365,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD22|Other Body Fluid|Hematology",2000001056,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD22|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3031560,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD25|Blood|Hematology",2000001057,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD25|Blood|Hematology","1970-01-01","2099-12-31",,3011497,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD25|Bone Marrow|Hematology",2000001058,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD25|Bone Marrow|Hematology","1970-01-01","2099-12-31",,3012205,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD25|Other Body Fluid|Hematology",2000001059,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD25|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3031912,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD3 %|Blood|Hematology",2000001060,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD3 %|Blood|Hematology","1970-01-01","2099-12-31",,3022533,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD3 Absolute Count|Blood|Hematology",2000001061,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD3 Absolute Count|Blood|Hematology","1970-01-01","2099-12-31",,3011412,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD34|Bone Marrow|Hematology",2000001062,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD34|Bone Marrow|Hematology","1970-01-01","2099-12-31",,40760512,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD34|Other Body Fluid|Hematology",2000001063,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD34|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,40763401,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD38|Blood|Hematology",2000001064,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD38|Blood|Hematology","1970-01-01","2099-12-31",,3019183,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD38|Bone Marrow|Hematology",2000001065,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD38|Bone Marrow|Hematology","1970-01-01","2099-12-31",,3030000,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD38|Other Body Fluid|Hematology",2000001066,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD38|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3030804,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD4/CD8 Ratio|Blood|Hematology",2000001067,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD4/CD8 Ratio|Blood|Hematology","1970-01-01","2099-12-31",,40757349,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD4/CD8 Ratio|Other Body Fluid|Hematology",2000001068,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD4/CD8 Ratio|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3014859,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD5 %|Blood|Hematology",2000001069,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD5 %|Blood|Hematology","1970-01-01","2099-12-31",,3020383,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD5 Absolute Count|Blood|Hematology",2000001070,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD5 Absolute Count|Blood|Hematology","1970-01-01","2099-12-31",,3018627,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD55|Blood|Hematology",2000001071,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD55|Blood|Hematology","1970-01-01","2099-12-31",,3007166,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD55|Bone Marrow|Hematology",2000001072,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD55|Bone Marrow|Hematology","1970-01-01","2099-12-31",,3033246,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD55|Other Body Fluid|Hematology",2000001073,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD55|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3043265,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD56|Blood|Hematology",2000001074,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD56|Blood|Hematology","1970-01-01","2099-12-31",,3005460,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD56|Bone Marrow|Hematology",2000001075,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD56|Bone Marrow|Hematology","1970-01-01","2099-12-31",,3010939,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD56|Other Body Fluid|Hematology",2000001076,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD56|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,40760536,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD57|Blood|Hematology",2000001077,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD57|Blood|Hematology","1970-01-01","2099-12-31",,3004291,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD57|Bone Marrow|Hematology",2000001078,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD57|Bone Marrow|Hematology","1970-01-01","2099-12-31",,3014360,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD57|Other Body Fluid|Hematology",2000001079,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD57|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3029617,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD59|Blood|Hematology",2000001080,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD59|Blood|Hematology","1970-01-01","2099-12-31",,3043394,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD59|Bone Marrow|Hematology",2000001081,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD59|Bone Marrow|Hematology","1970-01-01","2099-12-31",,3031007,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD59|Other Body Fluid|Hematology",2000001082,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD59|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3043227,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD71|Blood|Hematology",2000001083,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD71|Blood|Hematology","1970-01-01","2099-12-31",,3002153,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD71|Bone Marrow|Hematology",2000001084,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD71|Bone Marrow|Hematology","1970-01-01","2099-12-31",,3009872,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CD71|Other Body Fluid|Hematology",2000001085,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CD71|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,40760538,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Cellular Cast|Urine|Hematology",2000001086,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Cellular Cast|Urine|Hematology","1970-01-01","2099-12-31",,3042529,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Chloride, Whole Blood|Blood|Blood Gas",2000001087,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Chloride, Whole Blood|Blood|Blood Gas","1970-01-01","2099-12-31",,3018572,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Cholesterol, LDL, Measured|Blood|Chemistry",2000001088,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Cholesterol, LDL, Measured|Blood|Chemistry","1970-01-01","2099-12-31",,3001308,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Cholesterol, Pleural|Pleural|Chemistry",2000001089,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Cholesterol, Pleural|Pleural|Chemistry","1970-01-01","2099-12-31",,3033364,"Maps to","Mapped from","1970-01-01","2099-12-31", +"CK-MB Index|Blood|Chemistry",2000001090,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"CK-MB Index|Blood|Chemistry","1970-01-01","2099-12-31",,3016311,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Cocaine, Urine|Urine|Chemistry",2000001091,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Cocaine, Urine|Urine|Chemistry","1970-01-01","2099-12-31",,3016879,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Collagen|Blood|Hematology",2000001092,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Collagen|Blood|Hematology","1970-01-01","2099-12-31",,3007451,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Creatinine, Ascites|Ascites|Chemistry",2000001093,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Creatinine, Ascites|Ascites|Chemistry","1970-01-01","2099-12-31",,3016647,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Creatinine, Serum|Urine|Chemistry",2000001094,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Creatinine, Serum|Urine|Chemistry","1970-01-01","2099-12-31",,3016723,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Cytomegalovirus Viral Load|Blood|Chemistry",2000001095,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Cytomegalovirus Viral Load|Blood|Chemistry","1970-01-01","2099-12-31",,3012381,"Maps to","Mapped from","1970-01-01","2099-12-31", +"D-Dimer|Blood|Chemistry",2000001096,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"D-Dimer|Blood|Chemistry","1970-01-01","2099-12-31",,3051714,"Maps to","Mapped from","1970-01-01","2099-12-31", +"D-Dimer|Blood|Hematology",2000001097,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"D-Dimer|Blood|Hematology","1970-01-01","2099-12-31",,3051714,"Maps to","Mapped from","1970-01-01","2099-12-31", +"eAG|Blood|Chemistry",2000001098,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"eAG|Blood|Chemistry","1970-01-01","2099-12-31",,3005131,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Eosinophils|Ascites|Hematology",2000001099,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Eosinophils|Ascites|Hematology","1970-01-01","2099-12-31",,3019298,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Eosinophils|Cerebrospinal Fluid|Hematology",2000001100,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Eosinophils|Cerebrospinal Fluid|Hematology","1970-01-01","2099-12-31",,3022640,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Eosinophils|Joint Fluid|Hematology",2000001101,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Eosinophils|Joint Fluid|Hematology","1970-01-01","2099-12-31",,3035611,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Eosinophils|Other Body Fluid|Hematology",2000001102,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Eosinophils|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3021453,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Eosinophils|Pleural|Hematology",2000001103,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Eosinophils|Pleural|Hematology","1970-01-01","2099-12-31",,3001893,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Eosinophils|Urine|Hematology",2000001104,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Eosinophils|Urine|Hematology","1970-01-01","2099-12-31",,3037579,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Epinepherine|Blood|Hematology",2000001105,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Epinepherine|Blood|Hematology","1970-01-01","2099-12-31",,3003882,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Epithelial Casts|Urine|Hematology",2000001106,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Epithelial Casts|Urine|Hematology","1970-01-01","2099-12-31",,3024290,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Estimated GFR (MDRD equation)|Blood|Blood Gas",2000001107,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Estimated GFR (MDRD equation)|Blood|Blood Gas","1970-01-01","2099-12-31",,46236952,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Estimated GFR (MDRD equation)|Blood|Chemistry",2000001108,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Estimated GFR (MDRD equation)|Blood|Chemistry","1970-01-01","2099-12-31",,46236952,"Maps to","Mapped from","1970-01-01","2099-12-31", +"FetalFN|Other Body Fluid|Chemistry",2000001109,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"FetalFN|Other Body Fluid|Chemistry","1970-01-01","2099-12-31",,3036848,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Fibrinogen, Functional|Blood|Hematology",2000001110,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Fibrinogen, Functional|Blood|Hematology","1970-01-01","2099-12-31",,3016407,"Maps to","Mapped from","1970-01-01","2099-12-31", +"FMC-7|Blood|Hematology",2000001111,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"FMC-7|Blood|Hematology","1970-01-01","2099-12-31",,3003047,"Maps to","Mapped from","1970-01-01","2099-12-31", +"FMC-7|Bone Marrow|Hematology",2000001112,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"FMC-7|Bone Marrow|Hematology","1970-01-01","2099-12-31",,3028400,"Maps to","Mapped from","1970-01-01","2099-12-31", +"FMC-7|Other Body Fluid|Hematology",2000001113,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"FMC-7|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,40760540,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Free Calcium|Blood|Blood Gas",2000001114,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Free Calcium|Blood|Blood Gas","1970-01-01","2099-12-31",,3021119,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Free Calcium|Blood|Chemistry",2000001115,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Free Calcium|Blood|Chemistry","1970-01-01","2099-12-31",,3021119,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Glucose, CSF|Cerebrospinal Fluid|Chemistry",2000001116,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Glucose, CSF|Cerebrospinal Fluid|Chemistry","1970-01-01","2099-12-31",,3022548,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Glucose, Pleural|Pleural|Chemistry",2000001117,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Glucose, Pleural|Pleural|Chemistry","1970-01-01","2099-12-31",,3003403,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Glucose|Blood|Blood Gas",2000001118,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Glucose|Blood|Blood Gas","1970-01-01","2099-12-31",,3000483,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Glucose|Blood|Chemistry",2000001119,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Glucose|Blood|Chemistry","1970-01-01","2099-12-31",,3000483,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Glucose|Urine|Chemistry",2000001120,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Glucose|Urine|Chemistry","1970-01-01","2099-12-31",,3024629,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Granulocyte Count|Blood|Hematology",2000001121,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Granulocyte Count|Blood|Hematology","1970-01-01","2099-12-31",,3035715,"Maps to","Mapped from","1970-01-01","2099-12-31", +"HCG, Maternal Screen|Blood|Chemistry",2000001122,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"HCG, Maternal Screen|Blood|Chemistry","1970-01-01","2099-12-31",,3038136,"Maps to","Mapped from","1970-01-01","2099-12-31", +"HCG, Urine, Qualitative|Urine|Chemistry",2000001123,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"HCG, Urine, Qualitative|Urine|Chemistry","1970-01-01","2099-12-31",,3018954,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Hematocrit, Calculated|Blood|Blood Gas",2000001124,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Hematocrit, Calculated|Blood|Blood Gas","1970-01-01","2099-12-31",,3009542,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Hematocrit, Joint Fluid|Joint Fluid|Hematology",2000001125,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Hematocrit, Joint Fluid|Joint Fluid|Hematology","1970-01-01","2099-12-31",,42868642,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Hematocrit, Pleural|Pleural|Hematology",2000001126,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Hematocrit, Pleural|Pleural|Hematology","1970-01-01","2099-12-31",,42868643,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Hemoglobin|Blood|Blood Gas",2000001127,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Hemoglobin|Blood|Blood Gas","1970-01-01","2099-12-31",,3000963,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Hemoglobin|Blood|Chemistry",2000001128,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Hemoglobin|Blood|Chemistry","1970-01-01","2099-12-31",,3000963,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Hepatitis A Virus IgM Antibody|Blood|Chemistry",2000001129,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Hepatitis A Virus IgM Antibody|Blood|Chemistry","1970-01-01","2099-12-31",,3011922,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Hepatitis B Core Antibody, IgM|Blood|Chemistry",2000001130,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Hepatitis B Core Antibody, IgM|Blood|Chemistry","1970-01-01","2099-12-31",,3018806,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Immature Granulocytes|Blood|Hematology",2000001131,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Immature Granulocytes|Blood|Hematology","1970-01-01","2099-12-31",,3040168,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Influenza A by PCR|Other Body Fluid|Chemistry",2000001132,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Influenza A by PCR|Other Body Fluid|Chemistry","1970-01-01","2099-12-31",,3044938,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Influenza B by PCR|Other Body Fluid|Chemistry",2000001133,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Influenza B by PCR|Other Body Fluid|Chemistry","1970-01-01","2099-12-31",,3038288,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Inhibitor Screen|Blood|Hematology",2000001134,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Inhibitor Screen|Blood|Hematology","1970-01-01","2099-12-31",,3031335,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Inpatient Hematology/Oncology Smear|Blood|Hematology",2000001135,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Inpatient Hematology/Oncology Smear|Blood|Hematology","1970-01-01","2099-12-31",,3018760,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Intubated|Blood|Blood Gas",2000001136,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Intubated|Blood|Blood Gas","1970-01-01","2099-12-31",,45884415,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Iron Binding Capacity, Total|Blood|Chemistry",2000001137,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Iron Binding Capacity, Total|Blood|Chemistry","1970-01-01","2099-12-31",,3021044,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Lactate Dehydrogenase, Ascites|Ascites|Chemistry",2000001138,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Lactate Dehydrogenase, Ascites|Ascites|Chemistry","1970-01-01","2099-12-31",,40763080,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Length of Urine Collection|Urine|Chemistry",2000001139,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Length of Urine Collection|Urine|Chemistry","1970-01-01","2099-12-31",,3016750,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Lipase, Pleural|Pleural|Chemistry",2000001140,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Lipase, Pleural|Pleural|Chemistry","1970-01-01","2099-12-31",,3048752,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Lipase|Blood|Chemistry",2000001141,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Lipase|Blood|Chemistry","1970-01-01","2099-12-31",,3004905,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Lymphocytes, Percent|Blood|Hematology",2000001142,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Lymphocytes, Percent|Blood|Hematology","1970-01-01","2099-12-31",,3002030,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Macro Prolactin|Blood|Chemistry",2000001143,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Macro Prolactin|Blood|Chemistry","1970-01-01","2099-12-31",,44816844,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Macrophage|Ascites|Hematology",2000001144,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Macrophage|Ascites|Hematology","1970-01-01","2099-12-31",,3041635,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Macrophage|Cerebrospinal Fluid|Hematology",2000001145,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Macrophage|Cerebrospinal Fluid|Hematology","1970-01-01","2099-12-31",,3037871,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Macrophage|Joint Fluid|Hematology",2000001146,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Macrophage|Joint Fluid|Hematology","1970-01-01","2099-12-31",,3029170,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Macrophage|Other Body Fluid|Hematology",2000001147,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Macrophage|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3005087,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Metamyelocytes|Ascites|Hematology",2000001148,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Metamyelocytes|Ascites|Hematology","1970-01-01","2099-12-31",,3024936,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Metamyelocytes|Blood|Hematology",2000001149,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Metamyelocytes|Blood|Hematology","1970-01-01","2099-12-31",,3002179,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Metamyelocytes|Cerebrospinal Fluid|Hematology",2000001150,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Metamyelocytes|Cerebrospinal Fluid|Hematology","1970-01-01","2099-12-31",,3009788,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Metamyelocytes|Joint Fluid|Hematology",2000001151,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Metamyelocytes|Joint Fluid|Hematology","1970-01-01","2099-12-31",,3024936,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Metamyelocytes|Other Body Fluid|Hematology",2000001152,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Metamyelocytes|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3024936,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Metamyelocytes|Pleural|Hematology",2000001153,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Metamyelocytes|Pleural|Hematology","1970-01-01","2099-12-31",,3024936,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Methadone, Urine|Urine|Chemistry",2000001154,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Methadone, Urine|Urine|Chemistry","1970-01-01","2099-12-31",,3036180,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Methotrexate|Blood|Chemistry",2000001155,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Methotrexate|Blood|Chemistry","1970-01-01","2099-12-31",,3017115,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Monocytes|Ascites|Hematology",2000001156,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Monocytes|Ascites|Hematology","1970-01-01","2099-12-31",,3033483,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Monocytes|Cerebrospinal Fluid|Hematology",2000001157,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Monocytes|Cerebrospinal Fluid|Hematology","1970-01-01","2099-12-31",,3037577,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Monocytes|Joint Fluid|Hematology",2000001158,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Monocytes|Joint Fluid|Hematology","1970-01-01","2099-12-31",,3009361,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Monos|Other Body Fluid|Hematology",2000001159,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Monos|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3038104,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Monos|Pleural|Hematology",2000001160,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Monos|Pleural|Hematology","1970-01-01","2099-12-31",,3043387,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Myelocytes|Ascites|Hematology",2000001161,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Myelocytes|Ascites|Hematology","1970-01-01","2099-12-31",,3024394,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Myelocytes|Blood|Hematology",2000001162,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Myelocytes|Blood|Hematology","1970-01-01","2099-12-31",,3017181,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Myelocytes|Cerebrospinal Fluid|Hematology",2000001163,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Myelocytes|Cerebrospinal Fluid|Hematology","1970-01-01","2099-12-31",,3021345,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Myelocytes|Joint Fluid|Hematology",2000001164,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Myelocytes|Joint Fluid|Hematology","1970-01-01","2099-12-31",,3024394,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Myelocytes|Other Body Fluid|Hematology",2000001165,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Myelocytes|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3024394,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Myelocytes|Pleural|Hematology",2000001166,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Myelocytes|Pleural|Hematology","1970-01-01","2099-12-31",,3024394,"Maps to","Mapped from","1970-01-01","2099-12-31", +"NRBC|Blood|Hematology",2000001167,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"NRBC|Blood|Hematology","1970-01-01","2099-12-31",,40761514,"Maps to","Mapped from","1970-01-01","2099-12-31", +"NRBC|Cerebrospinal Fluid|Hematology",2000001168,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"NRBC|Cerebrospinal Fluid|Hematology","1970-01-01","2099-12-31",,40765178,"Maps to","Mapped from","1970-01-01","2099-12-31", +"NRBC|Joint Fluid|Hematology",2000001169,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"NRBC|Joint Fluid|Hematology","1970-01-01","2099-12-31",,40771130,"Maps to","Mapped from","1970-01-01","2099-12-31", +"NRBC|Other Body Fluid|Hematology",2000001170,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"NRBC|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,40765177,"Maps to","Mapped from","1970-01-01","2099-12-31", +"NRBC|Pleural|Hematology",2000001171,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"NRBC|Pleural|Hematology","1970-01-01","2099-12-31",,42868644,"Maps to","Mapped from","1970-01-01","2099-12-31", +"NTproBNP|Blood|Chemistry",2000001172,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"NTproBNP|Blood|Chemistry","1970-01-01","2099-12-31",,3029187,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Nucleated RBC|Ascites|Hematology",2000001173,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Nucleated RBC|Ascites|Hematology","1970-01-01","2099-12-31",,42868645,"Maps to","Mapped from","1970-01-01","2099-12-31", +"O2 Flow|Blood|Blood Gas",2000001174,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"O2 Flow|Blood|Blood Gas","1970-01-01","2099-12-31",,3005629,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Opiate Screen, Urine|Urine|Chemistry",2000001175,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Opiate Screen, Urine|Urine|Chemistry","1970-01-01","2099-12-31",,3027008,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Osmolality, Measured|Blood|Chemistry",2000001176,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Osmolality, Measured|Blood|Chemistry","1970-01-01","2099-12-31",,3008295,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Osmolality, Urine|Urine|Blood Gas",2000001177,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Osmolality, Urine|Urine|Blood Gas","1970-01-01","2099-12-31",,3026782,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Osmolality, Urine|Urine|Chemistry",2000001178,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Osmolality, Urine|Urine|Chemistry","1970-01-01","2099-12-31",,3026782,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Other Cell|Other Body Fluid|Hematology",2000001179,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Other Cell|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3023547,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Other|Ascites|Hematology",2000001180,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Other|Ascites|Hematology","1970-01-01","2099-12-31",,3008855,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Other|Cerebrospinal Fluid|Hematology",2000001181,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Other|Cerebrospinal Fluid|Hematology","1970-01-01","2099-12-31",,3022840,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Other|Joint Fluid|Hematology",2000001182,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Other|Joint Fluid|Hematology","1970-01-01","2099-12-31",,3013794,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Other|Pleural|Hematology",2000001183,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Other|Pleural|Hematology","1970-01-01","2099-12-31",,3015094,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Oval Fat Body|Urine|Hematology",2000001184,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Oval Fat Body|Urine|Hematology","1970-01-01","2099-12-31",,3032469,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Oxycodone|Urine|Chemistry",2000001185,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Oxycodone|Urine|Chemistry","1970-01-01","2099-12-31",,3000068,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Oxygen Saturation|Blood|Blood Gas",2000001186,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Oxygen Saturation|Blood|Blood Gas","1970-01-01","2099-12-31",,3013502,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Oxygen|Blood|Blood Gas",2000001187,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Oxygen|Blood|Blood Gas","1970-01-01","2099-12-31",,3024882,"Maps to","Mapped from","1970-01-01","2099-12-31", +"pCO2|Blood|Blood Gas",2000001188,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"pCO2|Blood|Blood Gas","1970-01-01","2099-12-31",,3013290,"Maps to","Mapped from","1970-01-01","2099-12-31", +"pCO2|Fluid|Blood Gas",2000001189,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"pCO2|Fluid|Blood Gas","1970-01-01","2099-12-31",,3027480,"Maps to","Mapped from","1970-01-01","2099-12-31", +"PEEP|Blood|Blood Gas",2000001190,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"PEEP|Blood|Blood Gas","1970-01-01","2099-12-31",,3022875,"Maps to","Mapped from","1970-01-01","2099-12-31", +"pH|Blood|Blood Gas",2000001191,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"pH|Blood|Blood Gas","1970-01-01","2099-12-31",,3010421,"Maps to","Mapped from","1970-01-01","2099-12-31", +"pH|Fluid|Blood Gas",2000001192,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"pH|Fluid|Blood Gas","1970-01-01","2099-12-31",,3018672,"Maps to","Mapped from","1970-01-01","2099-12-31", +"pH|Other Body Fluid|Blood Gas",2000001193,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"pH|Other Body Fluid|Blood Gas","1970-01-01","2099-12-31",,3018672,"Maps to","Mapped from","1970-01-01","2099-12-31", +"pH|Urine|Chemistry",2000001194,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"pH|Urine|Chemistry","1970-01-01","2099-12-31",,3015736,"Maps to","Mapped from","1970-01-01","2099-12-31", +"pO2|Blood|Blood Gas",2000001195,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"pO2|Blood|Blood Gas","1970-01-01","2099-12-31",,3027315,"Maps to","Mapped from","1970-01-01","2099-12-31", +"pO2|Fluid|Blood Gas",2000001196,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"pO2|Fluid|Blood Gas","1970-01-01","2099-12-31",,3010251,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Potassium, Whole Blood|Blood|Blood Gas",2000001197,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Potassium, Whole Blood|Blood|Blood Gas","1970-01-01","2099-12-31",,3005456,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Protein Electrophoresis|Blood|Chemistry",2000001198,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Protein Electrophoresis|Blood|Chemistry","1970-01-01","2099-12-31",,3004588,"Maps to","Mapped from","1970-01-01","2099-12-31", +"RBC Clumps|Urine|Hematology",2000001199,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"RBC Clumps|Urine|Hematology","1970-01-01","2099-12-31",,3033026,"Maps to","Mapped from","1970-01-01","2099-12-31", +"RBC Morphology|Blood|Hematology",2000001200,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"RBC Morphology|Blood|Hematology","1970-01-01","2099-12-31",,3014029,"Maps to","Mapped from","1970-01-01","2099-12-31", +"RBC, Ascites|Ascites|Hematology",2000001201,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"RBC, Ascites|Ascites|Hematology","1970-01-01","2099-12-31",,3009613,"Maps to","Mapped from","1970-01-01","2099-12-31", +"RBC, CSF|Cerebrospinal Fluid|Hematology",2000001202,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"RBC, CSF|Cerebrospinal Fluid|Hematology","1970-01-01","2099-12-31",,3012986,"Maps to","Mapped from","1970-01-01","2099-12-31", +"RBC, Joint Fluid|Joint Fluid|Hematology",2000001203,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"RBC, Joint Fluid|Joint Fluid|Hematology","1970-01-01","2099-12-31",,3010144,"Maps to","Mapped from","1970-01-01","2099-12-31", +"RBC, Pleural|Pleural|Hematology",2000001204,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"RBC, Pleural|Pleural|Hematology","1970-01-01","2099-12-31",,3028308,"Maps to","Mapped from","1970-01-01","2099-12-31", +"RDW-SD|Blood|Hematology",2000001205,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"RDW-SD|Blood|Hematology","1970-01-01","2099-12-31",,3002888,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Red Blood Cell Fragments|Blood|Hematology",2000001206,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Red Blood Cell Fragments|Blood|Hematology","1970-01-01","2099-12-31",,3019880,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Reducing Substances, Urine|Urine|Hematology",2000001207,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Reducing Substances, Urine|Urine|Hematology","1970-01-01","2099-12-31",,3035113,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Reptilase Time Control|Blood|Hematology",2000001208,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Reptilase Time Control|Blood|Hematology","1970-01-01","2099-12-31",,3011893,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Required O2|Blood|Blood Gas",2000001209,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Required O2|Blood|Blood Gas","1970-01-01","2099-12-31",,3020716,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Reticulocyte Count, Absolute|Blood|Hematology",2000001210,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Reticulocyte Count, Absolute|Blood|Hematology","1970-01-01","2099-12-31",,3023520,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Reticulocyte Count, Automated|Blood|Hematology",2000001211,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Reticulocyte Count, Automated|Blood|Hematology","1970-01-01","2099-12-31",,3007124,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Reticulocyte, Cellular Hemoglobin|Blood|Hematology",2000001212,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Reticulocyte, Cellular Hemoglobin|Blood|Hematology","1970-01-01","2099-12-31",,46234968,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Ristocetin|Blood|Hematology",2000001213,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Ristocetin|Blood|Hematology","1970-01-01","2099-12-31",,3018367,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Sodium, Whole Blood|Blood|Blood Gas",2000001214,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Sodium, Whole Blood|Blood|Blood Gas","1970-01-01","2099-12-31",,3000285,"Maps to","Mapped from","1970-01-01","2099-12-31", +"tacroFK|Blood|Chemistry",2000001215,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"tacroFK|Blood|Chemistry","1970-01-01","2099-12-31",,3026250,"Maps to","Mapped from","1970-01-01","2099-12-31", +"TdT|Blood|Hematology",2000001216,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"TdT|Blood|Hematology","1970-01-01","2099-12-31",,3039644,"Maps to","Mapped from","1970-01-01","2099-12-31", +"TdT|Bone Marrow|Hematology",2000001217,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"TdT|Bone Marrow|Hematology","1970-01-01","2099-12-31",,3032522,"Maps to","Mapped from","1970-01-01","2099-12-31", +"TdT|Other Body Fluid|Hematology",2000001218,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"TdT|Other Body Fluid|Hematology","1970-01-01","2099-12-31",,3041975,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Temperature|Blood|Blood Gas",2000001219,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Temperature|Blood|Blood Gas","1970-01-01","2099-12-31",,3020891,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Tidal Volume|Blood|Blood Gas",2000001220,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Tidal Volume|Blood|Blood Gas","1970-01-01","2099-12-31",,3012410,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Total Nucleated Cells, Ascites|Ascites|Hematology",2000001221,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Total Nucleated Cells, Ascites|Ascites|Hematology","1970-01-01","2099-12-31",,3041408,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Total Nucleated Cells, CSF|Cerebrospinal Fluid|Hematology",2000001222,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Total Nucleated Cells, CSF|Cerebrospinal Fluid|Hematology","1970-01-01","2099-12-31",,3041372,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Total Nucleated Cells, Pleural|Pleural|Hematology",2000001223,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Total Nucleated Cells, Pleural|Pleural|Hematology","1970-01-01","2099-12-31",,3041079,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Total Protein, Ascites|Ascites|Chemistry",2000001224,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Total Protein, Ascites|Ascites|Chemistry","1970-01-01","2099-12-31",,3002331,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Total Protein, CSF|Cerebrospinal Fluid|Chemistry",2000001225,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Total Protein, CSF|Cerebrospinal Fluid|Chemistry","1970-01-01","2099-12-31",,3019473,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Total Protein, Pleural|Pleural|Chemistry",2000001226,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Total Protein, Pleural|Pleural|Chemistry","1970-01-01","2099-12-31",,3003434,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Transitional Epithelial Cells|Urine|Hematology",2000001227,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Transitional Epithelial Cells|Urine|Hematology","1970-01-01","2099-12-31",,3035851,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Tricyclic Antidepressant Screen|Blood|Chemistry",2000001228,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Tricyclic Antidepressant Screen|Blood|Chemistry","1970-01-01","2099-12-31",,3025478,"Maps to","Mapped from","1970-01-01","2099-12-31", +"UE3, Maternal Screen|Blood|Chemistry",2000001229,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"UE3, Maternal Screen|Blood|Chemistry","1970-01-01","2099-12-31",,3025455,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Urea Nitrogen, Urine|Urine|Chemistry",2000001230,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Urea Nitrogen, Urine|Urine|Chemistry","1970-01-01","2099-12-31",,3011965,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Urine Creatinine|Urine|Chemistry",2000001231,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Urine Creatinine|Urine|Chemistry","1970-01-01","2099-12-31",,3017250,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Ventilation Rate|Blood|Blood Gas",2000001232,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Ventilation Rate|Blood|Blood Gas","1970-01-01","2099-12-31",,3024171,"Maps to","Mapped from","1970-01-01","2099-12-31", +"Ventilator|Blood|Blood Gas",2000001233,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"Ventilator|Blood|Blood Gas","1970-01-01","2099-12-31",,3004921,"Maps to","Mapped from","1970-01-01","2099-12-31", +"WBC Clumps|Urine|Hematology",2000001234,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"WBC Clumps|Urine|Hematology","1970-01-01","2099-12-31",,3043728,"Maps to","Mapped from","1970-01-01","2099-12-31", +"WBC Count|Blood|Hematology",2000001235,"mimiciv_meas_lab_loinc","Measurement","Lab Test",,"WBC Count|Blood|Hematology","1970-01-01","2099-12-31",,3000905,"Maps to","Mapped from","1970-01-01","2099-12-31", diff --git a/etl/ddl/ddl_cdm_5_3_1.sql b/etl/ddl/ddl_cdm_5_3_1.sql index 694ae31..a58f5c9 100644 --- a/etl/ddl/ddl_cdm_5_3_1.sql +++ b/etl/ddl/ddl_cdm_5_3_1.sql @@ -13,7 +13,7 @@ CREATE OR REPLACE TABLE `@etl_project`.@etl_dataset.cdm_cohort_definition ( definition_type_concept_id INT64 not null, cohort_definition_syntax STRING , subject_concept_id INT64 not null, - cohort_initiation_DATE DATE + cohort_initiation_date DATE ) ; diff --git a/etl/etl/cdm_cdm_source.sql b/etl/etl/cdm_cdm_source.sql index cf5be40..ffa0d55 100644 --- a/etl/etl/cdm_cdm_source.sql +++ b/etl/etl/cdm_cdm_source.sql @@ -46,7 +46,6 @@ SELECT 'MIMIC IV' AS cdm_source_name, 'mimiciv' AS cdm_source_abbreviation, 'PhysioNet' AS cdm_holder, - -- PhysioNet as data owner or Odysseus as who implemented the conversion? CONCAT('MIMIC-IV is a publicly available database of patients ', 'admitted to the Beth Israel Deaconess Medical Center in Boston, MA, USA.') AS source_description, 'https://mimic-iv.mit.edu/docs/' AS source_documentation_reference, diff --git a/etl/etl/lk_drug.sql b/etl/etl/lk_drug.sql index 9524163..f1b8157 100644 --- a/etl/etl/lk_drug.sql +++ b/etl/etl/lk_drug.sql @@ -29,13 +29,13 @@ -- ------------------------------------------------------------------- CREATE OR REPLACE TABLE `@etl_project`.@etl_dataset.lk_prescriptions_clean AS SELECT - -- 'drug:[' || COALESCE(drug, drug_name_poe, drug_name_generic,'') || ']'|| - 'drug:[' || COALESCE(drug,'') || ']'|| - 'prod_strength:[' || COALESCE(prod_strength,'') || ']'|| - 'drug_type:[' || COALESCE(drug_type,'') || ']'|| - -- 'formulary_drug_cd:[' || COALESCE(formulary_drug_cd,'') || ']' || - 'dose_unit_rx:[' || COALESCE(dose_unit_rx,'') || ']' - AS concept_name, + -- -- 'drug:[' || COALESCE(drug, drug_name_poe, drug_name_generic,'') || ']'|| + -- 'drug:[' || COALESCE(drug,'') || ']'|| + -- 'prod_strength:[' || COALESCE(prod_strength,'') || ']'|| + -- 'drug_type:[' || COALESCE(drug_type,'') || ']'|| + -- -- 'formulary_drug_cd:[' || COALESCE(formulary_drug_cd,'') || ']' || + -- 'dose_unit_rx:[' || COALESCE(dose_unit_rx,'') || ']' + -- AS concept_name, src.subject_id AS subject_id, src.hadm_id AS hadm_id, src.dose_val_rx AS dose_val_rx, @@ -48,10 +48,15 @@ SELECT 'NDC' AS ndc_source_vocabulary, src.form_val_disp AS form_val_disp, CAST(REGEXP_EXTRACT(src.form_val_disp, r'[-]?[\d]+[.]?[\d]*') AS FLOAT64) AS quantity, - COALESCE( - -- src.drug, src.drug_name_poe, src.drug_name_generic,'') - src.drug, '') - || ' ' || COALESCE(src.prod_strength, '') AS gcpt_source_code, + -- COALESCE( + -- -- src.drug, src.drug_name_poe, src.drug_name_generic,'') + -- src.drug, '') + -- || ' ' || COALESCE(src.prod_strength, '') + COALESCE(IF(src.drug IN ('Bag', 'Vial', 'Syringe', 'Syringe.', + 'Syringe (Neonatal)', 'Syringe (Chemo)', 'Soln', 'Soln.', + 'Sodium Chloride 0.9% Flush'), + pharm.medication, src.drug), '') || + ' ' || COALESCE(src.prod_strength, '') AS gcpt_source_code, -- medication/drug + prod_strength 'mimiciv_drug_ndc' AS gcpt_source_vocabulary, -- source_code = label src.pharmacy_id AS pharmacy_id, -- @@ -62,8 +67,12 @@ SELECT FROM `@etl_project`.@etl_dataset.src_prescriptions src -- pr +LEFT JOIN + `@etl_project`.@etl_dataset.src_pharmacy pharm + ON src.pharmacy_id = pharm.pharmacy_id WHERE src.starttime IS NOT NULL + AND src.drug IS NOT NULL ; -- ------------------------------------------------------------------- @@ -111,7 +120,7 @@ FROM LEFT JOIN `@etl_project`.@etl_dataset.voc_concept vc ON vc.concept_code = src.gcpt_source_code - AND vc.vocabulary_id = src.gcpt_source_vocabulary + AND vc.vocabulary_id = src.gcpt_source_vocabulary -- mimiciv_drug_ndc LEFT JOIN `@etl_project`.@etl_dataset.voc_concept_relationship vcr ON vc.concept_id = vcr.concept_id_1 @@ -169,7 +178,7 @@ SELECT 38000177 AS type_concept_id, src.quantity AS quantity, COALESCE(vc_route.target_concept_id, 0) AS route_concept_id, - COALESCE(vc_ndc.source_code, vc_gcpt.source_code, src.concept_name) AS source_code, + COALESCE(vc_ndc.source_code, vc_gcpt.source_code, src.gcpt_source_code) AS source_code, COALESCE(vc_ndc.source_concept_id, vc_gcpt.source_concept_id, 0) AS source_concept_id, src.route_source_code AS route_source_code, src.dose_unit_source_code AS dose_unit_source_code, diff --git a/etl/etl/lk_meas_labevents.sql b/etl/etl/lk_meas_labevents.sql index b15b92e..6abcb6e 100644 --- a/etl/etl/lk_meas_labevents.sql +++ b/etl/etl/lk_meas_labevents.sql @@ -36,6 +36,7 @@ -- ------------------------------------------------------------------- -- lk_meas_labevents_clean +-- source_code: label|fluid|category -- ------------------------------------------------------------------- CREATE OR REPLACE TABLE `@etl_project`.@etl_dataset.lk_meas_labevents_clean AS @@ -45,6 +46,13 @@ SELECT src.charttime AS start_datetime, -- measurement_datetime, src.hadm_id AS hadm_id, src.itemid AS itemid, -- dlab.itemid + COALESCE(dlab.loinc_code, + CONCAT(dlab.label, '|', dlab.fluid, '|', dlab.category) + ) AS source_code, + IF(dlab.loinc_code IS NOT NULL, + 'LOINC', + 'mimiciv_meas_lab_loinc' + ) AS source_vocabulary_id, src.value AS value, -- value_source_value REGEXP_EXTRACT(src.value, r'^(\<=|\>=|\>|\<|=|)') AS value_operator, REGEXP_EXTRACT(src.value, r'[-]?[\d]+[.]?[\d]*') AS value_number, -- assume "-0.34 etc" @@ -58,6 +66,9 @@ SELECT src.trace_id AS trace_id FROM `@etl_project`.@etl_dataset.src_labevents src +LEFT JOIN + `@etl_project`.@etl_dataset.src_d_labitems dlab + ON src.itemid = dlab.itemid ; -- ------------------------------------------------------------------- @@ -68,30 +79,31 @@ FROM -- a) see the joins, -- b) all dlab.itemid, all available concepts from LOINC and custom mapped dlab.label -- ------------------------------------------------------------------- +CREATE OR REPLACE TABLE `@etl_project`.@etl_dataset.tmp_labevents_source_code AS +SELECT DISTINCT + source_code, source_vocabulary_id +FROM + `@etl_project`.@etl_dataset.lk_meas_labevents_clean +; CREATE OR REPLACE TABLE `@etl_project`.@etl_dataset.lk_meas_d_labitems_concept AS SELECT - dlab.itemid AS itemid, -- PK - COALESCE(dlab.loinc_code, dlab.label) AS source_code, - vc.domain_id AS source_domain_id, - vc.vocabulary_id AS source_vocabulary_id, - vc.concept_id AS source_concept_id, - vc2.domain_id AS target_domain_id, - vc2.vocabulary_id AS target_vocabulary_id, - vc2.concept_id AS target_concept_id, - dlab.fluid AS fluid, -- - dlab.category AS category -- 'Blood Gas', 'Chemistry', 'Hematology' + dlab.source_code AS source_code, + dlab.source_vocabulary_id AS source_vocabulary_id, + vc.domain_id AS source_domain_id, + vc.concept_id AS source_concept_id, + vc2.vocabulary_id AS target_vocabulary_id, + vc2.domain_id AS target_domain_id, + vc2.concept_id AS target_concept_id FROM - `@etl_project`.@etl_dataset.src_d_labitems dlab + `@etl_project`.@etl_dataset.tmp_labevents_source_code dlab -- double check loinc codes: do we need to use event dates in join? -- and do we need to do any parsing/replacement to match codes? LEFT JOIN `@etl_project`.@etl_dataset.voc_concept vc - ON vc.concept_code = COALESCE(dlab.loinc_code, dlab.label) - AND ( - vc.vocabulary_id = 'LOINC' AND vc.domain_id = 'Measurement' - OR vc.vocabulary_id = 'mimiciv_meas_lab_loinc' - ) + ON vc.concept_code = dlab.source_code + AND vc.vocabulary_id = dlab.source_vocabulary_id + AND vc.domain_id = 'Measurement' LEFT JOIN `@etl_project`.@etl_dataset.voc_concept_relationship vcr ON vc.concept_id = vcr.concept_id_1 @@ -103,6 +115,8 @@ LEFT JOIN AND vc2.invalid_reason IS NULL ; +DROP TABLE IF EXISTS `@etl_project`.@etl_dataset.tmp_labevents_source_code; + -- ------------------------------------------------------------------- -- lk_meas_labevents_hadm_id -- pick additional hadm_id by event start_datetime @@ -143,7 +157,6 @@ SELECT COALESCE(labc.target_concept_id, 0) AS target_concept_id, COALESCE(labc.target_domain_id, labc.source_domain_id, 'Measurement') AS target_domain_id, src.start_datetime AS start_datetime, - labc.category AS category, opc.target_concept_id AS operator_concept_id, src.value_number AS value_as_number, IF(src.valueuom IS NOT NULL, @@ -165,7 +178,8 @@ FROM `@etl_project`.@etl_dataset.lk_meas_labevents_clean src INNER JOIN `@etl_project`.@etl_dataset.lk_meas_d_labitems_concept labc - ON labc.itemid = src.itemid + ON labc.source_code = src.source_code + AND labc.source_vocabulary_id = src.source_vocabulary_id LEFT JOIN `@etl_project`.@etl_dataset.lk_meas_operator_concept opc ON opc.source_code = src.value_operator diff --git a/etl/etl/lk_meas_unit.sql b/etl/etl/lk_meas_unit.sql index 1847207..9c845e7 100644 --- a/etl/etl/lk_meas_unit.sql +++ b/etl/etl/lk_meas_unit.sql @@ -48,10 +48,10 @@ SELECT ORDER BY UPPER(vc.vocabulary_id) ) AS row_num -- for de-duplication FROM - mimiciv_demo_cdm_2021_01_20.voc_concept vc + `@etl_project`.@etl_dataset.voc_concept vc WHERE -- gcpt_lab_unit_to_concept -> mimiciv_meas_unit - vc.vocabulary_id IN ('mimiciv_meas_unit', 'UCUM') + vc.vocabulary_id IN ('UCUM', 'mimiciv_meas_unit', 'mimiciv_meas_wf_unit') AND vc.domain_id = 'Unit' ; diff --git a/etl/staging/st_hosp.sql b/etl/staging/st_hosp.sql index ed85a0b..cfac7fc 100644 --- a/etl/staging/st_hosp.sql +++ b/etl/staging/st_hosp.sql @@ -279,3 +279,27 @@ SELECT FROM `@source_project`.@hosp_dataset.d_micro ; + +-- ------------------------------------------------------------------- +-- src_pharmacy +-- ------------------------------------------------------------------- + +CREATE OR REPLACE TABLE `@etl_project`.@etl_dataset.src_pharmacy AS +SELECT + pharmacy_id AS pharmacy_id, + medication AS medication, + -- hadm_id AS hadm_id, + -- subject_id AS subject_id, + -- starttime AS starttime, + -- stoptime AS stoptime, + -- route AS route, + -- + 'pharmacy' AS load_table_id, + FARM_FINGERPRINT(GENERATE_UUID()) AS load_row_id, + TO_JSON_STRING(STRUCT( + pharmacy_id AS pharmacy_id + )) AS trace_id +FROM + `@source_project`.@hosp_dataset.pharmacy +; + diff --git a/etl/unload/unload_to_atlas_gen.sql b/etl/unload/unload_to_atlas_gen.sql index 9c5cd46..efad56f 100644 --- a/etl/unload/unload_to_atlas_gen.sql +++ b/etl/unload/unload_to_atlas_gen.sql @@ -3,83 +3,568 @@ -- Unload to ATLAS-- Copy Vocabulary tables CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.concept AS -SELECT * FROM `@etl_project`.@etl_dataset.voc_concept; +SELECT + concept_id, + concept_name, + domain_id, + vocabulary_id, + concept_class_id, + standard_concept, + concept_code, + valid_start_DATE, + valid_end_DATE, + invalid_reason +FROM `@etl_project`.@etl_dataset.voc_concept; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.vocabulary AS -SELECT * FROM `@etl_project`.@etl_dataset.voc_vocabulary; +SELECT + vocabulary_id, + vocabulary_name, + vocabulary_reference, + vocabulary_version, + vocabulary_concept_id +FROM `@etl_project`.@etl_dataset.voc_vocabulary; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.domain AS -SELECT * FROM `@etl_project`.@etl_dataset.voc_domain; +SELECT + domain_id, + domain_name, + domain_concept_id +FROM `@etl_project`.@etl_dataset.voc_domain; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.concept_class AS -SELECT * FROM `@etl_project`.@etl_dataset.voc_concept_class; +SELECT + concept_class_id, + concept_class_name, + concept_class_concept_id +FROM `@etl_project`.@etl_dataset.voc_concept_class; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.concept_relationship AS -SELECT * FROM `@etl_project`.@etl_dataset.voc_concept_relationship; +SELECT + concept_id_1, + concept_id_2, + relationship_id, + valid_start_DATE, + valid_end_DATE, + invalid_reason +FROM `@etl_project`.@etl_dataset.voc_concept_relationship; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.relationship AS -SELECT * FROM `@etl_project`.@etl_dataset.voc_relationship; +SELECT + relationship_id, + relationship_name, + is_hierarchical, + defines_ancestry, + reverse_relationship_id, + relationship_concept_id +FROM `@etl_project`.@etl_dataset.voc_relationship; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.concept_synonym AS -SELECT * FROM `@etl_project`.@etl_dataset.voc_concept_synonym; +SELECT + concept_id, + concept_synonym_name, + language_concept_id +FROM `@etl_project`.@etl_dataset.voc_concept_synonym; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.concept_ancestor AS -SELECT * FROM `@etl_project`.@etl_dataset.voc_concept_ancestor; +SELECT + ancestor_concept_id, + descendant_concept_id, + min_levels_of_separation, + max_levels_of_separation +FROM `@etl_project`.@etl_dataset.voc_concept_ancestor; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.source_to_concept_map AS -SELECT * FROM `@etl_project`.@etl_dataset.voc_source_to_concept_map; +SELECT + source_code, + source_concept_id, + source_vocabulary_id, + source_code_description, + target_concept_id, + target_vocabulary_id, + valid_start_DATE, + valid_end_DATE, + invalid_reason +FROM `@etl_project`.@etl_dataset.voc_source_to_concept_map; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.drug_strength AS -SELECT * FROM `@etl_project`.@etl_dataset.voc_drug_strength; +SELECT + drug_concept_id, + ingredient_concept_id, + amount_value, + amount_unit_concept_id, + numerator_value, + numerator_unit_concept_id, + denominator_value, + denominator_unit_concept_id, + box_size, + valid_start_DATE, + valid_end_DATE, + invalid_reason +FROM `@etl_project`.@etl_dataset.voc_drug_strength; + -- Copy CDM tables CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.cohort_definition AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_cohort_definition; +SELECT + cohort_definition_id, + cohort_definition_name, + cohort_definition_description, + definition_type_concept_id, + cohort_definition_syntax, + subject_concept_id, + cohort_initiation_date +FROM `@etl_project`.@etl_dataset.cdm_cohort_definition; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.attribute_definition AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_attribute_definition; +SELECT + attribute_definition_id, + attribute_name, + attribute_description, + attribute_type_concept_id, + attribute_syntax +FROM `@etl_project`.@etl_dataset.cdm_attribute_definition; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.cdm_source AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_cdm_source; +SELECT + cdm_source_name, + cdm_source_abbreviation, + cdm_holder, + source_description, + source_documentation_reference, + cdm_etl_reference, + source_release_date, + cdm_release_date, + cdm_version, + vocabulary_version +FROM `@etl_project`.@etl_dataset.cdm_cdm_source; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.metadata AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_metadata; +SELECT + metadata_concept_id, + metadata_type_concept_id, + name, + value_as_string, + value_as_concept_id, + metadata_date, + metadata_datetime +FROM `@etl_project`.@etl_dataset.cdm_metadata; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.person AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_person; +SELECT + person_id, + gender_concept_id, + year_of_birth, + month_of_birth, + day_of_birth, + birth_datetime, + race_concept_id, + ethnicity_concept_id, + location_id, + provider_id, + care_site_id, + person_source_value, + gender_source_value, + gender_source_concept_id, + race_source_value, + race_source_concept_id, + ethnicity_source_value, + ethnicity_source_concept_id +FROM `@etl_project`.@etl_dataset.cdm_person; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.observation_period AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_observation_period; +SELECT + observation_period_id, + person_id, + observation_period_start_date, + observation_period_end_date, + period_type_concept_id +FROM `@etl_project`.@etl_dataset.cdm_observation_period; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.specimen AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_specimen; +SELECT + specimen_id, + person_id, + specimen_concept_id, + specimen_type_concept_id, + specimen_date, + specimen_datetime, + quantity, + unit_concept_id, + anatomic_site_concept_id, + disease_status_concept_id, + specimen_source_id, + specimen_source_value, + unit_source_value, + anatomic_site_source_value, + disease_status_source_value +FROM `@etl_project`.@etl_dataset.cdm_specimen; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.death AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_death; +SELECT + person_id, + death_date, + death_datetime, + death_type_concept_id, + cause_concept_id, + cause_source_value, + cause_source_concept_id +FROM `@etl_project`.@etl_dataset.cdm_death; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.visit_occurrence AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_visit_occurrence; +SELECT + visit_occurrence_id, + person_id, + visit_concept_id, + visit_start_date, + visit_start_datetime, + visit_end_date, + visit_end_datetime, + visit_type_concept_id, + provider_id, + care_site_id, + visit_source_value, + visit_source_concept_id, + admitting_source_concept_id, + admitting_source_value, + discharge_to_concept_id, + discharge_to_source_value, + preceding_visit_occurrence_id +FROM `@etl_project`.@etl_dataset.cdm_visit_occurrence; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.visit_detail AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_visit_detail; +SELECT + visit_detail_id, + person_id, + visit_detail_concept_id, + visit_detail_start_date, + visit_detail_start_datetime, + visit_detail_end_date, + visit_detail_end_datetime, + visit_detail_type_concept_id, + provider_id, + care_site_id, + admitting_source_concept_id, + discharge_to_concept_id, + preceding_visit_detail_id, + visit_detail_source_value, + visit_detail_source_concept_id, + admitting_source_value, + discharge_to_source_value, + visit_detail_parent_id, + visit_occurrence_id +FROM `@etl_project`.@etl_dataset.cdm_visit_detail; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.procedure_occurrence AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_procedure_occurrence; +SELECT + procedure_occurrence_id, + person_id, + procedure_concept_id, + procedure_date, + procedure_datetime, + procedure_type_concept_id, + modifier_concept_id, + quantity, + provider_id, + visit_occurrence_id, + visit_detail_id, + procedure_source_value, + procedure_source_concept_id, + modifier_source_value +FROM `@etl_project`.@etl_dataset.cdm_procedure_occurrence; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.drug_exposure AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_drug_exposure; +SELECT + drug_exposure_id, + person_id, + drug_concept_id, + drug_exposure_start_date, + drug_exposure_start_datetime, + drug_exposure_end_date, + drug_exposure_end_datetime, + verbatim_end_date, + drug_type_concept_id, + stop_reason, + refills, + quantity, + days_supply, + sig, + route_concept_id, + lot_number, + provider_id, + visit_occurrence_id, + visit_detail_id, + drug_source_value, + drug_source_concept_id, + route_source_value, + dose_unit_source_value +FROM `@etl_project`.@etl_dataset.cdm_drug_exposure; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.device_exposure AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_device_exposure; +SELECT + device_exposure_id, + person_id, + device_concept_id, + device_exposure_start_date, + device_exposure_start_datetime, + device_exposure_end_date, + device_exposure_end_datetime, + device_type_concept_id, + unique_device_id, + quantity, + provider_id, + visit_occurrence_id, + visit_detail_id, + device_source_value, + device_source_concept_id +FROM `@etl_project`.@etl_dataset.cdm_device_exposure; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.condition_occurrence AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_condition_occurrence; +SELECT + condition_occurrence_id, + person_id, + condition_concept_id, + condition_start_date, + condition_start_datetime, + condition_end_date, + condition_end_datetime, + condition_type_concept_id, + stop_reason, + provider_id, + visit_occurrence_id, + visit_detail_id, + condition_source_value, + condition_source_concept_id, + condition_status_source_value, + condition_status_concept_id +FROM `@etl_project`.@etl_dataset.cdm_condition_occurrence; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.measurement AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_measurement; +SELECT + measurement_id, + person_id, + measurement_concept_id, + measurement_date, + measurement_datetime, + measurement_time, + measurement_type_concept_id, + operator_concept_id, + value_as_number, + value_as_concept_id, + unit_concept_id, + range_low, + range_high, + provider_id, + visit_occurrence_id, + visit_detail_id, + measurement_source_value, + measurement_source_concept_id, + unit_source_value, + value_source_value +FROM `@etl_project`.@etl_dataset.cdm_measurement; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.note AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_note; +SELECT + note_id, + person_id, + note_date, + note_datetime, + note_type_concept_id, + note_class_concept_id, + note_title, + note_text, + encoding_concept_id, + language_concept_id, + provider_id, + visit_occurrence_id, + visit_detail_id, + note_source_value +FROM `@etl_project`.@etl_dataset.cdm_note; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.note_nlp AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_note_nlp; +SELECT + note_nlp_id, + note_id, + section_concept_id, + snippet, + offset, + lexical_variant, + note_nlp_concept_id, + note_nlp_source_concept_id, + nlp_system, + nlp_date, + nlp_datetime, + term_exists, + term_temporal, + term_modifiers +FROM `@etl_project`.@etl_dataset.cdm_note_nlp; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.observation AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_observation; +SELECT + observation_id, + person_id, + observation_concept_id, + observation_date, + observation_datetime, + observation_type_concept_id, + value_as_number, + value_as_string, + value_as_concept_id, + qualifier_concept_id, + unit_concept_id, + provider_id, + visit_occurrence_id, + visit_detail_id, + observation_source_value, + observation_source_concept_id, + unit_source_value, + qualifier_source_value +FROM `@etl_project`.@etl_dataset.cdm_observation; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.fact_relationship AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_fact_relationship; +SELECT + domain_concept_id_1, + fact_id_1, + domain_concept_id_2, + fact_id_2, + relationship_concept_id +FROM `@etl_project`.@etl_dataset.cdm_fact_relationship; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.location AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_location; +SELECT + location_id, + address_1, + address_2, + city, + state, + zip, + county, + location_source_value +FROM `@etl_project`.@etl_dataset.cdm_location; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.care_site AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_care_site; +SELECT + care_site_id, + care_site_name, + place_of_service_concept_id, + location_id, + care_site_source_value, + place_of_service_source_value +FROM `@etl_project`.@etl_dataset.cdm_care_site; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.provider AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_provider; +SELECT + provider_id, + provider_name, + npi, + dea, + specialty_concept_id, + care_site_id, + year_of_birth, + gender_concept_id, + provider_source_value, + specialty_source_value, + specialty_source_concept_id, + gender_source_value, + gender_source_concept_id +FROM `@etl_project`.@etl_dataset.cdm_provider; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.payer_plan_period AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_payer_plan_period; +SELECT + payer_plan_period_id, + person_id, + payer_plan_period_start_date, + payer_plan_period_end_date, + payer_concept_id, + payer_source_value, + payer_source_concept_id, + plan_concept_id, + plan_source_value, + plan_source_concept_id, + sponsor_concept_id, + sponsor_source_value, + sponsor_source_concept_id, + family_source_value, + stop_reason_concept_id, + stop_reason_source_value, + stop_reason_source_concept_id +FROM `@etl_project`.@etl_dataset.cdm_payer_plan_period; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.cost AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_cost; +SELECT + cost_id, + cost_event_id, + cost_domain_id, + cost_type_concept_id, + currency_concept_id, + total_charge, + total_cost, + total_paid, + paid_by_payer, + paid_by_patient, + paid_patient_copay, + paid_patient_coinsurance, + paid_patient_deductible, + paid_by_primary, + paid_ingredient_cost, + paid_dispensing_fee, + payer_plan_period_id, + amount_allowed, + revenue_code_concept_id, + revenue_code_source_value, + drg_concept_id, + drg_source_value +FROM `@etl_project`.@etl_dataset.cdm_cost; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.cohort AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_cohort; +SELECT + cohort_definition_id, + subject_id, + cohort_start_date, + cohort_end_date +FROM `@etl_project`.@etl_dataset.cdm_cohort; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.cohort_attribute AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_cohort_attribute; +SELECT + cohort_definition_id, + subject_id, + cohort_start_date, + cohort_end_date, + attribute_definition_id, + value_as_number, + value_as_concept_id +FROM `@etl_project`.@etl_dataset.cdm_cohort_attribute; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.drug_era AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_drug_era; +SELECT + drug_era_id, + person_id, + drug_concept_id, + drug_era_start_date, + drug_era_end_date, + drug_exposure_count, + gap_days +FROM `@etl_project`.@etl_dataset.cdm_drug_era; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.dose_era AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_dose_era; +SELECT + dose_era_id, + person_id, + drug_concept_id, + unit_concept_id, + dose_value, + dose_era_start_date, + dose_era_end_date +FROM `@etl_project`.@etl_dataset.cdm_dose_era; + CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.condition_era AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_condition_era; +SELECT + condition_era_id, + person_id, + condition_concept_id, + condition_era_start_date, + condition_era_end_date, + condition_occurrence_count +FROM `@etl_project`.@etl_dataset.cdm_condition_era; + diff --git a/etl/unload/unload_to_atlas_gen_1112.sql b/etl/unload/unload_to_atlas_gen_1112.sql deleted file mode 100644 index 8f01820..0000000 --- a/etl/unload/unload_to_atlas_gen_1112.sql +++ /dev/null @@ -1,85 +0,0 @@ --- bq_cdm_to_atlas generated script -- - --- Unload to ATLAS-- Copy Vocabulary tables - -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.concept AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.voc_concept; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.vocabulary AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.voc_vocabulary; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.domain AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.voc_domain; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.concept_class AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.voc_concept_class; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.concept_relationship AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.voc_concept_relationship; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.relationship AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.voc_relationship; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.concept_synonym AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.voc_concept_synonym; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.concept_ancestor AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.voc_concept_ancestor; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.source_to_concept_map AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.voc_source_to_concept_map; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.drug_strength AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.voc_drug_strength; - --- Copy CDM tables - -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.cohort_definition AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_cohort_definition; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.attribute_definition AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_attribute_definition; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.cdm_source AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_cdm_source; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.metadata AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_metadata; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.person AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_person; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.observation_period AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_observation_period; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.specimen AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_specimen; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.death AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_death; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.visit_occurrence AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_visit_occurrence; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.visit_detail AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_visit_detail; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.procedure_occurrence AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_procedure_occurrence; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.drug_exposure AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_drug_exposure; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.device_exposure AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_device_exposure; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.condition_occurrence AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_condition_occurrence; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.measurement AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_measurement; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.note AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_note; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.note_nlp AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_note_nlp; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.observation AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_observation; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.fact_relationship AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_fact_relationship; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.location AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_location; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.care_site AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_care_site; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.provider AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_provider; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.payer_plan_period AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_payer_plan_period; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.cost AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_cost; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.cohort AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_cohort; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.cohort_attribute AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_cohort_attribute; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.drug_era AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_drug_era; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.dose_era AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_dose_era; -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciv_202010_cdm_531.condition_era AS -SELECT * FROM `odysseus-mimic-dev`.mimiciv_cdm_tuf_10_ant_2020_10_21.cdm_condition_era; diff --git a/etl/unload/unload_to_atlas_gen_try_fields.sql b/etl/unload/unload_to_atlas_gen_try_fields.sql deleted file mode 100644 index 3add778..0000000 --- a/etl/unload/unload_to_atlas_gen_try_fields.sql +++ /dev/null @@ -1,11 +0,0 @@ --- bq_cdm_to_atlas generated script -- - --- Unload to ATLAS-- Copy Vocabulary tables - -CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.concept AS -SELECT * FROM `@etl_project`.@etl_dataset.voc_concept; - --- Copy CDM tables - -CREATE OR REPLACE TABLE `@atlas_project`.@atlas_dataset.cohort_definition AS -SELECT * FROM `@etl_project`.@etl_dataset.cdm_cohort_definition; diff --git a/scripts/bq_run_script.conf b/scripts/bq_run_script.conf index b4fc639..e244c48 100644 --- a/scripts/bq_run_script.conf +++ b/scripts/bq_run_script.conf @@ -2,14 +2,14 @@ "variables": { - "@bq_target_project": "odysseus-mimic-dev", - "@bq_target_dataset": "mimiciv_cdm_tuf_10_ant_2020_10_21", + "@bq_target_project": "project...", + "@bq_target_dataset": "dataset...", - "@etl_project": "odysseus-mimic-dev", - "@etl_dataset": "mimiciv_cdm_tuf_10_ant_2020_10_21", + "@etl_project": "project...", + "@etl_dataset": "dataset...", - "@source_project": "physionet-data", - "@hosp_dataset": "mimic_demo_hosp" + "@source_project": "project...", + "@hosp_dataset": "dataset..." } diff --git a/scripts/bq_run_script.py b/scripts/bq_run_script.py index e18ca12..94d6df7 100644 --- a/scripts/bq_run_script.py +++ b/scripts/bq_run_script.py @@ -17,9 +17,9 @@ config_default = { "variables": { - - "bq_target_project": "No project replacement by default", - "bq_target_dataset": "No dataset replacement by default" + + "@variable_1": "No project replacement by default", + "@variable_2": "No dataset replacement by default" } } @@ -243,4 +243,4 @@ def main(): print('bq_run_script.exit()', return_code) exit(return_code) -# last edit: 2021-01-20 +# last edit: 2021-02-07 diff --git a/scripts/bq_run_script.pyc b/scripts/bq_run_script.pyc deleted file mode 100644 index bcf73ad..0000000 Binary files a/scripts/bq_run_script.pyc and /dev/null differ diff --git a/scripts/delivery/README.md b/scripts/delivery/README.md index c89f44c..09d35da 100644 --- a/scripts/delivery/README.md +++ b/scripts/delivery/README.md @@ -2,16 +2,26 @@ ###Export And Load OMOP Tables To And From BigQuery### -###Latest updates (recent first):### +* The script `export_from_bq.py` exports BQ tables to the given Google Storage location, and optionally copies the created files to the given local location. +* Tables are exported into a single folder, to single files. If a table is too large, the script tries again and creates multiple files. +* To save files in the bucket only, select 'gs' export type, and to download locally, select 'local' export type. +* Copy parameters from `config_default` inside the script to a json .conf file to override default values. -*2020-12-21* +###Usage:### +`python scripts/delivery/export_from_bq.py -c scripts/delivery/export_from_bq_demo.conf` -###Usage:### +`python scripts/delivery/load_to_bq.py -c scripts/delivery/load_to_bq_demo.conf` -python scripts/delivery/export_from_bq.py scripts/delivery/export_from_bq_demo.conf +###Latest updates (recent first):### + +*2021-02-07* + +* Tables are exported into a single folder, to single files. If a table is too large, the script tries again and creates multiple files. + +*2020-12-21* -python scripts/delivery/load_to_bq.py scripts/delivery/load_to_bq_demo.conf +* This Readme is created. #####Tne End##### diff --git a/scripts/delivery/c.conf b/scripts/delivery/c.conf deleted file mode 100644 index 8c281b7..0000000 --- a/scripts/delivery/c.conf +++ /dev/null @@ -1,7 +0,0 @@ -{ - - "bq_target_project": "my_project", - "bq_target_dataset": "my_dataset", - "csv_path": "somewhere" - -} \ No newline at end of file diff --git a/scripts/delivery/cdm_schemas/condition_occurrence.bak.json b/scripts/delivery/cdm_schemas/condition_occurrence.bak.json new file mode 100644 index 0000000..4072d55 --- /dev/null +++ b/scripts/delivery/cdm_schemas/condition_occurrence.bak.json @@ -0,0 +1,18 @@ +[ + {"name": "condition_occurrence_id", "type": "INTEGER", "mode": "REQUIRED" }, + {"name": "person_id", "type": "INTEGER", "mode": "REQUIRED" }, + {"name": "condition_concept_id", "type": "INTEGER", "mode": "REQUIRED" }, + {"name": "condition_start_date", "type": "DATE", "mode": "REQUIRED" }, + {"name": "condition_start_datetime", "type": "DATETIME" }, + {"name": "condition_end_date", "type": "DATE" }, + {"name": "condition_end_datetime", "type": "DATETIME" }, + {"name": "condition_type_concept_id", "type": "INTEGER", "mode": "REQUIRED" }, + {"name": "condition_status_concept_id", "type": "INTEGER"}, + {"name": "stop_reason", "type": "STRING" }, + {"name": "provider_id", "type": "INTEGER" }, + {"name": "visit_occurrence_id", "type": "INTEGER" }, + {"name": "visit_detail_id", "type": "INTEGER" }, + {"name": "condition_source_value", "type": "STRING" }, + {"name": "condition_source_concept_id", "type": "INTEGER" }, + {"name": "condition_status_source_value", "type": "STRING"} +] diff --git a/scripts/delivery/cdm_schemas/condition_occurrence.json b/scripts/delivery/cdm_schemas/condition_occurrence.json index 4072d55..1ef0093 100644 --- a/scripts/delivery/cdm_schemas/condition_occurrence.json +++ b/scripts/delivery/cdm_schemas/condition_occurrence.json @@ -7,12 +7,12 @@ {"name": "condition_end_date", "type": "DATE" }, {"name": "condition_end_datetime", "type": "DATETIME" }, {"name": "condition_type_concept_id", "type": "INTEGER", "mode": "REQUIRED" }, - {"name": "condition_status_concept_id", "type": "INTEGER"}, {"name": "stop_reason", "type": "STRING" }, {"name": "provider_id", "type": "INTEGER" }, {"name": "visit_occurrence_id", "type": "INTEGER" }, {"name": "visit_detail_id", "type": "INTEGER" }, {"name": "condition_source_value", "type": "STRING" }, {"name": "condition_source_concept_id", "type": "INTEGER" }, - {"name": "condition_status_source_value", "type": "STRING"} + {"name": "condition_status_source_value", "type": "STRING"}, + {"name": "condition_status_concept_id", "type": "INTEGER"} ] diff --git a/scripts/delivery/cdm_schemas/observation.bak.json b/scripts/delivery/cdm_schemas/observation.bak.json new file mode 100644 index 0000000..214b64d --- /dev/null +++ b/scripts/delivery/cdm_schemas/observation.bak.json @@ -0,0 +1,20 @@ +[ + {"name": "observation_id", "type": "INTEGER", "mode": "REQUIRED" }, + {"name": "person_id", "type": "INTEGER", "mode": "REQUIRED" }, + {"name": "observation_concept_id", "type": "INTEGER", "mode": "REQUIRED" }, + {"name": "observation_date", "type": "DATE", "mode": "REQUIRED" }, + {"name": "observation_datetime", "type": "DATETIME" }, + {"name": "observation_type_concept_id", "type": "INTEGER", "mode": "REQUIRED" }, + {"name": "value_as_number", "type": "FLOAT" }, + {"name": "value_as_string", "type": "STRING" }, + {"name": "value_as_concept_id", "type": "INTEGER" }, + {"name": "qualifier_concept_id", "type": "INTEGER" }, + {"name": "unit_concept_id", "type": "INTEGER" }, + {"name": "provider_id", "type": "INTEGER" }, + {"name": "visit_occurrence_id", "type": "INTEGER"}, + {"name": "visit_detail_id", "type": "INTEGER" }, + {"name": "observation_source_value", "type": "STRING" }, + {"name": "observation_source_concept_id", "type": "INTEGER" }, + {"name": "unit_source_value", "type": "STRING" }, + {"name": "qualifier_source_value", "type": "STRING"} +] \ No newline at end of file diff --git a/scripts/delivery/export_from_bq.py b/scripts/delivery/export_from_bq.py index bed823b..ece362c 100644 --- a/scripts/delivery/export_from_bq.py +++ b/scripts/delivery/export_from_bq.py @@ -6,6 +6,7 @@ import getopt import sys import json +import datetime # export_cmd = 'bq --location=US extract --destination_format=CSV --compression=NONE --field_delimiter=\t --print_header=TRUE \ # {proj}:{db}.{table} gs://{bucket}/csv/{table}/*{ext}' @@ -20,14 +21,14 @@ config_default = { - "bq_project": "odysseus-mimic-dev", - "bq_dataset": "mimiciv_demo_202012_cdm_531", + "bq_project": "...", + "bq_dataset": "...", "destination_format": "csv", - "destination_type": "gs", + "destination_type": "local", - "gs_bucket_path": "gs://mimic_iv_to_omop/export_cdm_demo", - "local_path": "../export_cdm", + "gs_bucket_path": "gs://...", + "local_path": "../...", "destination_formats": [ @@ -47,45 +48,56 @@ ], "bq_tables": [ - "observation", - "drug_era", - "fact_relationship", - "observation_period", - "cohort", - "visit_occurrence", - "drug_strength", - "condition_era", - "cdm_source", - "measurement", - "domain", - "note_nlp", - "metadata", + + # dimension tables + "location", + "care_site", "provider", "person", - "concept_relationship", - "cohort_attribute", + + # event tables + "death", + "visit_occurrence", + "visit_detail", + "condition_occurrence", "procedure_occurrence", - "care_site", - "vocabulary", "specimen", - "death", - "source_to_concept_map", + "measurement", + "observation", + "drug_exposure", "device_exposure", - "relationship", - "payer_plan_period", - "concept", - "location", - "condition_occurrence", - "concept_synonym", + "fact_relationship", + "observation_period", + + # tables which are not populated + "cohort", + "cohort_attribute", "cohort_definition", "attribute_definition", - "drug_exposure", - "note", - "concept_ancestor", "cost", + "note", + "note_nlp", + "metadata", + "payer_plan_period", + + # eras and cdm_source + "condition_era", + "drug_era", "dose_era", + "cdm_source", + + # vocabulary tables + "source_to_concept_map", "concept_class", - "visit_detail" + "domain", + "relationship", + "vocabulary", + "concept", + "concept_relationship", # to shard + "concept_ancestor", # to shard + "concept_synonym", + "drug_strength" + ] } @@ -131,7 +143,7 @@ def read_params(): if opt == '-f' or opt == '--format': params['destination_format'] = arg - print('the params given', params) + print('\nthe params given', params) return params # ---------------------------------------------------- @@ -149,6 +161,8 @@ def read_config(params): if os.path.isfile(config_file): with open(config_file) as f: config_read = json.load(f) + else: + print('Config file is not found:', config_file) # config has lower priority for k in config_default: @@ -168,11 +182,14 @@ def read_config(params): # commands # gs destination - always required step +# export_cmd_onedir = 'bq --location=US extract --destination_format={format_u} --compression=NONE {more_params} \ +# {proj}:{db}.{table} {bucket_path}/{format_l}/{table}{ext}' export_cmd_onedir = 'bq --location=US extract --destination_format={format_u} --compression=NONE {more_params} \ - {proj}:{db}.{table} {bucket_path}/{format_l}/{table}-*{ext}' + {proj}:{db}.{table} {bucket_path}/{table}{shard_mark}{ext}' # final destination - optional step -copy_cmd_onedir = 'gsutil cp {bucket_path}/{format_l}/{table}-*{ext} {local_path}/{format_l}/' +# copy_cmd_onedir = 'gsutil cp {bucket_path}/{format_l}/{table}{ext} {local_path}/{format_l}/' +copy_cmd_onedir = 'gsutil cp {bucket_path}/{table}*{ext} {local_path}/' # func @@ -189,6 +206,7 @@ def mkdir_if_needed(p): def main(): rc = 0 + duration = datetime.datetime.now() params = read_params() config = read_config(params) @@ -209,33 +227,40 @@ def main(): bq_dataset = config["bq_dataset"] - # gs destination - required step always for bq_table in bq_tables: + # gs destination - required step always bqc = export_cmd_onedir.format( format_u = destination_format.upper(), format_l = destination_format.lower(), more_params = more_params, proj = bq_project, db = bq_dataset, table = bq_table, bucket_path = gs_bucket_path, - ext = '.' + destination_format + ext = '.' + destination_format, + shard_mark = '' ) print(bqc) rc = os.system(bqc) - if rc != 0: break - - # final destination - if this type of export is selected - - if destination_type == 'local' and rc == 0: + - # create path if not exists - mkdir_if_needed(local_destination_path) - mkdir_if_needed('{0}/{1}'.format(local_destination_path, destination_format)) - - for bq_table in bq_tables: + if rc != 0: # a possible obstacle is when a table is too large. Then try to shard export. + bqc = export_cmd_onedir.format( + format_u = destination_format.upper(), format_l = destination_format.lower(), + more_params = more_params, + proj = bq_project, db = bq_dataset, table = bq_table, + bucket_path = gs_bucket_path, + ext = '.' + destination_format, + shard_mark = '*' + ) + print(bqc) + rc = os.system(bqc) - mkdir_if_needed('{0}/{1}/{2}'.format( - local_destination_path, destination_format, bq_table)) + if rc != 0: break + # final destination - if this type of export is selected + if destination_type == 'local' and rc == 0: + # create path if not exists + mkdir_if_needed(local_destination_path) + # mkdir_if_needed('{0}/{1}'.format(local_destination_path, destination_format)) bqc = copy_cmd_onedir.format( format_l = destination_format.lower(), table = bq_table, @@ -247,6 +272,9 @@ def main(): rc = os.system(bqc) if rc != 0: break + duration = datetime.datetime.now() - duration + print('Run time: {0}'.format(duration)) # timedelta HH:MM:SS.f + return rc # go @@ -254,4 +282,4 @@ def main(): rc = main() exit(rc) -# last edit: 2020-12-21 \ No newline at end of file +# last edit: 2021-02-07 diff --git a/scripts/delivery/export_from_bq_achilles_results.conf b/scripts/delivery/export_from_bq_achilles_results.conf new file mode 100644 index 0000000..b5fd3d4 --- /dev/null +++ b/scripts/delivery/export_from_bq_achilles_results.conf @@ -0,0 +1,20 @@ +{ + + "bq_project": "project...", + "bq_dataset": "dataset...", + + "destination_format": "csv", + "destination_type": "local", + + "gs_bucket_path": "gs://bucket.../folder...", + "local_path": "../folder...", + + "bq_tables": [ + "achilles_analysis", + "achilles_heel_results", + "achilles_results", + "achilles_results_derived", + "achilles_results_dist" + ] + +} diff --git a/scripts/delivery/export_from_bq_demo.conf b/scripts/delivery/export_from_bq_demo.conf index de7f921..475fc7c 100644 --- a/scripts/delivery/export_from_bq_demo.conf +++ b/scripts/delivery/export_from_bq_demo.conf @@ -1,11 +1,12 @@ { - "bq_project": "odysseus-mimic-dev", - "bq_dataset": "mimiciv_demo_202012_cdm_531", + "bq_project": "project...", + "bq_dataset": "dataset...", "destination_format": "csv", - "destination_type": "gs", + "destination_type": "local", + + "gs_bucket_path": "gs://bucket.../export_folder_csv...", + "local_path": "../export_folder_csv..." - "gs_bucket_path": "gs://mimic_iv_to_omop/export_cdm_demo", - "local_path": "../export_cdm_demo" } diff --git a/scripts/delivery/load_to_bq.py b/scripts/delivery/load_to_bq.py index 91e54c7..5ba2072 100644 --- a/scripts/delivery/load_to_bq.py +++ b/scripts/delivery/load_to_bq.py @@ -26,13 +26,11 @@ config_default = { - "bq_target_project": "odysseus-mimic-dev", + "bq_target_project": "project...", "bq_target_dataset": "mimiciv_delivery_demo", - "overwrite": "no", - "source_format": "csv", - "csv_path": "gs://mimic_iv_to_omop/export_cdm_demo", + "csv_path": "gs://...", "csv_delimiter": ",", "csv_quote": "\\\"", "schemas_dir_all_csv": "scripts/delivery/cdm_schemas_custom", @@ -41,45 +39,56 @@ "bq_tables": [ - "observation", - "drug_era", - "fact_relationship", - "observation_period", - "cohort", - "visit_occurrence", - "drug_strength", - "condition_era", - "cdm_source", - "measurement", - "domain", - "note_nlp", - "metadata", + + # dimension tables + "location", + "care_site", "provider", "person", - "concept_relationship", - "cohort_attribute", + + # event tables + "death", + "visit_occurrence", + "visit_detail", + "condition_occurrence", "procedure_occurrence", - "care_site", - "vocabulary", "specimen", - "death", - "source_to_concept_map", + "measurement", + "observation", + "drug_exposure", "device_exposure", - "relationship", - "payer_plan_period", - "concept", - "location", - "condition_occurrence", - "concept_synonym", + "fact_relationship", + "observation_period", + + # tables which are not populated + "cohort", + "cohort_attribute", "cohort_definition", "attribute_definition", - "drug_exposure", - "note", - "concept_ancestor", "cost", + "note", + "note_nlp", + "metadata", + "payer_plan_period", + + # eras and cdm_source + "condition_era", + "drug_era", "dose_era", + "cdm_source", + + # vocabulary tables + "source_to_concept_map", "concept_class", - "visit_detail" + "domain", + "relationship", + "vocabulary", + "concept", + "concept_relationship", # to shard + "concept_ancestor", # to shard + "concept_synonym", + "drug_strength" + ] @@ -166,12 +175,14 @@ def create_bq_dataset(ds_name): ''' def load_table(table, files_path, field_delimiter, quote, config): + print('Try to load: {0}'.format(table)) + return_code = 0 schema_path = '{dir}/{table}.json' - bq_load_command = 'bq --location=US load {replace} --source_format={format_u} {more_params} \ - {dataset}.{prefix}{table} {files_path}/{format_l}/{table}-*{ext} {schema_path}' + bq_load_command = 'bq --location=US load --replace --source_format={format_u} {more_params} \ + {dataset}.{prefix}{table} {files_path}/{table}{shard_mark}{ext} {schema_path}' more_params_csv = "" + \ " --allow_quoted_newlines=True " + \ @@ -187,8 +198,9 @@ def load_table(table, files_path, field_delimiter, quote, config): table_schema = schema_path.format(dir=config['schemas_dir_all_csv'], table=table) if os.path.isfile(table_schema): + + # try to load single file bqc = bq_load_command.format( \ - replace = '--replace' if config["overwrite"] == 'yes' else '', \ format_u = config["source_format"].upper(), \ more_params = more_params, \ dataset = config['bq_target_dataset'], \ @@ -197,11 +209,30 @@ def load_table(table, files_path, field_delimiter, quote, config): files_path = files_path, \ format_l = config["source_format"].lower(), \ ext = '.' + config["source_format"], \ - schema_path = table_schema + schema_path = table_schema, + shard_mark = '' ) print('To BQ: ' + bqc) - rc = os.system(bqc) + + # if failed (URI not found), try to load multiple files + if rc != 0: + bqc = bq_load_command.format( \ + format_u = config["source_format"].upper(), \ + more_params = more_params, \ + dataset = config['bq_target_dataset'], \ + prefix = config['bq_temp_table_prefix'], \ + table = table, \ + files_path = files_path, \ + format_l = config["source_format"].lower(), \ + ext = '.' + config["source_format"], \ + schema_path = table_schema, + shard_mark = '*' + ) + print('To BQ: ' + bqc) + rc = os.system(bqc) + + if rc != 0: return_code = 2 # error during execution of the command @@ -240,37 +271,33 @@ def nice_message(s_filename, status, msg): ''' def main(): - params = read_params() + rc = 0 + duration = datetime.datetime.now() + params = read_params() config = read_config(params.get('config_file')) - file_path = '{files_path}/{table}-*{ext}' - s_done = [] s_done.append(nice_message('start...', 0, '')) create_bq_dataset(config['bq_target_dataset']) - rca = 0 for table in config['bq_tables']: - # files_path = file_path.format( - # files_path=config['csv_path'], table=table, ext='.csv') - rc = load_table(table, config['csv_path'], config['csv_delimiter'], \ config['csv_quote'], config) if rc != 0: break - # rca += 1 - # continue s_done.append(nice_message('finish', 0, '')) print('\nTables are loaded to {pr}.{ds}'.format(pr=config['bq_target_project'], ds=config['bq_target_dataset'])) for a in s_done: print(a) + duration = datetime.datetime.now() - duration + print('Run time: {0}'.format(duration)) # timedelta HH:MM:SS.f - return rc #rca + return rc # ---------------------------------------------------- # go @@ -280,4 +307,4 @@ def main(): exit(return_code) -# last edit: 2020-12-21 +# last edit: 2021-02-07 diff --git a/scripts/delivery/load_to_bq_demo.conf b/scripts/delivery/load_to_bq_demo.conf index 5234e7e..e96b8c4 100644 --- a/scripts/delivery/load_to_bq_demo.conf +++ b/scripts/delivery/load_to_bq_demo.conf @@ -1,12 +1,12 @@ { - "bq_target_project": "odysseus-mimic-dev", - "bq_target_dataset": "mimiciv_delivery_demo", - "csv_path": "gs://mimic_iv_to_omop/export_cdm_demo", + "bq_target_project": "project...", + "bq_target_dataset": "dataset...", + "csv_path": "gs://bucket.../export_folder...", "csv_delimiter": ",", "csv_quote": "\\\"", - "schemas_dir_all_csv": "scripts/delivery/cdm_schemas_custom", + "schemas_dir_all_csv": "scripts/delivery/cdm_schemas", "source_format": "csv" diff --git a/scripts/export_bq_to_gs_draft.py b/scripts/export_bq_to_gs_draft.py deleted file mode 100644 index bbe3365..0000000 --- a/scripts/export_bq_to_gs_draft.py +++ /dev/null @@ -1,129 +0,0 @@ -# -# Export given tables from BigQuery to Google Storage -# - -import os - -# export_cmd = 'bq --location=US extract --destination_format=CSV --compression=NONE --field_delimiter=\t --print_header=TRUE \ -# {proj}:{db}.{table} gs://{bucket}/csv/{prefix}{table}/*{ext}' - -# variables - -destination_formats = [ - { - "id": "avro", - "more_params": "--use_avro_logical_types" - }, - { - "id": "csv", - "more_params": "--field_delimiter=\",\" --print_header=TRUE" - } -] - -# data -bq_project = 'odysseus-mimic-dev' -bq_dataset = 'mimiciv_cdm_tuf_10_ant_2020_10_21' -bq_table_prefix = 'cdm_' -bq_tables = [ - 'person', - 'death' - ] - -# gs destination -gs_bucket_path = 'gs://mimic_iv_to_omop/export_cdm' -gs_file_prefix = '' - -export_cmd_multdir = 'bq --location=US extract --destination_format={format_u} --compression=NONE {more_params} \ - {proj}:{db}.{prefix_source}{table} {bucket_path}/{format_l}/{prefix_dest}{table}/*{ext}' -export_cmd_onedir = 'bq --location=US extract --destination_format={format_u} --compression=NONE {more_params} \ - {proj}:{db}.{prefix_source}{table} {bucket_path}/{format_l}/{prefix_dest}{table}-*{ext}' - -# final destination -local_destination_path = '../export_cdm' -copy_cmd_multdir = 'gsutil cp {bucket_path}/{format_l}/{prefix_dest}{table}/*{ext} {local_path}/{format_l}/{prefix_dest}{table}/' -copy_cmd_onedir = 'gsutil cp {bucket_path}/{format_l}/{prefix_dest}{table}-*{ext} {local_path}/{format_l}/' - -# func - -def mkdir_if_needed(p): - if not os.path.isdir(p): - bqc = 'mkdir {0}'.format(p) - print(bqc) - rc = os.system(bqc) - if rc != 0: exit(rc) - - -# iterate through both formats - -for d in destination_formats: - - # if d["id"] == 'avro': continue - - destination_format = d["id"] - more_params = d["more_params"] - - # create path if not exists - - mkdir_if_needed(local_destination_path) - mkdir_if_needed('{0}/{1}'.format(local_destination_path, destination_format)) - - # gs destination - - for bq_table in bq_tables: - - bqc = export_cmd_onedir.format( - format_u = destination_format.upper(), format_l = destination_format.lower(), - more_params = more_params, - proj = bq_project, db = bq_dataset, prefix_source = bq_table_prefix, table = bq_table, - bucket_path = gs_bucket_path, prefix_dest = gs_file_prefix, - ext = '.' + destination_format - ) - print(bqc) - rc = os.system(bqc) - if rc != 0: exit(rc) - - bqc = export_cmd_multdir.format( - format_u = destination_format.upper(), format_l = destination_format.lower(), - more_params = more_params, - proj = bq_project, db = bq_dataset, prefix_source = bq_table_prefix, table = bq_table, - bucket_path = gs_bucket_path, prefix_dest = gs_file_prefix, - ext = '.' + destination_format - ) - print(bqc) - rc = os.system(bqc) - if rc != 0: exit(rc) - - # final destination - - for bq_table in bq_tables: - - mkdir_if_needed('{0}/{1}/{2}{3}'.format( - local_destination_path, destination_format, gs_file_prefix, bq_table)) - - bqc = copy_cmd_onedir.format( - format_l = destination_format.lower(), - table = bq_table, - bucket_path = gs_bucket_path, - prefix_dest = gs_file_prefix, - ext = '.' + destination_format, - local_path = local_destination_path - ) - print(bqc) - rc = os.system(bqc) - if rc != 0: exit(rc) - - - bqc = copy_cmd_multdir.format( - format_l = destination_format.lower(), - table = bq_table, - bucket_path = gs_bucket_path, - prefix_dest = gs_file_prefix, - ext = '.' + destination_format, - local_path = local_destination_path - ) - print(bqc) - rc = os.system(bqc) - if rc != 0: exit(rc) -exit(0) - -# last edit: 2020-12-03 \ No newline at end of file diff --git a/scripts/export_output_of_bqc.txt b/scripts/export_output_of_bqc.txt deleted file mode 100644 index 6590544..0000000 --- a/scripts/export_output_of_bqc.txt +++ /dev/null @@ -1,44 +0,0 @@ -Waiting on bqjob_r11cbf3abc63c5026_00000176709d5270_1 ... (0s) Current status: DONE -+-----------------------+ -| table_name | -+-----------------------+ -| observation | -| drug_era | -| fact_relationship | -| observation_period | -| cohort | -| visit_occurrence | -| drug_strength | -| condition_era | -| cdm_source | -| measurement | -| domain | -| note_nlp | -| metadata | -| provider | -| person | -| concept_relationship | -| cohort_attribute | -| procedure_occurrence | -| care_site | -| vocabulary | -| specimen | -| death | -| source_to_concept_map | -| device_exposure | -| relationship | -| payer_plan_period | -| concept | -| location | -| condition_occurrence | -| concept_synonym | -| cohort_definition | -| attribute_definition | -| drug_exposure | -| note | -| concept_ancestor | -| cost | -| dose_era | -| concept_class | -| visit_detail | -+-----------------------+ diff --git a/scripts/gen_bq_tmp_navigate_fields.py b/scripts/gen_bq_tmp_navigate_fields.py deleted file mode 100644 index 175341c..0000000 --- a/scripts/gen_bq_tmp_navigate_fields.py +++ /dev/null @@ -1,250 +0,0 @@ -# -# Generate SQL script to convert datetime fields in the imported dataset -# - -import os -import sys -import getopt -import json -import re -# import bq_run_script - -# ---------------------------------------------------- -''' - python gen_bq_tmp_to_cdm_delivery.py - To create config file, copy the config_default content to a .conf file and set the desired values - target dataset name template: name_dateref_cdm_version (i.e. synpuf_2018q3_cdm_53) -''' -# ---------------------------------------------------- - -config_default = { - - "output_sql_file": "scripts/delivery/crete_cdm_from_tmp_gen.sql", - - "bq_target_project": "odysseus-mimic-dev", - "bq_target_dataset": "mimiciv_delivery_demo", - - "bq_temp_table_prefix": "tmp_", - - "schema_dir": "scripts/delivery/cdm_schemas_custom", - - "bq_tables": - [ - "observation", - "drug_era", - "fact_relationship", - "observation_period", - "cohort", - "visit_occurrence", - "drug_strength", - "condition_era", - "cdm_source", - "measurement", - "domain", - "note_nlp", - "metadata", - "provider", - "person", - "concept_relationship", - "cohort_attribute", - "procedure_occurrence", - "care_site", - "vocabulary", - "specimen", - "death", - "source_to_concept_map", - "device_exposure", - "relationship", - "payer_plan_period", - "concept", - "location", - "condition_occurrence", - "concept_synonym", - "cohort_definition", - "attribute_definition", - "drug_exposure", - "note", - "concept_ancestor", - "cost", - "dose_era", - "concept_class", - "visit_detail" - ] - -} - -# ---------------------------------------------------- -# read_params() -# ---------------------------------------------------- - -def read_params(): - - print('Read params...') - params = { - "config_file": "optional: indicate '-c' for 'config', json file name. Default config file name is the same as script." - } - - # Parsing command line arguments - try: - opts, args = getopt.getopt(sys.argv[1:],"c:",["config="]) - # if len(opts) == 0: - # raise getopt.GetoptError("read_params() error", "Mandatory argument is missing.") - except getopt.GetoptError: - print("Please indicate correct params:") - print(params) - print("for example:\npython gen_bq_tmp_to_cdm_delivery.py --config gen_bq_tmp_to_cdm_delivery.conf") - sys.exit(2) - - st = [] - - params['config_file'] = os.path.basename(__file__).replace('py', 'conf') - - for opt, arg in opts: - - if opt == '-c' or opt == '--config': - if os.path.isfile(arg): - params['config_file'] = arg - else: - params['config_file'] = '' - - - print(params) - return params - -# ---------------------------------------------------- -# read_config() -# ---------------------------------------------------- - -def read_config(config_file): - - print('Read config...') - config = {} - config_read = {} - - if os.path.isfile(config_file): - with open(config_file) as f: - config_read = json.load(f) - - for k in config_default: - s = config_read.get(k, config_default[k]) - config[k] = s - - print(config) - return config - - -# ------------------------------------------------------------------------ -# Get concept fields and Co from schemas provided -# {tables: [{table, fields: [field]}]} -# ------------------------------------------------------------------------ -def generate_fields_for_table(table_name, schema_dir): - - table_schema = '{dir}/{table}.json'.format(dir=schema_dir, table=table_name) - - with open(table_schema) as f: - j_raw = json.load(f) - - target_fields = [] - - for fld in j_raw: - field_name = fld['name'] - field_type = fld['type'] - - if field_type == 'DATETIME': - field = 'PARSE_DATETIME(\'%Y-%m-%d %h:%m:s\', {field_name}) AS {field_name}' - else: - if field_type == 'DATE': - field = 'PARSE_DATE(\'%Y-%m-%d\', {field_name}) AS {field_name}' - else: - field = field_name - - # plus comma - if len(target_fields) < len(j_raw) - 1: - field = field + ',' - - target_fields.append(target_field) - - return target_fields - - - -''' ----------------------------------------------------- - generate_queries() - returns a list of query strings ----------------------------------------------------- -''' - -def generate_queries(header, result_tables, config): - - s_queries = ['-- {0}\n\n'.format(header)] - - # config['bq_target_project'] - - # to add list of fields and exclude not standard fields - s_query = 'CREATE OR REPLACE TABLE `{target_project}`.{target_dataset}.{table} AS \n' + \ - 'SELECT {fields} FROM `{cdm_project}`.{cdm_dataset}.{prefix}{table};\n' - - for t in result_tables: - - flds = generate_fields_for_table(t, config['schema_dir']) - - s_queries.append(s_query.format( - target_project = config['bq_target_project'], - target_dataset = config['bq_target_dataset'], - cdm_project = config['bq_cdm_project'], - cdm_dataset = config['bq_cdm_dataset'], - prefix = result_tables['prefix'], - table = t, - fields = flsds - )) - - return s_queries - -''' ----------------------------------------------------- - main() - return code: - 0 = success - ... ----------------------------------------------------- -''' - -def main(): - return_code = 0 - - params = read_params() - - config = read_config(params.get('config_file')) - - print('Generating script...') - - s_cdm_tables = config['bq_tables'] - s_cdm_queries = generate_queries('Convert DATE and DATETIME fields', s_cdm_tables, config) - - with open(config['output_sql_file'], 'w') as f: - - f.write('-- bq_cdm_to_target generated script --\n\n') - f.write('-- Unload to target') - - for s in s_cdm_queries: - f.write(s) - - f.close() - print('Generated', f.name) - - # for s in s_cdm_queries: - # rc = bq_run_script.run_query(s) - # if rc != 0: - # return_code += 1 - # continue - - return return_code - -# ---------------------------------------------------- -# go -# ---------------------------------------------------- -return_code = main() - -exit(return_code) - diff --git a/scripts/gen_bq_cdm_to_atlas.conf b/scripts/gen_sql_helper/gen_cdm_to_atlas.conf similarity index 71% rename from scripts/gen_bq_cdm_to_atlas.conf rename to scripts/gen_sql_helper/gen_cdm_to_atlas.conf index 608db66..34ad982 100644 --- a/scripts/gen_bq_cdm_to_atlas.conf +++ b/scripts/gen_sql_helper/gen_cdm_to_atlas.conf @@ -11,5 +11,8 @@ "voc_ddl": "etl/ddl/ddl_voc_5_3_1.sql", "cdm_ddl": "etl/ddl/ddl_cdm_5_3_1.sql", - "output_sql_file": "etl/unload/unload_to_atlas_gen_try_fields.sql" + "fields_to_remove": ["unit_id", "load_table_id", "load_row_id", "trace_id"], + + "output_sql_file": "etl/unload/unload_to_atlas_gen.sql" + } diff --git a/scripts/gen_bq_cdm_to_atlas.py b/scripts/gen_sql_helper/gen_cdm_to_atlas.py similarity index 66% rename from scripts/gen_bq_cdm_to_atlas.py rename to scripts/gen_sql_helper/gen_cdm_to_atlas.py index 2d50b95..f0a2dc5 100644 --- a/scripts/gen_bq_cdm_to_atlas.py +++ b/scripts/gen_sql_helper/gen_cdm_to_atlas.py @@ -7,11 +7,10 @@ import getopt import json import re -# import bq_run_script # ---------------------------------------------------- ''' - python gen_bq_cdm_to_atlas.py + python gen_cdm_to_atlas.py To create config file, copy the config_default content to a .conf file and set the desired values ATLAS dataset name template: name_dateref_cdm_version (i.e. synpuf_2018q3_cdm_53) ''' @@ -19,19 +18,21 @@ config_default = { - "bq_cdm_project": "a_project", - "bq_cdm_dataset": "a_dataset", + "bq_cdm_project": "@etl_project", + "bq_cdm_dataset": "@etl_dataset", - "bq_atlas_project": "a_project", - "bq_atlas_dataset": "name_dateref_cdm_version", + "bq_atlas_project": "@atlas_project", + "bq_atlas_dataset": "@atlas_dataset", "cdm_prefix": "cdm_", "voc_prefix": "voc_", - "voc_ddl": "../etl/ddl_voc_5_3_1.sql", - "cdm_ddl": "../etl/ddl_cdm_5_3_1.sql", + "voc_ddl": "etl/ddl/ddl_voc_5_3_1.sql", + "cdm_ddl": "etl/ddl/ddl_cdm_5_3_1.sql", - "output_sql_file": "../etl/output_file.sql" + "fields_to_remove": ["unit_id", "load_table_id", "load_row_id", "trace_id"], + + "output_sql_file": "etl/unload/unload_to_atlas_gen.sql" } # ---------------------------------------------------- @@ -53,7 +54,7 @@ def read_params(): except getopt.GetoptError: print("Please indicate correct params:") print(params) - print("for example:\npython gen_bq_cdm_to_atlas.py --config gen_bq_cdm_to_atlas.conf") + print("for example:\npython gen_cdm_to_atlas.py --config gen_cdm_to_atlas.conf") sys.exit(2) st = [] @@ -108,27 +109,58 @@ def read_ddl(ddl_file, cdm_dataset, prefix): "tables": [] } + print('\nread structure for {0} tables...'.format(prefix)) + if os.path.isfile(ddl_file): - s_raw = open(ddl_file).read().split('\n') - s_ddl = filter(lambda x: x.strip()[0:2] != '--', s_raw) + s_text = open(ddl_file).read() pr = cdm_dataset + '.' + prefix - print('read_ddl.tables_template', pr) - s_tables = filter(lambda x: pr in x, s_ddl) + s_raw_lines = s_text.split('\n') + s_comments_removed = filter(lambda x: x.strip()[0:2] != '--', s_raw_lines) + s_raw_tables = filter(lambda x: pr in x, s_comments_removed) + + # iterate through raw lines like # CREATE OR REPLACE TABLE dataset.cdm_cohort_definition ( - for s_t in s_tables: - t = s_t.partition(pr)[2].partition('(')[0].strip() - result_tables['tables'].append(t) + for raw_table in s_raw_tables: - print('read_ddl.s_t', s_t) - print('read_ddl.t', t) - break + table_name = raw_table.partition(pr)[2].partition('(')[0].strip() - # print(result_tables['tables'][1]) + flds_str = get_fields_for_table(raw_table, s_text) + + result_table = {"fields": flds_str, "table_name": table_name} + result_tables['tables'].append(result_table) + print(table_name) return result_tables +''' +------------------------------------------------------------------------ + Get fields list from DDL + raw_table - starting mark + s_text - ddl text +------------------------------------------------------------------------ +''' +def get_fields_for_table(raw_table, s_text): + + search_pattern = '(?:{mark1}\\n)([\\S\\s\\n]+?)(?:{mark2})'.format(mark1=raw_table.replace('(', '\\('), mark2 = '\\)\\n;') + + flds_raw = re.search(search_pattern, s_text).group(1).split('\n') + + flds_list = [x.strip().partition(' ')[0] for x in flds_raw] + + fields_to_remove = ['unit_id', 'load_table_id', 'load_row_id', 'trace_id'] + fields_to_remove.append('') + fields_to_remove.append('(') # quick bugfix + for f in flds_list: + if f in fields_to_remove: flds_list.remove(f) + + flds_str = ' ' + ',\n '.join(flds_list) + + return flds_str + + + ''' ---------------------------------------------------- generate_queries() @@ -144,7 +176,7 @@ def generate_queries(header, result_tables, config): # to add list of fields and exclude not standard fields s_query = 'CREATE OR REPLACE TABLE `{atlas_project}`.{atlas_dataset}.{table} AS \n' + \ - 'SELECT * FROM `{cdm_project}`.{cdm_dataset}.{prefix}{table};\n' + 'SELECT\n{fields}\nFROM `{cdm_project}`.{cdm_dataset}.{prefix}{table};\n\n' for t in result_tables['tables']: s_queries.append(s_query.format( @@ -153,7 +185,8 @@ def generate_queries(header, result_tables, config): cdm_project = config['bq_cdm_project'], cdm_dataset = config['bq_cdm_dataset'], prefix = result_tables['prefix'], - table = t + table = t['table_name'], + fields = t['fields'] )) return s_queries @@ -196,7 +229,7 @@ def main(): f.write(s) f.close() - print('Generated', f.name) + print('\nGenerated', f.name) # for s in s_cdm_queries: # rc = bq_run_script.run_query(s) diff --git a/scripts/unload_bq_parquet_draft.py b/scripts/unload_bq_parquet_draft.py deleted file mode 100644 index 2821dde..0000000 --- a/scripts/unload_bq_parquet_draft.py +++ /dev/null @@ -1,166 +0,0 @@ -# -# Unload hive tables to BigQuery -# -# Version to create Achilles compatible tables -# - -# pyspark --packages com.databricks:spark-avro_2.11:4.0.0 - -# for report/**/*.json -# python webreports/manage.py eject epic_ehr.dev - -## -- go - -from pyspark.sql import SparkSession -import os - -# 0 = don't, 1 = do -do_load_cdm = 1 -do_load_voc = 1 -do_backup_cdm = 0 -do_backup_voc = 0 -# do_remove_dev_features = 0 - -# dev databases -db_hive_cdm = 'starr_epic_etl_ref' -db_hive_voc = 'vocabulary2_ref' - -# dev etl config -db_bq_cdm = 'starr_epic_staging' -db_bq_cdm_bak = '' -gs_path_cdm = 'gs://starr_epic_etl_ext/parquet/' - -# dev voc config -db_bq_voc = 'starr_epic_vocabulary' -db_bq_voc_bak = 'starr_epic_vocabulary_0514' -gs_path_voc = 'gs://starr_epic_vocabulary/parquet/' - -file_extension = 'parquet' -# file_format_name = 'com.databricks.spark.avro' -file_format_name = 'parquet' -bq_load_command = "bq --location=US load --replace --source_format=PARQUET {table_name} \"{files_path}\"" -bq_backup_command = "bq cp --force {src_table_name} {dst_table_name}" - -# tables list -tables = [ -'cdm_care_site', -'cdm_cdm_source', -'cdm_cohort', -'cdm_cohort_attribute', -'cdm_condition_era', -'cdm_condition_occurrence', -'cdm_cost', -'cdm_death', -'cdm_device_exposure', -'cdm_dose_era', -'cdm_drug_era', -'cdm_drug_exposure', -'cdm_fact_relationship', -'cdm_location', -'cdm_measurement', -'cdm_metadata', -'cdm_note', -'cdm_note_nlp', -'cdm_observation', -'cdm_observation_period', -'cdm_payer_plan_period', -'cdm_person', -'cdm_procedure_occurrence', -'cdm_provider', -'cdm_source_to_concept_map', -'cdm_specimen', -'cdm_visit_detail', -'cdm_visit_occurrence', -'join_voc_drug', -'lk_clean_drug', -'lk_clean_drug_codes', -'lk_clean_drug_med_admin', -'lk_clean_drug_order_med', -'lk_clean_dx', -'lk_clean_mapped_drug', -'lk_clean_mapped_dx', -'lk_clean_mapped_meas', -'lk_clean_mapped_obs', -'lk_clean_mapped_proc', -'lk_clean_meas', -'lk_clean_obs', -'lk_clean_proc', -'lk_concept_drug', -'lk_concept_dx', -'lk_concept_meas', -'lk_concept_obs', -'lk_concept_proc', -'lk_concept_route', -'lk_drug_codes_ndc', -'lk_drug_codes_rx0', -'lk_drug_codes_rx1', -'lk_person', -'stcm_invalid', -'stcm_valid' -] - -vocs = [ - 'voc_concept', - 'voc_concept_ancestor', - 'voc_concept_class', - 'voc_concept_relationship', - 'voc_concept_synonym', - 'voc_domain', - 'voc_drug_strength', - 'voc_relationship', - 'voc_vocabulary' -] - -done = [] - -spark = SparkSession.builder.appName('unload_bq_dev').enableHiveSupport().getOrCreate() - - -for table in tables: - if do_backup_cdm == 1: - bqc = bq_backup_command.format(src_table_name = db_bq_cdm + '.' + table, \ - dst_table_name = db_bq_cdm_bak + '.' + table) - print('backup BQ table: ' + bqc) - os.system(bqc) - done.append(bqc) - - if do_load_cdm == 1: - print('Unloading ' + db_hive_cdm + '.' + table) - os.system("gsutil rm " + gs_path_cdm + table + '/*') - df = spark.table(db_hive_cdm + '.' + table) - print('To gs: ' + gs_path_cdm + table) - df.write.format(file_format_name).save(gs_path_cdm + table) - - bqc = bq_load_command.format(table_name = db_bq_cdm + '.' + table, \ - files_path = gs_path_cdm + table + '/*.' + file_extension) - print('To BQ: ' + bqc) - os.system(bqc) - done.append(bqc) - - -for table in vocs: - if do_backup_voc == 1: - bqc = bq_backup_command.format(src_table_name = db_bq_voc + '.' + table, \ - dst_table_name = db_bq_backup + '.' + table) - print('backup BQ table: ' + bqc) - os.system(bqc) - done.append(bqc) - - if do_load_voc == 1: - print('Unloading ' + db_hive_voc + '.' + table) - os.system("gsutil rm " + gs_path_voc + table + '/*') - df = spark.table(db_hive_voc + '.' + table) - print('To gs: ' + gs_path_voc + table) - df.write.format(file_format_name).save(gs_path_voc + table) - - bqc = bq_load_command.format(table_name = db_bq_voc + '.' + table, \ - files_path = gs_path_voc + table + '/*.' + file_extension) - print('To BQ: ' + bqc) - os.system(bqc) - done.append(bqc) - -# print('Now in ' + db_bq) -# os.system("bq ls " + db_bq) -print('\nCommands executed:') -for a in done: - print(a) diff --git a/scripts/wf_read.py b/scripts/wf_read.py index 847563b..731e144 100644 --- a/scripts/wf_read.py +++ b/scripts/wf_read.py @@ -36,14 +36,6 @@ } -################################################################################## - -# gsutil ls gs://mimic_iv_to_omop/waveforms/source_data/csv/ - -# subprocess.check_output('gsutil ls gs://mimic_iv_to_omop/waveforms/source_data/csv/', shell=True) - -################################################################################## - # ---------------------------------------------------- # read_params() # ---------------------------------------------------- @@ -258,6 +250,4 @@ def main(): # ------------- if __name__ == '__main__': - # test1.py executed as script - # do something main() \ No newline at end of file diff --git a/vocabulary_refresh/vocabulary_refresh.conf b/vocabulary_refresh/vocabulary_refresh.conf index 57173c4..518edbf 100644 --- a/vocabulary_refresh/vocabulary_refresh.conf +++ b/vocabulary_refresh/vocabulary_refresh.conf @@ -1,17 +1,17 @@ { - "bq_target_project": "odysseus-mimic-dev", - "bq_target_dataset": "vocabulary_2020_09_11", + "bq_target_project": "project...", + "bq_target_dataset": "vocabulary_dataset...", "bq_project_template": "@bq_target_project", "bq_dataset_template": "@bq_target_dataset", - "local_athena_csv_path": "../../vocabulary_2020_09_11", - "gs_athena_csv_path": "gs://mimic_iv_to_omop/vocab_2020_09_11", + "local_athena_csv_path": "../../vocabulary_dataset...", + "gs_athena_csv_path": "gs://bucket.../vocabulary_folder...", "athena_csv_delimiter": "\t", "athena_csv_quote": "", "local_mapping_csv_path": "../custom_mapping_csv", - "gs_mapping_csv_path": "gs://mimic_iv_to_omop/custom_mapping", + "gs_mapping_csv_path": "gs://bucket.../custom_mapping_folder...", "mapping_csv_delimiter": ",", "mapping_csv_quote": "\\\"", diff --git a/vocabulary_refresh/z_refresh_results.sql b/vocabulary_refresh/z_refresh_results.sql deleted file mode 100644 index 6cccd4d..0000000 --- a/vocabulary_refresh/z_refresh_results.sql +++ /dev/null @@ -1,181 +0,0 @@ --- ------------------------------------------------------------------- --- @2020, Odysseus Data Services, Inc. All rights reserved --- MIMIC IV CDM Conversion --- ------------------------------------------------------------------- - --- ----------------------------------------------------------------------------------- --- 2020-09-19 --- TUF-11, Load vocabularies to `odysseus-mimic-dev.vocabulary_2020_09_11` --- ----------------------------------------------------------------------------------- - -+----------+--------------------------------------------------------------------------+-----------+ -| check_id | check_name | row_count | -+----------+--------------------------------------------------------------------------+-----------+ -| 7 | wrong valid_start_date, valid_end_date or invalid_reason for the concept | 4271 | -+----------+--------------------------------------------------------------------------+-----------+ - - -SELECT c.vocabulary_id, count(*) as cnt -FROM `@bq_target_project.@bq_target_dataset`.z_check_voc_5 ch -INNER JOIN `@bq_target_project.@bq_target_dataset`.concept c - ON ch.concept_id_1 = c.concept_id -- concept_id_1 -GROUP BY c.vocabulary_id -ORDER BY c.vocabulary_id -; - -SELECT c.vocabulary_id, count(*) as cnt -FROM `@bq_target_project.@bq_target_dataset`.z_check_voc_7 ch -INNER JOIN `@bq_target_project.@bq_target_dataset`.concept c - ON ch.concept_id = c.concept_id -- concept_id (todo: unification) -GROUP BY c.vocabulary_id -ORDER BY c.vocabulary_id -; - -SELECT c.vocabulary_id, count(*) as cnt -FROM `@bq_target_project.@bq_target_dataset`.z_check_voc_12 ch -INNER JOIN `@bq_target_project.@bq_target_dataset`.concept c - ON ch.concept_id_1 = c.concept_id -GROUP BY c.vocabulary_id -ORDER BY c.vocabulary_id -; - - --- ----------------------------------------------------------------------------------- --- 2020-10-19 --- TUF-55, Load custom mapping --- ----------------------------------------------------------------------------------- - -+----------+--------------------------------------------------------------------------+-----------+ -| check_id | check_name | row_count | -+----------+--------------------------------------------------------------------------+-----------+ -| 5 | wrong relationships: Maps to TO D OR U or replacement relationships TO D | 26 | -| 7 | wrong valid_start_date, valid_end_date or invalid_reason for the concept | 4271 | -| 12 | wrong relationships: Maps to/Maps to value not to S | 32 | -| 15 | nonexistent classes | 1 | -| 16 | nonexistent domains | 1 | -+----------+--------------------------------------------------------------------------+-----------+ - --- 5 | wrong relationships: mimiciv_drug_ndc needs to be re-mapped | 26 | --- 7 | ICD10PSC, invalid_reason is null, and valid_end_date = 2019-12-31 | 4271 | --- 12 | wrong relationships: mimiciv_drug_ndc needs to be re-mapped | 32 | --- 15 | nonexistent classes - mimiciv_marital_status - fixed --- 16 | nonexistent domains - mimiciv_marital_status - fixed - --- ----------------------------------------------------------------------------------- --- 2020-11-18 --- TUF-55, Load custom mapping for microbiology and more --- ----------------------------------------------------------------------------------- - -+----------+--------------------------------------------------------------------------+-----------+ -| check_id | check_name | row_count | -+----------+--------------------------------------------------------------------------+-----------+ -| 5 | wrong relationships: Maps to TO D OR U or replacement relationships TO D | 26 | -| 7 | wrong valid_start_date, valid_end_date or invalid_reason for the concept | 4271 | -| 12 | wrong relationships: Maps to/Maps to value not to S | 32 | -+----------+--------------------------------------------------------------------------+-----------+ - --- ----------------------------------------------------------------------------------- --- 2020-11-18 --- TUF-55, Load custom mapping for microbiology and more --- ----------------------------------------------------------------------------------- - -+----------+--------------------------------------------------------------------------+-----------+ -| check_id | check_name | row_count | -+----------+--------------------------------------------------------------------------+-----------+ -| 5 | wrong relationships: Maps to TO D OR U or replacement relationships TO D | 27 | -| 7 | wrong valid_start_date, valid_end_date or invalid_reason for the concept | 4271 | -| 12 | wrong relationships: Maps to/Maps to value not to S | 36 | -| 16 | nonexistent domains | 1 | -+----------+--------------------------------------------------------------------------+-----------+ - --- | 16 | nonexistent domains - fixed, will be applied at the next loading | 1 | - --- ----------------------------------------------------------------------------------- --- 2020-12-10 --- TUF-??, Load custom mapping for after Full set Metrics report --- ----------------------------------------------------------------------------------- - -+----------+--------------------------------------------------------------------------+-----------+ -| check_id | check_name | row_count | -+----------+--------------------------------------------------------------------------+-----------+ -| 5 | wrong relationships: Maps to TO D OR U or replacement relationships TO D | 29 | -| 7 | wrong valid_start_date, valid_end_date or invalid_reason for the concept | 4271 | -| 12 | wrong relationships: Maps to/Maps to value not to S | 38 | -+----------+--------------------------------------------------------------------------+-----------+ - --- ----------------------------------------------------------------------------------- --- 2021-01-20 --- Load custom mapping added by comments in DQD error report --- ----------------------------------------------------------------------------------- - -+----------+--------------------------------------------------------------------------+-----------+ -| check_id | check_name | row_count | -+----------+--------------------------------------------------------------------------+-----------+ -| 5 | wrong relationships: Maps to TO D OR U or replacement relationships TO D | 57 | +28 -| 7 | wrong valid_start_date, valid_end_date or invalid_reason for the concept | 4271 | -| 12 | wrong relationships: Maps to/Maps to value not to S | 73 | +35 -+----------+--------------------------------------------------------------------------+-----------+ - --- ----------------------------------------------------------------------------------- --- 2021-01-21 --- Fix and load again: custom mapping added by comments in DQD error report --- ----------------------------------------------------------------------------------- - -+----------+--------------------------------------------------------------------------+-----------+ -| check_id | check_name | row_count | -+----------+--------------------------------------------------------------------------+-----------+ -| 5 | wrong relationships: Maps to TO D OR U or replacement relationships TO D | 28 | -1 -| 7 | wrong valid_start_date, valid_end_date or invalid_reason for the concept | 4271 | -| 12 | wrong relationships: Maps to/Maps to value not to S | 36 | -2 -+----------+--------------------------------------------------------------------------+-----------+ - --- ----------------------------------------------------------------------------------- --- 2021-01-25 --- Re-mapped by comments in DQD error report --- ----------------------------------------------------------------------------------- - -+----------+--------------------------------------------------------------------------+-----------+ -| check_id | check_name | row_count | -+----------+--------------------------------------------------------------------------+-----------+ -| 7 | wrong valid_start_date, valid_end_date or invalid_reason for the concept | 4271 | -+----------+--------------------------------------------------------------------------+-----------+ - --- ----------------------------------------------------------------------------------- --- 2021-01-26 --- Remove duplicated measurement mapping --- ----------------------------------------------------------------------------------- - -+----------+--------------------------------------------------------------------------+-----------+ -| check_id | check_name | row_count | -+----------+--------------------------------------------------------------------------+-----------+ -| 7 | wrong valid_start_date, valid_end_date or invalid_reason for the concept | 4271 | -+----------+--------------------------------------------------------------------------+-----------+ - - - --- ----------------------------------------------------------------------------------- - -SELECT DISTINCT - 12 check_id, - 'wrong relationships: "Maps to"/"Maps to value" not to "S"' AS check_name, - c1.vocabulary_id AS source_vocabulary_id, - c1.concept_id AS source_concept_id, - c1.concept_code AS source_concept_code, - r.relationship_id AS relationship_id, - c2.concept_id AS target_concept_id, - c2.concept_code AS target_concept_code, - c2.concept_name AS target_concept_name, - c2.invalid_reason AS target_invalid_reason, - c2.valid_end_date AS target_valid_end_date -FROM - `odysseus-mimic-dev`.vocabulary_2020_09_11.concept c1, - `odysseus-mimic-dev`.vocabulary_2020_09_11.concept c2, - `odysseus-mimic-dev`.vocabulary_2020_09_11.concept_relationship r -WHERE - c1.concept_id = r.concept_id_1 - AND c2.concept_id = r.concept_id_2 - AND coalesce(c2.standard_concept,'C')<>'S' - AND r.relationship_id IN ('Maps to','Maps to value') - AND r.invalid_reason IS NULL -ORDER BY source_vocabulary_id, source_concept_id -; diff --git a/z_more/OMOP CDM bigquery ddl.txt b/z_more/OMOP CDM bigquery ddl.txt new file mode 100644 index 0000000..581d410 --- /dev/null +++ b/z_more/OMOP CDM bigquery ddl.txt @@ -0,0 +1,653 @@ +/*OMOP CDM v5.3.1 14June2018*/ +#standardsql +create table concept ( + concept_id INT64 not null , + concept_name STRING not null , + domain_id STRING not null , + vocabulary_id STRING not null , + concept_class_id STRING not null , + standard_concept STRING , + concept_code STRING not null , + valid_start_DATE DATE not null , + valid_end_DATE DATE not null , + invalid_reason STRING +) +; + + +create table vocabulary ( + vocabulary_id STRING not null, + vocabulary_name STRING not null, + vocabulary_reference STRING not null, + vocabulary_version STRING , + vocabulary_concept_id INT64 not null +) +; + + +create table domain ( + domain_id STRING not null, + domain_name STRING not null, + domain_concept_id INT64 not null +) +; + + +create table concept_class ( + concept_class_id STRING not null, + concept_class_name STRING not null, + concept_class_concept_id INT64 not null +) +; + + +create table concept_relationship ( + concept_id_1 INT64 not null, + concept_id_2 INT64 not null, + relationship_id STRING not null, + valid_start_DATE DATE not null, + valid_end_DATE DATE not null, + invalid_reason STRING + ) +; + + +create table relationship ( + relationship_id STRING not null, + relationship_name STRING not null, + is_hierarchical STRING not null, + defines_ancestry STRING not null, + reverse_relationship_id STRING not null, + relationship_concept_id INT64 not null +) +; + + +create table concept_synonym ( + concept_id INT64 not null, + concept_synonym_name STRING not null, + language_concept_id INT64 not null +) +; + + +create table concept_ancestor ( + ancestor_concept_id INT64 not null, + descendant_concept_id INT64 not null, + min_levels_of_separation INT64 not null, + max_levels_of_separation INT64 not null +) +; + + +create table source_to_concept_map ( + source_code STRING not null, + source_concept_id INT64 not null, + source_vocabulary_id STRING not null, + source_code_description STRING , + target_concept_id INT64 not null, + target_vocabulary_id STRING not null, + valid_start_DATE DATE not null, + valid_end_DATE DATE not null, + invalid_reason STRING +) +; + + + + +create table drug_strength ( + drug_concept_id INT64 not null, + ingredient_concept_id INT64 not null, + amount_value FLOAT64 , + amount_unit_concept_id INT64 , + numerator_value FLOAT64 , + numerator_unit_concept_id INT64 , + denominator_value FLOAT64 , + denominator_unit_concept_id INT64 , + box_size INT64 , + valid_start_DATE DATE not null, + valid_end_DATE DATE not null, + invalid_reason STRING +) +; + + + +create table cohort_definition ( + cohort_definition_id INT64 not null, + cohort_definition_name STRING not null, + cohort_definition_description STRING , + definition_type_concept_id INT64 not null, + cohort_definition_syntax STRING , + subject_concept_id INT64 not null, + cohort_initiation_DATE DATE +) +; + + +create table attribute_definition ( + attribute_definition_id INT64 not null, + attribute_name STRING not null, + attribute_description STRING , + attribute_type_concept_id INT64 not null, + attribute_syntax STRING +) +; + + +create table cdm_source +( + cdm_source_name STRING not null , + cdm_source_abbreviation STRING , + cdm_holder STRING , + source_description STRING , + source_documentation_reference STRING , + cdm_etl_reference STRING , + source_release_DATE DATE , + cdm_release_DATE DATE , + cdm_version STRING , + vocabulary_version STRING +) +; + + +create table metadata +( + metadata_concept_id INT64 not null , + metadata_type_concept_id INT64 not null , + name STRING not null , + value_as_string STRING , + value_as_concept_id INT64 , + metadata_DATE DATE , + metadata_DATETIME DATETIME +) +; + + + +--HINT DISTRIBUTE_ON_KEY(person_id) +create table person +( + person_id INT64 not null , + gender_concept_id INT64 not null , + year_of_birth INT64 not null , + month_of_birth INT64 , + day_of_birth INT64 , + birth_DATETIME DATETIME , + race_concept_id INT64 not null, + ethnicity_concept_id INT64 not null, + location_id INT64 , + provider_id INT64 , + care_site_id INT64 , + person_source_value STRING , + gender_source_value STRING , + gender_source_concept_id INT64 , + race_source_value STRING , + race_source_concept_id INT64 , + ethnicity_source_value STRING , + ethnicity_source_concept_id INT64 +) +; + + +--HINT DISTRIBUTE_ON_KEY(person_id) +create table observation_period +( + observation_period_id INT64 not null , + person_id INT64 not null , + observation_period_start_DATE DATE not null , + observation_period_end_DATE DATE not null , + period_type_concept_id INT64 not null +) +; + + +--HINT DISTRIBUTE_ON_KEY(person_id) +create table specimen +( + specimen_id INT64 not null , + person_id INT64 not null , + specimen_concept_id INT64 not null , + specimen_type_concept_id INT64 not null , + specimen_DATE DATE not null , + specimen_DATETIME DATETIME , + quantity FLOAT64 , + unit_concept_id INT64 , + anatomic_site_concept_id INT64 , + disease_status_concept_id INT64 , + specimen_source_id STRING , + specimen_source_value STRING , + unit_source_value STRING , + anatomic_site_source_value STRING , + disease_status_source_value STRING +) +; + + +--HINT DISTRIBUTE_ON_KEY(person_id) +create table death +( + person_id INT64 not null , + death_DATE DATE not null , + death_DATETIME DATETIME , + death_type_concept_id INT64 not null , + cause_concept_id INT64 , + cause_source_value STRING , + cause_source_concept_id INT64 +) +; + + +--HINT DISTRIBUTE_ON_KEY(person_id) +create table visit_occurrence +( + visit_occurrence_id INT64 not null , + person_id INT64 not null , + visit_concept_id INT64 not null , + visit_start_DATE DATE not null , + visit_start_DATETIME DATETIME , + visit_end_DATE DATE not null , + visit_end_DATETIME DATETIME , + visit_type_concept_id INT64 not null , + provider_id INT64 , + care_site_id INT64 , + visit_source_value STRING , + visit_source_concept_id INT64 , + admitting_source_concept_id INT64 , + admitting_source_value STRING , + discharge_to_concept_id INT64 , + discharge_to_source_value STRING , + preceding_visit_occurrence_id INT64 +) +; + + +--HINT DISTRIBUTE_ON_KEY(person_id) +create table visit_detail +( + visit_detail_id INT64 not null , + person_id INT64 not null , + visit_detail_concept_id INT64 not null , + visit_detail_start_DATE DATE not null , + visit_detail_start_DATETIME DATETIME , + visit_detail_end_DATE DATE not null , + visit_detail_end_DATETIME DATETIME , + visit_deatil_type_concept_id INT64 not null , + provider_id INT64 , + care_site_id INT64 , + admitting_source_concept_id INT64 , + discharge_to_concept_id INT64 , + preceding_visit_detail_id INT64 , + visit_detail_source_value STRING , + visit_deatil_source_concept_id INT64 , + admitting_source_value STRING , + discharge_to_source_value STRING , + visit_detail_parent_id INT64 , + visit_occurrence_id INT64 not null +) +; + + +--HINT DISTRIBUTE_ON_KEY(person_id) +create table procedure_occurrence +( + procedure_occurrence_id INT64 not null , + person_id INT64 not null , + procedure_concept_id INT64 not null , + procedure_DATE DATE not null , + procedure_DATETIME DATETIME , + procedure_type_concept_id INT64 not null , + modifier_concept_id INT64 , + quantity INT64 , + provider_id INT64 , + visit_occurrence_id INT64 , + visit_detail_id INT64 , + procedure_source_value STRING , + procedure_source_concept_id INT64 , + modifier_source_value STRING +) +; + + +--HINT DISTRIBUTE_ON_KEY(person_id) +create table drug_exposure +( + drug_exposure_id INT64 not null , + person_id INT64 not null , + drug_concept_id INT64 not null , + drug_exposure_start_DATE DATE not null , + drug_exposure_start_DATETIME DATETIME , + drug_exposure_end_DATE DATE not null , + drug_exposure_end_DATETIME DATETIME , + verbatim_end_DATE DATE , + drug_type_concept_id INT64 not null , + stop_reason STRING , + refills INT64 , + quantity FLOAT64 , + days_supply INT64 , + sig STRING , + route_concept_id INT64 , + lot_number STRING , + provider_id INT64 , + visit_occurrence_id INT64 , + visit_detail_id INT64 , + drug_source_value STRING , + drug_source_concept_id INT64 , + route_source_value STRING , + dose_unit_source_value STRING +) +; + + +--HINT DISTRIBUTE_ON_KEY(person_id) +create table device_exposure +( + device_exposure_id INT64 not null , + person_id INT64 not null , + device_concept_id INT64 not null , + device_exposure_start_DATE DATE not null , + device_exposure_start_DATETIME DATETIME , + device_exposure_end_DATE DATE , + device_exposure_end_DATETIME DATETIME , + device_type_concept_id INT64 not null , + unique_device_id STRING , + quantity INT64 , + provider_id INT64 , + visit_occurrence_id INT64 , + visit_detail_id INT64 , + device_source_value STRING , + device_source_concept_id INT64 +) +; + + +--HINT DISTRIBUTE_ON_KEY(person_id) +create table condition_occurrence +( + condition_occurrence_id INT64 not null , + person_id INT64 not null , + condition_concept_id INT64 not null , + condition_start_DATE DATE not null , + condition_start_DATETIME DATETIME , + condition_end_DATE DATE , + condition_end_DATETIME DATETIME , + condition_type_concept_id INT64 not null , + stop_reason STRING , + provider_id INT64 , + visit_occurrence_id INT64 , + visit_detail_id INT64 , + condition_source_value STRING , + condition_source_concept_id INT64 , + condition_status_source_value STRING , + condition_status_concept_id INT64 +) +; + + +--HINT DISTRIBUTE_ON_KEY(person_id) +create table measurement +( + measurement_id INT64 not null , + person_id INT64 not null , + measurement_concept_id INT64 not null , + measurement_DATE DATE not null , + measurement_DATETIME DATETIME , + measurement_time STRING , + measurement_type_concept_id INT64 not null , + operator_concept_id INT64 , + value_as_number FLOAT64 , + value_as_concept_id INT64 , + unit_concept_id INT64 , + range_low FLOAT64 , + range_high FLOAT64 , + provider_id INT64 , + visit_occurrence_id INT64 , + visit_detail_id INT64 , + measurement_source_value STRING , + measurement_source_concept_id INT64 , + unit_source_value STRING , + value_source_value STRING +) +; + + +--HINT DISTRIBUTE_ON_KEY(person_id) +create table note +( + note_id INT64 not null , + person_id INT64 not null , + note_DATE DATE not null , + note_DATETIME DATETIME , + note_type_concept_id INT64 not null , + note_class_concept_id INT64 not null , + note_title STRING , + note_text STRING , + encoding_concept_id INT64 not null , + language_concept_id INT64 not null , + provider_id INT64 , + visit_occurrence_id INT64 , + visit_detail_id INT64 , + note_source_value STRING +) +; + + + +create table note_nlp +( + note_nlp_id INT64 , + note_id INT64 , + section_concept_id INT64 , + snippet STRING , + "offset" STRING , + lexical_variant STRING not null , + note_nlp_concept_id INT64 , + note_nlp_source_concept_id INT64 , + nlp_system STRING , + nlp_DATE DATE not null , + nlp_DATETIME DATETIME , + term_exists STRING , + term_temporal STRING , + term_modifiers STRING +) +; + + +--HINT DISTRIBUTE_ON_KEY(person_id) +create table observation +( + observation_id INT64 not null , + person_id INT64 not null , + observation_concept_id INT64 not null , + observation_DATE DATE not null , + observation_DATETIME DATETIME , + observation_type_concept_id INT64 not null , + value_as_number FLOAT64 , + value_as_string STRING , + value_as_concept_id INT64 , + qualifier_concept_id INT64 , + unit_concept_id INT64 , + provider_id INT64 , + visit_occurrence_id INT64 , + visit_detail_id INT64 , + observation_source_value STRING , + observation_source_concept_id INT64 , + unit_source_value STRING , + qualifier_source_value STRING +) +; + + +create table fact_relationship +( + domain_concept_id_1 INT64 not null , + fact_id_1 INT64 not null , + domain_concept_id_2 INT64 not null , + fact_id_2 INT64 not null , + relationship_concept_id INT64 not null +) +; + + +create table location +( + location_id INT64 not null , + address_1 STRING , + address_2 STRING , + city STRING , + state STRING , + zip STRING , + county STRING , + location_source_value STRING +) +; + + +create table care_site +( + care_site_id INT64 not null , + care_site_name STRING , + place_of_service_concept_id INT64 , + location_id INT64 , + care_site_source_value STRING , + place_of_service_source_value STRING +) +; + + +create table provider +( + provider_id INT64 not null , + provider_name STRING , + npi STRING , + dea STRING , + specialty_concept_id INT64 , + care_site_id INT64 , + year_of_birth INT64 , + gender_concept_id INT64 , + provider_source_value STRING , + specialty_source_value STRING , + specialty_source_concept_id INT64 , + gender_source_value STRING , + gender_source_concept_id INT64 +) +; + + +--HINT DISTRIBUTE_ON_KEY(person_id) +create table payer_plan_period +( + payer_plan_period_id INT64 not null , + person_id INT64 not null , + payer_plan_period_start_DATE DATE not null , + payer_plan_period_end_DATE DATE not null , + payer_concept_id INT64 , + payer_source_value STRING , + payer_source_concept_id INT64 , + plan_concept_id INT64 , + plan_source_value STRING , + plan_source_concept_id INT64 , + sponsor_concept_id INT64 , + sponsor_source_value STRING , + sponsor_source_concept_id INT64 , + family_source_value STRING , + stop_reason_concept_id INT64 , + stop_reason_source_value STRING , + stop_reason_source_concept_id INT64 +) +; + + +create table cost +( + cost_id INT64 not null , + cost_event_id INT64 not null , + cost_domain_id STRING not null , + cost_type_concept_id INT64 not null , + currency_concept_id INT64 , + total_charge FLOAT64 , + total_cost FLOAT64 , + total_paid FLOAT64 , + paid_by_payer FLOAT64 , + paid_by_patient FLOAT64 , + paid_patient_copay FLOAT64 , + paid_patient_coinsurance FLOAT64 , + paid_patient_deductible FLOAT64 , + paid_by_primary FLOAT64 , + paid_ingredient_cost FLOAT64 , + paid_dispensing_fee FLOAT64 , + payer_plan_period_id INT64 , + amount_allowed FLOAT64 , + revenue_code_concept_id INT64 , + revenue_code_source_value STRING , + drg_concept_id INT64 , + drg_source_value STRING +) +; + + +--HINT DISTRIBUTE_ON_KEY(subject_id) +create table cohort +( + cohort_definition_id INT64 not null , + subject_id INT64 not null , + cohort_start_DATE DATE not null , + cohort_end_DATE DATE not null +) +; + + +--HINT DISTRIBUTE_ON_KEY(subject_id) +create table cohort_attribute +( + cohort_definition_id INT64 not null , + subject_id INT64 not null , + cohort_start_DATE DATE not null , + cohort_end_DATE DATE not null , + attribute_definition_id INT64 not null , + value_as_number FLOAT64 , + value_as_concept_id INT64 +) +; + + +--HINT DISTRIBUTE_ON_KEY(person_id) +create table drug_era +( + drug_era_id INT64 not null , + person_id INT64 not null , + drug_concept_id INT64 not null , + drug_era_start_DATE DATE not null , + drug_era_end_DATE DATE not null , + drug_exposure_count INT64 , + gap_days INT64 +) +; + + +--HINT DISTRIBUTE_ON_KEY(person_id) +create table dose_era +( + dose_era_id INT64 not null , + person_id INT64 not null , + drug_concept_id INT64 not null , + unit_concept_id INT64 not null , + dose_value FLOAT64 not null , + dose_era_start_DATE DATE not null , + dose_era_end_DATE DATE not null +) +; + + +--HINT DISTRIBUTE_ON_KEY(person_id) +create table condition_era +( + condition_era_id INT64 not null , + person_id INT64 not null , + condition_concept_id INT64 not null , + condition_era_start_DATE DATE not null , + condition_era_end_DATE DATE not null , + condition_occurrence_count INT64 +) +; diff --git a/scripts/bq_load_mimiciii_custom_mapping.py b/z_more/bq_load_mimiciii_custom_mapping.py similarity index 97% rename from scripts/bq_load_mimiciii_custom_mapping.py rename to z_more/bq_load_mimiciii_custom_mapping.py index 93c428e..128bf80 100644 --- a/scripts/bq_load_mimiciii_custom_mapping.py +++ b/z_more/bq_load_mimiciii_custom_mapping.py @@ -17,8 +17,8 @@ config_default = { - "bq_target_project": "odysseus-mimic-dev", - "bq_target_dataset": "mimiciii_extras_concept", + "bq_target_project": "project...", + "bq_target_dataset": "dataset...", "local_mapping_csv_path": "../../mimic-omop/extras/concept", "files_extension": ".csv", diff --git a/z_more/bq_load_one_time.py b/z_more/bq_load_one_time.py new file mode 100644 index 0000000..e68dcac --- /dev/null +++ b/z_more/bq_load_one_time.py @@ -0,0 +1,173 @@ +# +# Load mimiciii CSV custom mapping from local path to BigQuery +# +# Run from current directory +# + +import os +import sys +import getopt +import json + +# ---------------------------------------------------- +''' + python bq_load_one_time.py +''' +# ---------------------------------------------------- + +config_default = { + + "bq_target_project": "project...", + "bq_target_dataset": "dataset...", + + "local_mapping_csv_path": "../folder...", + "files_extension": ".csv", + "mapping_csv_delimiter": ",", + "mapping_csv_quote": "\\\"", + "files": [ + "filename_without_extension" + ] +} +# if "files" array is empty, all csv in the given directory will be loaded + + +# ---------------------------------------------------- +# read_params() +# ---------------------------------------------------- + +def read_params(): + + print('Read params...') + params = { + "config_file": "optional: indicate '-c' for 'config', json file name. Defaults are hard-coded" + } + + # Parsing command line arguments + try: + opts, args = getopt.getopt(sys.argv[1:],"c:",["config="]) + # if len(opts) == 0: + # raise getopt.GetoptError("read_params() error", "Mandatory argument is missing.") + except getopt.GetoptError: + print("Please indicate correct params:") + print(params) + print("for example:\npython bq_load_one_time.py --config config.conf") + sys.exit(2) + + st = [] + for opt, arg in opts: + + if opt == '-c' or opt == '--config': + if os.path.isfile(arg): + params['config_file'] = arg + else: + params['config_file'] = '' + print(params) + return params + +# ---------------------------------------------------- +# read_config() +# ---------------------------------------------------- + +def read_config(config_file): + + print('Read config...') + config = {} + config_read = {} + + if os.path.isfile(config_file): + with open(config_file) as f: + config_read = json.load(f) + + for k in config_default: + s = config_read.get(k, config_default[k]) + config[k] = s + + print(config) + return config + +''' +---------------------------------------------------- + load_table() + return codes: 0, 1, 2 +---------------------------------------------------- +''' + +def load_table(table, file_path, config): + + return_code = 0 + + bq_table = '{dataset}.{prefix}{table}' + + bq_load_command = \ + "bq --location=US load --replace " + \ + " --source_format=CSV " + \ + " --allow_quoted_newlines=True " + \ + " --skip_leading_rows=1 " + \ + " --field_delimiter=\"{field_delimiter}\" " + \ + " --quote=\"{quote}\" " + \ + " --autodetect " + \ + " {table_name} " + \ + " \"{files_path}\" " + + table_path = bq_table.format(dataset=config['bq_target_dataset'], prefix="", table=table) + + if os.path.isfile(file_path): + bqc = bq_load_command.format( \ + table_name = table_path, \ + files_path = file_path, \ + field_delimiter=config['mapping_csv_delimiter'], \ + quote=config['mapping_csv_quote'] + ) + print('To BQ: ' + bqc) + + try: + os.system(bqc) + except Exception as e: + return_code = 2 # error during execution of the command + raise e + else: + return_code = 1 # file not found + print ('Source file {f} is not found.'.format(f=file_path)) + + return return_code + +''' +---------------------------------------------------- + main() + return code: + 0 = success + lower byte = number of Athena tables failed to load + upper byte = if Custom mapping table failed to load +---------------------------------------------------- +''' + +def main(): + params = read_params() + + config = read_config(params.get('config_file')) + + return_code = 0 + + print('Loading tables...') + + + for entry in os.scandir(config['local_mapping_csv_path']): + if entry.name.endswith(config['files_extension']): + table = entry.name.strip(config['files_extension']) + print('Found: ', table) + if table in config['files'] or len(config['files']) == 0 : + print('Loading: ', table) + rc = load_table(table, entry.path, config) + if rc != 0: + return_code += 1 + continue + + return return_code + +# ---------------------------------------------------- +# go +# ---------------------------------------------------- +return_code = main() + +exit(return_code) + diff --git a/scripts/bq_mimiciii_custom_mapping_00.sql b/z_more/bq_mimiciii_custom_mapping_00.sql similarity index 93% rename from scripts/bq_mimiciii_custom_mapping_00.sql rename to z_more/bq_mimiciii_custom_mapping_00.sql index abbde28..0d8f2bc 100644 --- a/scripts/bq_mimiciii_custom_mapping_00.sql +++ b/z_more/bq_mimiciii_custom_mapping_00.sql @@ -3,7 +3,7 @@ -- mimic-omop/etl/StandardizedVocabularies/CONCEPT/etl.sql -- ------------------------------------------------------------------- -CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciii_extras_concept.concept_iii +CREATE OR REPLACE TABLE `project...`.dataset....concept_iii ( concept_id INT64, concept_name STRING, @@ -15,7 +15,7 @@ CREATE OR REPLACE TABLE `odysseus-mimic-dev`.mimiciii_extras_concept.concept_iii valid_end_date DATE ); -INSERT INTO `odysseus-mimic-dev`.mimiciii_extras_concept.concept_iii +INSERT INTO `project...`.dataset....concept_iii ( concept_id, concept_name, diff --git a/z_more/raw_case055_ecg_lines3.csv b/z_more/raw_case055_ecg_lines3.csv deleted file mode 100644 index 217d1e6..0000000 --- a/z_more/raw_case055_ecg_lines3.csv +++ /dev/null @@ -1,501 +0,0 @@ -row_id,line_1,line_2,line_3 -1,189,189,198 -2,184,187,192 -3,179,185,186 -4,174,183,180 -5,169,181,173 -6,164,178,167 -7,158,176,160 -8,153,174,152 -9,147,171,144 -10,141,168,136 -11,134,166,128 -12,128,163,118 -13,121,159,109 -14,114,156,100 -15,107,153,90 -16,99,149,81 -17,92,146,71 -18,84,143,62 -19,77,139,52 -20,70,136,43 -21,63,133,34 -22,56,130,26 -23,50,127,18 -24,44,124,10 -25,39,121,3 -26,33,119,-3 -27,29,117,-9 -28,25,115,-15 -29,23,113,-19 -30,18,112,-23 -31,16,110,-26 -32,13,109,-29 -33,12,108,-31 -34,10,108,-32 -35,9,107,-33 -36,9,107,-33 -37,8,107,-33 -38,8,107,-32 -39,9,107,-31 -40,9,108,-30 -41,10,108,-28 -42,11,109,-26 -43,12,110,-24 -44,14,110,-22 -45,14,111,-19 -46,17,112,-16 -47,19,113,-14 -48,21,114,-11 -49,23,115,-8 -50,25,116,-5 -51,27,117,-2 -52,29,119,0 -53,31,120,2 -54,33,121,5 -55,34,122,7 -56,36,124,9 -57,38,125,12 -58,39,126,14 -59,41,128,16 -60,42,129,18 -61,43,130,22 -62,44,132,21 -63,45,133,23 -64,46,134,24 -65,47,135,25 -66,47,136,26 -67,48,138,26 -68,48,138,27 -69,48,139,27 -70,48,140,28 -71,48,141,28 -72,48,142,28 -73,48,142,28 -74,48,143,29 -75,48,143,29 -76,48,144,30 -77,48,145,30 -78,49,145,30 -79,49,146,31 -80,49,146,31 -81,49,147,32 -82,49,147,33 -83,50,148,33 -84,51,149,34 -85,52,150,35 -86,53,152,36 -87,55,155,38 -88,57,158,40 -89,60,163,42 -90,64,169,45 -91,67,176,48 -92,72,184,52 -93,76,193,55 -94,81,202,58 -95,85,212,61 -96,89,222,64 -97,94,231,67 -98,97,241,70 -99,101,250,73 -100,105,260,76 -101,110,270,80 -102,114,281,84 -103,119,293,89 -104,125,305,94 -105,130,317,100 -106,135,329,105 -107,140,339,110 -108,144,348,115 -109,147,353,118 -110,149,356,120 -111,150,355,121 -112,149,351,121 -113,147,344,119 -114,144,335,117 -115,140,325,114 -116,136,314,110 -117,131,302,106 -118,126,289,101 -119,121,277,97 -120,116,264,92 -121,110,252,88 -122,105,240,83 -123,99,228,79 -124,93,217,74 -125,88,206,70 -126,82,197,65 -127,77,188,61 -128,72,180,56 -129,67,172,52 -130,62,165,47 -131,57,158,43 -132,52,151,40 -133,48,145,36 -134,44,140,33 -135,41,134,30 -136,37,129,28 -137,35,124,26 -138,32,119,24 -139,30,115,23 -140,29,112,22 -141,28,108,21 -142,27,106,21 -143,27,104,21 -144,27,102,21 -145,28,100,22 -146,28,99,23 -147,29,98,24 -148,30,97,25 -149,31,96,27 -150,32,96,28 -151,33,95,29 -152,34,95,30 -153,35,95,30 -154,35,94,31 -155,36,94,31 -156,36,94,31 -157,36,94,31 -158,36,94,30 -159,36,94,29 -160,35,93,28 -161,35,93,27 -162,34,93,25 -163,33,93,23 -164,32,93,22 -165,31,93,19 -166,30,93,17 -167,28,93,15 -168,27,93,15 -169,26,94,10 -170,25,95,8 -171,24,97,7 -172,25,100,8 -173,28,105,14 -174,34,112,20 -175,43,120,33 -176,56,129,53 -177,74,139,80 -178,96,147,114 -179,124,152,156 -180,155,152,204 -181,190,148,257 -182,228,138,314 -183,268,122,373 -184,306,100,431 -185,343,74,487 -186,376,45,539 -187,404,15,585 -188,426,-14,626 -189,443,-44,661 -190,454,-73,689 -191,457,-104,707 -192,452,-135,713 -193,436,-167,704 -194,408,-199,675 -195,364,-232,621 -196,302,-262,536 -197,219,-291,420 -198,117,-315,274 -199,-1,-335,101 -200,-134,-352,-91 -201,-271,-363,-293 -202,-404,-371,-488 -203,-522,-374,-663 -204,-617,-375,-803 -205,-682,-372,-900 -206,-715,-365,-952 -207,-718,-355,-961 -208,-694,-341,-929 -209,-648,-322,-867 -210,-587,-297,-787 -211,-518,-268,-698 -212,-448,-233,-608 -213,-380,-195,-522 -214,-317,-154,-443 -215,-260,-112,-373 -216,-209,-69,-310 -217,-163,-28,-254 -218,-123,10,-205 -219,-89,47,-162 -220,-58,79,-125 -221,-32,107,-91 -222,-10,130,-62 -223,8,150,-37 -224,24,166,-16 -225,37,178,1 -226,48,188,18 -227,56,196,29 -228,63,202,38 -229,69,206,46 -230,73,210,53 -231,77,212,58 -232,80,214,62 -233,83,215,65 -234,85,216,68 -235,87,217,71 -236,89,218,73 -237,90,219,75 -238,92,220,77 -239,93,220,78 -240,94,221,79 -241,95,221,81 -242,96,222,82 -243,97,222,83 -244,98,223,84 -245,99,223,85 -246,100,223,87 -247,101,224,88 -248,102,224,90 -249,103,225,91 -250,104,225,93 -251,106,225,95 -252,107,226,97 -253,109,226,100 -254,111,227,102 -255,113,227,105 -256,114,228,108 -257,116,228,111 -258,118,229,114 -259,120,229,117 -260,123,229,120 -261,125,230,124 -262,127,230,127 -263,129,230,131 -264,131,230,134 -265,133,231,138 -266,135,231,141 -267,138,231,145 -268,140,231,149 -269,142,231,153 -270,144,231,157 -271,147,231,161 -272,149,231,164 -273,151,232,168 -274,154,232,172 -275,156,232,176 -276,158,232,180 -277,161,231,183 -278,163,231,186 -279,166,231,190 -280,169,231,193 -281,171,231,196 -282,174,231,200 -283,177,231,203 -284,180,230,207 -285,183,230,211 -286,187,230,215 -287,190,230,219 -288,194,230,224 -289,197,230,228 -290,201,230,233 -291,205,229,238 -292,209,229,243 -293,212,229,248 -294,216,228,253 -295,220,228,257 -296,223,227,262 -297,227,227,266 -298,230,226,270 -299,233,225,274 -300,236,224,277 -301,239,224,280 -302,241,223,282 -303,243,222,284 -304,244,221,285 -305,245,220,285 -306,245,219,285 -307,245,218,285 -308,245,217,284 -309,244,216,282 -310,243,214,279 -311,241,213,276 -312,239,211,273 -313,237,210,269 -314,234,208,264 -315,231,207,259 -316,227,205,254 -317,223,203,249 -318,219,201,243 -319,215,199,238 -320,211,198,232 -321,206,196,227 -322,201,194,221 -323,196,192,216 -324,192,190,210 -325,186,188,204 -326,181,185,197 -327,176,183,190 -328,170,180,182 -329,164,177,174 -330,158,175,165 -331,151,172,156 -332,144,169,146 -333,136,165,136 -334,129,162,126 -335,121,159,116 -336,113,156,106 -337,105,152,96 -338,97,149,86 -339,89,146,75 -340,81,142,65 -341,74,139,55 -342,66,136,45 -343,59,133,36 -344,52,130,27 -345,46,127,19 -346,40,124,11 -347,35,122,4 -348,30,120,-1 -349,26,118,-7 -350,22,116,-13 -351,18,114,-17 -352,15,113,-21 -353,12,112,-24 -354,11,111,-27 -355,9,110,-29 -356,8,110,-30 -357,7,109,-31 -358,6,109,-31 -359,6,109,-31 -360,7,109,-30 -361,7,109,-29 -362,8,110,-28 -363,9,110,-26 -364,10,111,-24 -365,12,112,-21 -366,14,113,-19 -367,15,113,-16 -368,17,114,-13 -369,19,115,-10 -370,21,116,-7 -371,23,117,-4 -372,25,118,-1 -373,27,119,1 -374,29,120,4 -375,31,122,7 -376,33,123,9 -377,35,124,12 -378,36,125,15 -379,38,127,17 -380,39,128,19 -381,41,129,21 -382,42,130,23 -383,43,132,25 -384,44,133,26 -385,45,134,28 -386,45,136,29 -387,46,137,30 -388,46,138,30 -389,47,139,31 -390,47,140,31 -391,47,140,32 -392,47,141,32 -393,47,142,32 -394,47,142,33 -395,47,143,33 -396,47,144,33 -397,47,144,34 -398,48,145,34 -399,48,145,34 -400,48,146,35 -401,49,147,36 -402,49,147,36 -403,49,148,37 -404,50,149,38 -405,51,151,38 -406,52,152,40 -407,54,154,41 -408,56,157,43 -409,58,160,45 -410,60,165,47 -411,63,171,50 -412,67,177,52 -413,71,185,55 -414,75,193,58 -415,79,202,61 -416,83,211,64 -417,87,221,67 -418,91,231,70 -419,95,241,73 -420,99,251,76 -421,103,261,79 -422,107,271,83 -423,112,282,87 -424,116,293,91 -425,121,304,95 -426,126,315,100 -427,130,325,105 -428,134,334,109 -429,138,341,114 -430,141,346,117 -431,143,349,120 -432,145,349,122 -433,145,348,122 -434,144,344,121 -435,142,337,119 -436,139,328,117 -437,135,318,113 -438,131,306,109 -439,126,294,105 -440,121,282,100 -441,116,269,96 -442,111,257,92 -443,106,245,87 -444,101,233,83 -445,96,222,78 -446,90,211,74 -447,85,201,69 -448,80,192,65 -449,75,183,60 -450,70,175,56 -451,65,168,52 -452,60,161,48 -453,56,154,44 -454,52,148,41 -455,48,142,37 -456,44,137,31 -457,41,132,32 -458,38,127,30 -459,35,123,28 -460,33,119,27 -461,31,115,26 -462,30,112,25 -463,29,109,24 -464,28,107,24 -465,28,105,25 -466,28,103,25 -467,28,102,26 -468,29,101,27 -469,29,100,28 -470,30,100,29 -471,31,99,31 -472,32,99,32 -473,33,98,30 -474,33,98,31 -475,34,97,34 -476,34,97,35 -477,35,96,35 -478,35,96,35 -479,35,96,34 -480,34,96,34 -481,34,96,30 -482,33,96,29 -483,33,96,30 -484,32,96,28 -485,31,96,26 -486,30,96,24 -487,28,96,22 -488,27,96,20 -489,26,97,17 -490,25,98,16 -491,26,99,15 -492,27,102,16 -493,29,105,18 -494,33,109,23 -495,39,115,40 -496,48,121,43 -497,61,127,62 -498,78,133,89 -499,101,137,123 -500,129,140,164 diff --git a/z_more/run_times.txt b/z_more/run_times.txt index 45985c0..9d25be6 100644 --- a/z_more/run_times.txt +++ b/z_more/run_times.txt @@ -2,1054 +2,170 @@ -- Latest first --- 2021-01-25 - --- Demo, Metrics - -Scripts executed: -2021-01-25 16:02:27.723968 | start... | Done. -2021-01-25 16:04:05.567044 | test/metrics_gen/me_totals_from_conf.sql | Done. -2021-01-25 16:04:46.657039 | test/metrics_gen/me_persons_visits.sql | Done. -2021-01-25 16:15:32.026610 | test/metrics_gen/me_top100_from_conf.sql | Done. -2021-01-25 16:19:18.181827 | test/metrics_gen/me_mapping_rate_from_conf.sql | Done. -2021-01-25 16:20:38.939306 | test/metrics_gen/me_tops_together_from_conf.sql | Done. -Run time: 0:18:11.216931 - --- Demo, UT - -2021-01-25 15:55:21.912923 | start... | Done. -2021-01-25 15:55:25.747468 | test/ut/ut_start.sql | Done. -2021-01-25 15:57:52.391108 | test/ut/ut_basic_gen.sql | Done. -2021-01-25 15:58:05.650929 | test/ut/ut_care_site.sql | Done. -2021-01-25 15:58:29.382913 | test/ut/ut_person.sql | Done. -2021-01-25 15:58:32.869332 | test/ut/ut_death.sql | Done. -Run time: 0:03:10.956973 - --- Demo, ETL - -2021-01-25 15:38:00.542515 | start... | Done. -2021-01-25 15:38:06.692654 | etl/etl/cdm_location.sql | Done. -2021-01-25 15:38:19.525379 | etl/etl/cdm_care_site.sql | Done. -2021-01-25 15:38:39.069990 | etl/etl/cdm_person.sql | Done. -2021-01-25 15:38:50.122467 | etl/etl/cdm_death.sql | Done. -2021-01-25 15:39:05.886970 | etl/etl/lk_vis_part_1.sql | Done. -2021-01-25 15:39:21.139806 | etl/etl/lk_meas_unit.sql | Done. -2021-01-25 15:39:42.090416 | etl/etl/lk_meas_chartevents.sql | Done. -2021-01-25 15:40:01.873271 | etl/etl/lk_meas_labevents.sql | Done. -2021-01-25 15:40:40.553324 | etl/etl/lk_meas_specimen.sql | Done. -2021-01-25 15:40:49.912980 | etl/etl/lk_meas_waveform.sql | Done. -2021-01-25 15:41:32.241283 | etl/etl/lk_vis_part_2.sql | Done. -2021-01-25 15:41:40.386030 | etl/etl/cdm_visit_occurrence.sql | Done. -2021-01-25 15:41:47.428182 | etl/etl/cdm_visit_detail.sql | Done. -2021-01-25 15:42:05.334573 | etl/etl/lk_cond_diagnoses.sql | Done. -2021-01-25 15:42:59.341272 | etl/etl/lk_procedure.sql | Done. -2021-01-25 15:43:15.549578 | etl/etl/lk_observation.sql | Done. -2021-01-25 15:43:24.713556 | etl/etl/cdm_condition_occurrence.sql | Done. -2021-01-25 15:43:35.202638 | etl/etl/cdm_procedure_occurrence.sql | Done. -2021-01-25 15:43:40.430023 | etl/etl/cdm_specimen.sql | Done. -2021-01-25 15:44:03.648278 | etl/etl/cdm_measurement.sql | Done. -2021-01-25 15:44:31.916397 | etl/etl/lk_drug.sql | Done. -2021-01-25 15:44:37.992183 | etl/etl/cdm_drug_exposure.sql | Done. -2021-01-25 15:44:43.592115 | etl/etl/cdm_device_exposure.sql | Done. -2021-01-25 15:44:59.293399 | etl/etl/cdm_observation.sql | Done. -2021-01-25 15:45:41.094215 | etl/etl/cdm_observation_period.sql | Done. -2021-01-25 15:45:51.489281 | etl/etl/cdm_finalize_person.sql | Done. -2021-01-25 15:46:07.037938 | etl/etl/cdm_fact_relationship.sql | Done. -2021-01-25 15:46:39.397303 | etl/etl/cdm_condition_era.sql | Done. -2021-01-25 15:48:01.117558 | etl/etl/cdm_drug_era.sql | Done. -2021-01-25 15:48:51.067124 | etl/etl/cdm_dose_era.sql | Done. -2021-01-25 15:48:56.500304 | etl/etl/cdm_cdm_source.sql | Done. -Run time: 0:10:55.958477 - --- Demo, Staging - -2021-01-25 15:32:23.537120 | start... | Done. -2021-01-25 15:32:33.862225 | etl/staging/st_core.sql | Done. -2021-01-25 15:33:09.001534 | etl/staging/st_hosp.sql | Done. -2021-01-25 15:33:47.254478 | etl/staging/st_icu.sql | Done. -2021-01-25 15:34:03.764207 | etl/staging/st_waveform.sql | Done. -2021-01-25 15:36:21.136849 | etl/staging/voc_copy_to_target_dataset.sql | Done. -Run time: 0:03:57.600825 +-- 2021-02-07 +-- Demo, Import --- 2021-01-21 +2021-02-08 01:26:32.043764 | start... | Done. +2021-02-08 01:33:27.362026 | finish | Done. +Run time: 0:06:55.327274 --- Demo, ETL - -2021-01-21 21:15:27.640712 | start... | Done. -2021-01-21 21:15:34.385690 | etl/etl/cdm_location.sql | Done. -2021-01-21 21:15:46.694509 | etl/etl/cdm_care_site.sql | Done. -2021-01-21 21:16:03.113927 | etl/etl/cdm_person.sql | Done. -2021-01-21 21:16:11.346814 | etl/etl/cdm_death.sql | Done. -2021-01-21 21:16:25.606187 | etl/etl/lk_vis_part_1.sql | Done. -2021-01-21 21:16:54.553595 | etl/etl/lk_meas_labevents.sql | Done. -2021-01-21 21:17:27.650821 | etl/etl/lk_meas_specimen.sql | Done. -2021-01-21 21:17:37.410262 | etl/etl/lk_meas_waveform.sql | Done. -2021-01-21 21:17:53.794449 | etl/etl/lk_meas_chartevents.sql | Done. -2021-01-21 21:18:30.107798 | etl/etl/lk_vis_part_2.sql | Done. -2021-01-21 21:18:36.550315 | etl/etl/cdm_visit_occurrence.sql | Done. -2021-01-21 21:18:41.529688 | etl/etl/cdm_visit_detail.sql | Done. -2021-01-21 21:18:57.567440 | etl/etl/lk_cond_diagnoses.sql | Done. -2021-01-21 21:19:05.936243 | etl/etl/cdm_condition_occurrence.sql | Done. -2021-01-21 21:19:58.687657 | etl/etl/lk_procedure.sql | Done. -2021-01-21 21:20:04.857867 | etl/etl/cdm_procedure_occurrence.sql | Done. -2021-01-21 21:20:09.773823 | etl/etl/cdm_specimen.sql | Done. -2021-01-21 21:20:31.118784 | etl/etl/cdm_measurement.sql | Done. -2021-01-21 21:21:07.838926 | etl/etl/cdm_drug_exposure.sql | Done. -2021-01-21 21:21:12.971177 | etl/etl/cdm_device_exposure.sql | Done. -2021-01-21 21:21:43.229186 | etl/etl/cdm_observation.sql | Done. -2021-01-21 21:22:24.150162 | etl/etl/cdm_observation_period.sql | Done. -2021-01-21 21:22:34.039530 | etl/etl/cdm_finalize_person.sql | Done. -2021-01-21 21:22:48.382316 | etl/etl/cdm_fact_relationship.sql | Done. -2021-01-21 21:23:18.162927 | etl/etl/cdm_condition_era.sql | Done. -2021-01-21 21:24:38.327347 | etl/etl/cdm_drug_era.sql | Done. -2021-01-21 21:25:27.303825 | etl/etl/cdm_dose_era.sql | Done. -2021-01-21 21:25:32.250207 | etl/etl/cdm_cdm_source.sql | Done. -Run time: 0:10:04.610031 - --- Demo, Staging +-- Demo, Unload -2021-01-21 18:53:22.384510 | start... | Done. -2021-01-21 18:53:33.011608 | etl/staging/st_core.sql | Done. -2021-01-21 18:54:14.124312 | etl/staging/st_hosp.sql | Done. -2021-01-21 18:54:32.744718 | etl/staging/st_icu.sql | Done. -2021-01-21 18:54:47.774141 | etl/staging/st_waveform.sql | Done. -2021-01-21 18:57:02.110188 | etl/staging/voc_copy_to_target_dataset.sql | Done. -Run time: 0:03:39.726415 +2021-02-07 20:41:38.834255 | start... | Done. +2021-02-07 20:45:21.650984 | etl/unload/unload_to_atlas_gen.sql | Done. +Run time: 0:03:42.817078 +-- Demo, Metrics --- 2021-01-20 +2021-02-07 20:01:05.015641 | start... | Done. +2021-02-07 20:02:38.931540 | test/metrics_gen/me_totals_from_conf.sql | Done. +2021-02-07 20:03:14.308787 | test/metrics_gen/me_persons_visits.sql | Done. +2021-02-07 20:14:00.675134 | test/metrics_gen/me_top100_from_conf.sql | Done. +2021-02-07 20:17:52.845240 | test/metrics_gen/me_mapping_rate_from_conf.sql | Done. +2021-02-07 20:19:14.291149 | test/metrics_gen/me_tops_together_from_conf.sql | Done. +Run time: 0:18:09.276004 -- Demo, UT -2021-01-21 04:48:14.512791 | start... | Done. -2021-01-21 04:48:17.721578 | test/ut/ut_start.sql | Done. -2021-01-21 04:50:36.758536 | test/ut/ut_basic_gen.sql | Done. -2021-01-21 04:50:49.723441 | test/ut/ut_care_site.sql | Done. -2021-01-21 04:51:17.428874 | test/ut/ut_person.sql | Done. -2021-01-21 04:51:21.739064 | test/ut/ut_death.sql | Done. -Run time: 0:03:07.226658 +2021-02-07 19:53:09.306123 | start... | Done. +2021-02-07 19:53:12.520296 | test/ut/ut_start.sql | Done. +2021-02-07 19:55:23.689475 | test/ut/ut_basic_gen.sql | Done. +2021-02-07 19:55:36.452001 | test/ut/ut_care_site.sql | Done. +2021-02-07 19:56:00.138541 | test/ut/ut_person.sql | Done. +2021-02-07 19:56:03.367457 | test/ut/ut_death.sql | Done. +Run time: 0:02:54.061735 -- Demo, ETL -2021-01-21 04:36:28.421138 | start... | Done. -2021-01-21 04:36:33.839346 | etl/etl/cdm_location.sql | Done. -2021-01-21 04:36:44.464760 | etl/etl/cdm_care_site.sql | Done. -2021-01-21 04:37:01.207410 | etl/etl/cdm_person.sql | Done. -2021-01-21 04:37:09.300628 | etl/etl/cdm_death.sql | Done. -2021-01-21 04:37:25.267274 | etl/etl/lk_vis_part_1.sql | Done. -2021-01-21 04:37:57.266760 | etl/etl/lk_meas_labevents.sql | Done. -2021-01-21 04:38:33.838157 | etl/etl/lk_meas_specimen.sql | Done. -2021-01-21 04:38:44.684838 | etl/etl/lk_meas_waveform.sql | Done. -2021-01-21 04:39:05.789547 | etl/etl/lk_meas_chartevents.sql | Done. -2021-01-21 04:39:42.388572 | etl/etl/lk_vis_part_2.sql | Done. -2021-01-21 04:39:48.223679 | etl/etl/cdm_visit_occurrence.sql | Done. -2021-01-21 04:39:55.823277 | etl/etl/cdm_visit_detail.sql | Done. -2021-01-21 04:40:13.149504 | etl/etl/lk_cond_diagnoses.sql | Done. -2021-01-21 04:40:25.391709 | etl/etl/cdm_condition_occurrence.sql | Done. -2021-01-21 04:41:18.903145 | etl/etl/lk_procedure.sql | Done. -2021-01-21 04:41:25.019445 | etl/etl/cdm_procedure_occurrence.sql | Done. -2021-01-21 04:41:29.881286 | etl/etl/cdm_specimen.sql | Done. -2021-01-21 04:41:57.325492 | etl/etl/cdm_measurement.sql | Done. -2021-01-21 04:42:35.842770 | etl/etl/cdm_drug_exposure.sql | Done. -2021-01-21 04:42:41.750100 | etl/etl/cdm_device_exposure.sql | Done. -2021-01-21 04:43:14.571793 | etl/etl/cdm_observation.sql | Done. -2021-01-21 04:43:55.254799 | etl/etl/cdm_observation_period.sql | Done. -2021-01-21 04:44:04.613675 | etl/etl/cdm_finalize_person.sql | Done. -2021-01-21 04:44:19.437029 | etl/etl/cdm_fact_relationship.sql | Done. -2021-01-21 04:44:53.224229 | etl/etl/cdm_condition_era.sql | Done. -2021-01-21 04:46:23.604346 | etl/etl/cdm_drug_era.sql | Done. -2021-01-21 04:47:11.429710 | etl/etl/cdm_dose_era.sql | Done. -2021-01-21 04:47:16.353953 | etl/etl/cdm_cdm_source.sql | Done. -Run time: 0:10:47.933388 +2021-02-07 19:41:35.878360 | start... | Done. +2021-02-07 19:41:41.669017 | etl/etl/cdm_location.sql | Done. +2021-02-07 19:41:52.630893 | etl/etl/cdm_care_site.sql | Done. +2021-02-07 19:42:07.752318 | etl/etl/cdm_person.sql | Done. +2021-02-07 19:42:16.027570 | etl/etl/cdm_death.sql | Done. +2021-02-07 19:42:29.190679 | etl/etl/lk_vis_part_1.sql | Done. +2021-02-07 19:42:42.195879 | etl/etl/lk_meas_unit.sql | Done. +2021-02-07 19:43:01.587376 | etl/etl/lk_meas_chartevents.sql | Done. +2021-02-07 19:43:24.462840 | etl/etl/lk_meas_labevents.sql | Done. +2021-02-07 19:43:57.178925 | etl/etl/lk_meas_specimen.sql | Done. +2021-02-07 19:44:05.880748 | etl/etl/lk_meas_waveform.sql | Done. +2021-02-07 19:44:41.990284 | etl/etl/lk_vis_part_2.sql | Done. +2021-02-07 19:44:48.569681 | etl/etl/cdm_visit_occurrence.sql | Done. +2021-02-07 19:44:53.615641 | etl/etl/cdm_visit_detail.sql | Done. +2021-02-07 19:45:10.407396 | etl/etl/lk_cond_diagnoses.sql | Done. +2021-02-07 19:46:04.596944 | etl/etl/lk_procedure.sql | Done. +2021-02-07 19:46:21.376305 | etl/etl/lk_observation.sql | Done. +2021-02-07 19:46:29.973043 | etl/etl/cdm_condition_occurrence.sql | Done. +2021-02-07 19:46:38.438664 | etl/etl/cdm_procedure_occurrence.sql | Done. +2021-02-07 19:46:44.079913 | etl/etl/cdm_specimen.sql | Done. +2021-02-07 19:47:06.054711 | etl/etl/cdm_measurement.sql | Done. +2021-02-07 19:47:33.694153 | etl/etl/lk_drug.sql | Done. +2021-02-07 19:47:39.367475 | etl/etl/cdm_drug_exposure.sql | Done. +2021-02-07 19:47:44.596266 | etl/etl/cdm_device_exposure.sql | Done. +2021-02-07 19:48:00.601887 | etl/etl/cdm_observation.sql | Done. +2021-02-07 19:48:39.824057 | etl/etl/cdm_observation_period.sql | Done. +2021-02-07 19:48:49.516694 | etl/etl/cdm_finalize_person.sql | Done. +2021-02-07 19:49:05.600093 | etl/etl/cdm_fact_relationship.sql | Done. +2021-02-07 19:49:37.810369 | etl/etl/cdm_condition_era.sql | Done. +2021-02-07 19:50:51.701499 | etl/etl/cdm_drug_era.sql | Done. +2021-02-07 19:51:38.379853 | etl/etl/cdm_dose_era.sql | Done. +2021-02-07 19:51:43.396040 | etl/etl/cdm_cdm_source.sql | Done. +Run time: 0:10:07.518291 -- Demo, Staging -2021-01-21 03:59:15.737354 | start... | Done. -2021-01-21 03:59:26.201907 | etl/staging/st_core.sql | Done. -2021-01-21 04:00:00.340295 | etl/staging/st_hosp.sql | Done. -2021-01-21 04:00:22.431369 | etl/staging/st_icu.sql | Done. -2021-01-21 04:00:40.629421 | etl/staging/st_waveform.sql | Done. -2021-01-21 04:03:26.769577 | etl/staging/voc_copy_to_target_dataset.sql | Done. -Run time: 0:04:11.032582 +2021-02-07 19:34:00.401342 | start... | Done. +2021-02-07 19:34:10.050103 | etl/staging/st_core.sql | Done. +2021-02-07 19:34:51.299810 | etl/staging/st_hosp.sql | Done. +2021-02-07 19:35:12.756485 | etl/staging/st_icu.sql | Done. +2021-02-07 19:35:23.301847 | etl/staging/st_waveform_poc2.sql | Done. +2021-02-07 19:37:39.038713 | etl/staging/voc_copy_to_target_dataset.sql | Done. +Run time: 0:03:38.637809 -- Demo, DDL -2021-01-21 03:56:34.785897 | start... | Done. -2021-01-21 03:56:56.579198 | etl/ddl/ddl_voc_5_3_1.sql | Done. -2021-01-21 03:57:52.574057 | etl/ddl/ddl_cdm_5_3_1.sql | Done. -Run time: 0:01:17.788478 +2021-02-07 19:32:12.435977 | start... | Done. +2021-02-07 19:32:35.345953 | etl/ddl/ddl_voc_5_3_1.sql | Done. +2021-02-07 19:33:36.640411 | etl/ddl/ddl_cdm_5_3_1.sql | Done. +Run time: 0:01:24.204834 - --- 2021-01-18 - --- Full, UT - -2021-01-18 17:34:53.788672 | start... | Done. -2021-01-18 17:34:57.146184 | test/ut/ut_start.sql | Done. -2021-01-18 17:35:09.006192 | test/ut/ut_care_site.sql | Done. -2021-01-18 17:35:33.343051 | test/ut/ut_person.sql | Done. -2021-01-18 17:42:02.659733 | test/ut/ut_basic_gen.sql | Done. -(~7 min) - --- Full, Unload - -2021-01-18 15:51:34.536104 | start... | Done. -2021-01-18 15:59:04.146309 | etl/unload/unload_to_atlas_gen.sql | Done. -(~7.5 min) +-- 2021-02-01 -- Full, Metrics -2021-01-18 14:45:46.388359 | start... | Done. -2021-01-18 14:47:16.154754 | test/metrics_gen/me_totals_from_conf.sql | Done. -2021-01-18 14:47:53.613119 | test/metrics_gen/me_persons_visits.sql | Done. -2021-01-18 15:04:06.996703 | test/metrics_gen/me_top100_from_conf.sql | Done. -2021-01-18 15:08:54.668460 | test/metrics_gen/me_mapping_rate_from_conf.sql | Done. -2021-01-18 15:10:20.743040 | test/metrics_gen/me_tops_together_from_conf.sql | Done. - - --- Full, ETL - -2021-01-18 04:53:11.659580 | start... | Done. -2021-01-18 04:53:17.092472 | etl/etl/cdm_location.sql | Done. -2021-01-18 04:53:30.135270 | etl/etl/cdm_care_site.sql | Done. -2021-01-18 04:53:52.008494 | etl/etl/cdm_person.sql | Done. -2021-01-18 04:54:01.423160 | etl/etl/cdm_death.sql | Done. -2021-01-18 04:55:42.656222 | etl/etl/lk_meas_chartevents.sql | Done. -2021-01-18 04:56:26.857996 | etl/etl/lk_vis_part_1.sql | Done. -2021-01-18 05:00:05.559277 | etl/etl/lk_meas_labevents.sql | Done. -2021-01-18 08:00:05.568533 | etl/etl/lk_meas_specimen.sql | Done. -2021-01-18 08:00:12.427193 | etl/etl/lk_meas_waveform.sql | Done. -2021-01-18 08:02:53.021383 | etl/etl/lk_vis_part_2.sql | Done. -2021-01-18 08:03:33.123988 | etl/etl/cdm_visit_occurrence.sql | Done. -2021-01-18 08:04:13.376927 | etl/etl/cdm_visit_detail.sql | Done. -2021-01-18 08:05:13.486043 | etl/etl/lk_cond_diagnoses.sql | Done. -2021-01-18 08:05:58.221436 | etl/etl/cdm_condition_occurrence.sql | Done. -2021-01-18 08:08:02.019723 | etl/etl/lk_procedure.sql | Done. -2021-01-18 08:08:30.937060 | etl/etl/cdm_procedure_occurrence.sql | Done. -2021-01-18 08:08:38.394836 | etl/etl/cdm_specimen.sql | Done. -2021-01-18 08:10:37.788172 | etl/etl/cdm_measurement.sql | Done. -2021-01-18 08:13:04.730681 | etl/etl/cdm_drug_exposure.sql | Done. -2021-01-18 08:13:14.474556 | etl/etl/cdm_device_exposure.sql | Done. -2021-01-18 08:15:04.686976 | etl/etl/cdm_observation.sql | Done. -2021-01-18 08:16:17.232775 | etl/etl/cdm_observation_period.sql | Done. -2021-01-18 08:16:31.320647 | etl/etl/cdm_finalize_person.sql | Done. -2021-01-18 08:16:57.528508 | etl/etl/cdm_fact_relationship.sql | Done. -2021-01-18 08:18:55.272266 | etl/etl/cdm_condition_era.sql | Done. -2021-01-18 08:24:08.804015 | etl/etl/cdm_drug_era.sql | Done. -2021-01-18 08:25:55.178817 | etl/etl/cdm_dose_era.sql | Done. -2021-01-18 08:26:00.042461 | etl/etl/cdm_cdm_source.sql | Done. - --- 2021-01-18 - --- Demo, QA - --- Demo, UT - -2021-01-18 16:43:52.915543 | start... | Done. -2021-01-18 16:43:56.345673 | test/ut/ut_start.sql | Done. -2021-01-18 16:44:08.408408 | test/ut/ut_care_site.sql | Done. -2021-01-18 16:44:31.167414 | test/ut/ut_person.sql | Done. -2021-01-18 16:46:45.277413 | test/ut/ut_basic_gen.sql | Done. -(~3 min) - --- Demo, Unload - -2021-01-18 16:03:55.367871 | start... | Done. -2021-01-18 16:08:14.738937 | etl/unload/unload_to_atlas_gen.sql | Done. -(~5.5 min) - --- Demo, UT - -2021-01-18 04:43:26.425302 | start... | Done. -2021-01-18 04:43:29.457063 | test/ut/ut_start.sql | Done. -2021-01-18 04:43:42.330041 | test/ut/ut_care_site.sql | Done. -2021-01-18 04:44:06.253969 | test/ut/ut_person.sql | Done. -2021-01-18 04:44:24.786094 | test/ut/ut_visit_occurrence.sql | Done. -2021-01-18 04:45:02.274196 | test/ut/ut_visit_detail.sql | Done. -2021-01-18 04:45:24.498389 | test/ut/ut_condition_occurrence.sql | Done. - - --- Demo, Metrics - -2021-01-18 04:20:40.705342 | start... | Done. -2021-01-18 04:22:01.716203 | test/metrics_gen/me_totals_from_conf.sql | Done. -2021-01-18 04:22:30.540511 | test/metrics_gen/me_persons_visits.sql | Done. -2021-01-18 04:33:16.657316 | test/metrics_gen/me_top100_from_conf.sql | Done. -2021-01-18 04:37:02.237825 | test/metrics_gen/me_mapping_rate_from_conf.sql | Done. -2021-01-18 04:38:19.849113 | test/metrics_gen/me_tops_together_from_conf.sql | Done. -(~18 min) - --- Demo, ETL - -2021-01-18 04:09:45.553079 | start... | Done. -2021-01-18 04:09:51.007268 | etl/etl/cdm_location.sql | Done. -2021-01-18 04:10:01.336416 | etl/etl/cdm_care_site.sql | Done. -2021-01-18 04:10:18.375472 | etl/etl/cdm_person.sql | Done. -2021-01-18 04:10:26.673001 | etl/etl/cdm_death.sql | Done. -2021-01-18 04:10:44.959061 | etl/etl/lk_meas_chartevents.sql | Done. -2021-01-18 04:10:56.675130 | etl/etl/lk_vis_part_1.sql | Done. -2021-01-18 04:11:19.289788 | etl/etl/lk_meas_labevents.sql | Done. -2021-01-18 04:11:57.027752 | etl/etl/lk_meas_specimen.sql | Done. -2021-01-18 04:12:02.593919 | etl/etl/lk_meas_waveform.sql | Done. -2021-01-18 04:12:44.642847 | etl/etl/lk_vis_part_2.sql | Done. -2021-01-18 04:12:51.927320 | etl/etl/cdm_visit_occurrence.sql | Done. -2021-01-18 04:12:58.078606 | etl/etl/cdm_visit_detail.sql | Done. -2021-01-18 04:13:18.030016 | etl/etl/lk_cond_diagnoses.sql | Done. -2021-01-18 04:13:26.370144 | etl/etl/cdm_condition_occurrence.sql | Done. -2021-01-18 04:14:19.138809 | etl/etl/lk_procedure.sql | Done. -2021-01-18 04:14:24.110284 | etl/etl/cdm_procedure_occurrence.sql | Done. -2021-01-18 04:14:28.975938 | etl/etl/cdm_specimen.sql | Done. -2021-01-18 04:14:49.443905 | etl/etl/cdm_measurement.sql | Done. -2021-01-18 04:15:25.956977 | etl/etl/cdm_drug_exposure.sql | Done. -2021-01-18 04:15:31.096663 | etl/etl/cdm_device_exposure.sql | Done. -2021-01-18 04:16:02.115142 | etl/etl/cdm_observation.sql | Done. -2021-01-18 04:16:36.427838 | etl/etl/cdm_observation_period.sql | Done. -2021-01-18 04:16:45.762407 | etl/etl/cdm_finalize_person.sql | Done. -2021-01-18 04:16:59.820533 | etl/etl/cdm_fact_relationship.sql | Done. -2021-01-18 04:17:31.264207 | etl/etl/cdm_condition_era.sql | Done. -2021-01-18 04:18:46.217480 | etl/etl/cdm_drug_era.sql | Done. -2021-01-18 04:19:29.343022 | etl/etl/cdm_dose_era.sql | Done. -2021-01-18 04:19:34.337346 | etl/etl/cdm_cdm_source.sql | Done. -(~10 min) - --- Demo, Staging - -2021-01-18 03:51:31.033304 | start... | Done. -2021-01-18 03:51:40.439832 | etl/staging/st_core.sql | Done. -2021-01-18 03:52:12.571247 | etl/staging/st_hosp.sql | Done. -2021-01-18 03:52:29.703236 | etl/staging/st_icu.sql | Done. -2021-01-18 03:52:44.550196 | etl/staging/st_waveform.sql | Done. -2021-01-18 03:54:50.783451 | etl/staging/voc_copy_to_target_dataset.sql | Done. -(~4.5 min) - --- Demo, DDL - -2021-01-18 03:47:47.931646 | start... | Done. -2021-01-18 03:48:07.208000 | etl/ddl/ddl_voc_5_3_1.sql | Done. -2021-01-18 03:49:01.018135 | etl/ddl/ddl_cdm_5_3_1.sql | Done. -(~ 1.5 min) - - - --- 2021-01-13 - -Tables are loaded to odysseus-mimic-dev.mimiciv_delivery_demo -2021-01-13 19:32:44.236253 | start... | Done. -2021-01-13 19:39:05.603448 | finish | Done. -(~6.5 min) - - --- 2021-01-11 - --- Demo, ETL - -2021-01-11 16:38:48.358978 | start... | Done. -2021-01-11 16:38:53.882777 | etl/etl/cdm_location.sql | Done. -2021-01-11 16:39:05.974609 | etl/etl/cdm_care_site.sql | Done. -2021-01-11 16:39:23.743262 | etl/etl/cdm_person.sql | Done. -2021-01-11 16:39:32.434887 | etl/etl/cdm_death.sql | Done. -2021-01-11 16:39:55.911108 | etl/etl/lk_vis_adm_transfers.sql | Done. -2021-01-11 16:40:02.209643 | etl/etl/cdm_visit_occurrence.sql | Done. -2021-01-11 16:40:11.601048 | etl/etl/cdm_visit_detail.sql | Done. -2021-01-11 16:40:29.411414 | etl/etl/lk_cond_diagnoses.sql | Done. -2021-01-11 16:41:27.755120 | etl/etl/cdm_procedure_occurrence.sql | Done. -2021-01-11 16:41:51.177565 | etl/etl/lk_meas_labevents.sql | Done. -2021-01-11 16:42:21.297519 | etl/etl/lk_meas_specimen.sql | Done. -2021-01-11 16:42:39.303075 | etl/etl/lk_meas_chartevents.sql | Done. -2021-01-11 16:42:46.364949 | etl/etl/lk_meas_waveform.sql | Done. -2021-01-11 16:42:56.215450 | etl/etl/cdm_condition_occurrence.sql | Done. -2021-01-11 16:43:01.553228 | etl/etl/cdm_specimen.sql | Done. -2021-01-11 16:43:21.887027 | etl/etl/cdm_measurement.sql | Done. -2021-01-11 16:43:53.560169 | etl/etl/cdm_drug_exposure.sql | Done. -2021-01-11 16:43:58.890483 | etl/etl/cdm_device_exposure.sql | Done. -2021-01-11 16:44:30.182720 | etl/etl/cdm_observation.sql | Done. -2021-01-11 16:45:05.706069 | etl/etl/cdm_observation_period.sql | Done. -2021-01-11 16:45:15.332970 | etl/etl/cdm_finalize_person.sql | Done. -2021-01-11 16:45:31.309185 | etl/etl/cdm_fact_relationship.sql | Done. -2021-01-11 16:46:01.791292 | etl/etl/cdm_condition_era.sql | Done. -2021-01-11 16:47:19.610331 | etl/etl/cdm_drug_era.sql | Done. -2021-01-11 16:48:04.483818 | etl/etl/cdm_dose_era.sql | Done. -2021-01-11 16:48:09.822784 | etl/etl/cdm_cdm_source.sql | Done. -(~10 min) - - --- 2020-12-18 +2021-02-01 12:32:36.585224 | start... | Done. +2021-02-01 12:33:56.558121 | test/metrics_gen/me_totals_from_conf.sql | Done. +2021-02-01 12:34:26.312668 | test/metrics_gen/me_persons_visits.sql | Done. +2021-02-01 12:48:16.280970 | test/metrics_gen/me_top100_from_conf.sql | Done. +2021-02-01 12:52:12.620973 | test/metrics_gen/me_mapping_rate_from_conf.sql | Done. +2021-02-01 12:53:31.296555 | test/metrics_gen/me_tops_together_from_conf.sql | Done. +Run time: 0:20:54.711732 -- Full, Unload -2020-12-19 10:52:51.771722 | start... | Done. -2020-12-19 11:01:18.998849 | etl/unload/unload_to_atlas_gen.sql | Done. -(~8.5 min) +2021-02-01 12:21:10.138534 | start... | Done. +2021-02-01 12:30:05.518137 | etl/unload/unload_to_atlas_gen.sql | Done. +Run time: 0:08:55.380064 --- Full, Metrics +-- Full, UT -2020-12-19 03:41:22.668550 | start... | Done. -2020-12-19 03:42:48.951362 | test/metrics_gen/me_totals_from_conf.sql | Done. -2020-12-19 03:43:19.944651 | test/metrics_gen/me_persons_visits.sql | Done. -2020-12-19 04:00:51.208233 | test/metrics_gen/me_top100_from_conf.sql | Done. -2020-12-19 04:06:39.274746 | test/metrics_gen/me_mapping_rate_from_conf.sql | Done. -2020-12-19 04:08:05.459475 | test/metrics_gen/me_tops_together_from_conf.sql | Done. -(~27 min) +2021-02-01 12:13:09.845544 | start... | Done. +2021-02-01 12:13:13.458153 | test/ut/ut_start.sql | Done. +2021-02-01 12:17:42.456686 | test/ut/ut_basic_gen.sql | Done. +2021-02-01 12:17:54.206726 | test/ut/ut_care_site.sql | Done. +2021-02-01 12:18:21.304533 | test/ut/ut_person.sql | Done. +2021-02-01 12:18:25.507846 | test/ut/ut_death.sql | Done. +Run time: 0:05:15.662685 -- Full, ETL -2020-12-19 02:50:41.006042 | start... | Done. -2020-12-19 02:50:46.548931 | etl/etl/cdm_location.sql | Done. -2020-12-19 02:50:59.570136 | etl/etl/cdm_care_site.sql | Done. -2020-12-19 02:51:25.289946 | etl/etl/cdm_person.sql | Done. -2020-12-19 02:51:34.739604 | etl/etl/cdm_death.sql | Done. -2020-12-19 02:53:06.816355 | etl/etl/lk_vis_adm_transfers.sql | Done. -2020-12-19 02:53:22.663470 | etl/etl/cdm_visit_occurrence.sql | Done. -2020-12-19 02:54:07.709813 | etl/etl/cdm_visit_detail.sql | Done. -2020-12-19 02:55:32.871329 | etl/etl/lk_cond_diagnoses.sql | Done. -2020-12-19 02:58:37.445227 | etl/etl/cdm_procedure_occurrence.sql | Done. -2020-12-19 03:04:50.989198 | etl/etl/lk_meas_labevents.sql | Done. -2020-12-19 03:06:48.659878 | etl/etl/lk_meas_specimen.sql | Done. -2020-12-19 03:08:32.100144 | etl/etl/lk_meas_chartevents.sql | Done. -2020-12-19 03:08:39.128121 | etl/etl/lk_meas_waveform.sql | Done. -2020-12-19 03:09:27.357603 | etl/etl/cdm_condition_occurrence.sql | Done. -2020-12-19 03:09:36.368862 | etl/etl/cdm_specimen.sql | Done. -2020-12-19 03:11:49.041100 | etl/etl/cdm_measurement.sql | Done. -2020-12-19 03:14:47.410574 | etl/etl/cdm_drug_exposure.sql | Done. -2020-12-19 03:14:57.847310 | etl/etl/cdm_device_exposure.sql | Done. -2020-12-19 03:17:11.317368 | etl/etl/cdm_observation.sql | Done. -2020-12-19 03:18:42.771454 | etl/etl/cdm_observation_period.sql | Done. -2020-12-19 03:19:13.753584 | etl/etl/cdm_fact_relationship.sql | Done. -2020-12-19 03:22:00.303577 | etl/etl/cdm_condition_era.sql | Done. -2020-12-19 03:29:40.816633 | etl/etl/cdm_drug_era.sql | Done. -2020-12-19 03:31:36.161218 | etl/etl/cdm_dose_era.sql | Done. -2020-12-19 03:31:41.495708 | etl/etl/cdm_cdm_source.sql | Done. -(~41 min) +2021-02-01 11:30:30.496765 | start... | Done. +2021-02-01 11:30:36.102429 | etl/etl/cdm_location.sql | Done. +2021-02-01 11:30:50.125385 | etl/etl/cdm_care_site.sql | Done. +2021-02-01 11:31:14.497912 | etl/etl/cdm_person.sql | Done. +2021-02-01 11:31:25.212210 | etl/etl/cdm_death.sql | Done. +2021-02-01 11:32:06.944537 | etl/etl/lk_vis_part_1.sql | Done. +2021-02-01 11:32:19.594330 | etl/etl/lk_meas_unit.sql | Done. +2021-02-01 11:33:57.424294 | etl/etl/lk_meas_chartevents.sql | Done. +2021-02-01 11:37:52.007524 | etl/etl/lk_meas_labevents.sql | Done. +2021-02-01 11:39:18.935556 | etl/etl/lk_meas_specimen.sql | Done. +2021-02-01 11:39:27.710942 | etl/etl/lk_meas_waveform.sql | Done. +2021-02-01 11:42:06.699599 | etl/etl/lk_vis_part_2.sql | Done. +2021-02-01 11:42:47.131847 | etl/etl/cdm_visit_occurrence.sql | Done. +2021-02-01 11:43:27.519468 | etl/etl/cdm_visit_detail.sql | Done. +2021-02-01 11:44:27.927231 | etl/etl/lk_cond_diagnoses.sql | Done. +2021-02-01 11:46:47.515317 | etl/etl/lk_procedure.sql | Done. +2021-02-01 11:47:41.602241 | etl/etl/lk_observation.sql | Done. +2021-02-01 11:48:38.577531 | etl/etl/cdm_condition_occurrence.sql | Done. +2021-02-01 11:49:13.141245 | etl/etl/cdm_procedure_occurrence.sql | Done. +2021-02-01 11:49:20.329936 | etl/etl/cdm_specimen.sql | Done. +2021-02-01 11:52:03.317777 | etl/etl/cdm_measurement.sql | Done. +2021-02-01 11:53:55.060745 | etl/etl/lk_drug.sql | Done. +2021-02-01 11:54:34.935528 | etl/etl/cdm_drug_exposure.sql | Done. +2021-02-01 11:54:44.607862 | etl/etl/cdm_device_exposure.sql | Done. +2021-02-01 11:55:42.335365 | etl/etl/cdm_observation.sql | Done. +2021-02-01 11:57:02.351234 | etl/etl/cdm_observation_period.sql | Done. +2021-02-01 11:57:17.825421 | etl/etl/cdm_finalize_person.sql | Done. +2021-02-01 11:57:47.625918 | etl/etl/cdm_fact_relationship.sql | Done. +2021-02-01 12:00:27.789578 | etl/etl/cdm_condition_era.sql | Done. +2021-02-01 12:06:29.135312 | etl/etl/cdm_drug_era.sql | Done. +2021-02-01 12:08:04.816710 | etl/etl/cdm_dose_era.sql | Done. +2021-02-01 12:08:09.819447 | etl/etl/cdm_cdm_source.sql | Done. +Run time: 0:37:39.323269 -- Full, Staging -2020-12-19 02:33:57.658756 | start... | Done. -2020-12-19 02:34:43.963411 | etl/staging/st_core.sql | Done. -2020-12-19 02:40:05.572521 | etl/staging/st_hosp.sql | Done. -2020-12-19 02:44:06.008928 | etl/staging/st_icu.sql | Done. -2020-12-19 02:44:23.726455 | etl/staging/st_waveform.sql | Done. -2020-12-19 02:46:52.022472 | etl/staging/voc_copy_to_target_dataset.sql | Done. -(~13 min) +2021-02-01 11:18:53.333955 | start... | Done. +2021-02-01 11:19:25.558111 | etl/staging/st_core.sql | Done. +2021-02-01 11:23:09.853351 | etl/staging/st_hosp.sql | Done. +2021-02-01 11:25:46.888626 | etl/staging/st_icu.sql | Done. +2021-02-01 11:25:56.861233 | etl/staging/st_waveform_poc2.sql | Done. +2021-02-01 11:28:10.779585 | etl/staging/voc_copy_to_target_dataset.sql | Done. +Run time: 0:09:17.446043 -- Full, DDL -2020-12-19 02:31:46.178230 | start... | Done. -2020-12-19 02:32:08.492265 | etl/ddl/ddl_voc_5_3_1.sql | Done. -2020-12-19 02:33:17.627992 | etl/ddl/ddl_cdm_5_3_1.sql | Done. - - --- 2020-12-17 - --- Demo, Metrics - -2020-12-17 19:47:25.230713 | start... | Done. -2020-12-17 19:48:54.592341 | test/metrics_gen/me_totals_from_conf.sql | Done. -2020-12-17 19:49:31.113126 | test/metrics_gen/me_persons_visits.sql | Done. -2020-12-17 20:05:04.879379 | test/metrics_gen/me_top100_from_conf.sql | Done. -2020-12-17 20:10:03.222425 | test/metrics_gen/me_mapping_rate_from_conf.sql | Done. -2020-12-17 20:11:30.438142 | test/metrics_gen/me_tops_together_from_conf.sql | Done. -(all ~24 min, top 15:33, MR 4:99) - --- Demo, unload to Atlas - -2020-12-17 19:16:32.750256 | start... | Done. -2020-12-17 19:20:48.412555 | etl/unload/unload_to_atlas_gen.sql | Done. - --- Demo, ETL - -2020-12-17 19:06:34.452205 | start... | Done. -2020-12-17 19:06:39.456641 | etl/etl/cdm_location.sql | Done. -2020-12-17 19:06:50.839769 | etl/etl/cdm_care_site.sql | Done. -2020-12-17 19:07:06.463266 | etl/etl/cdm_person.sql | Done. -2020-12-17 19:07:14.470678 | etl/etl/cdm_death.sql | Done. -2020-12-17 19:07:41.475591 | etl/etl/lk_vis_adm_transfers.sql | Done. -2020-12-17 19:07:47.699529 | etl/etl/cdm_visit_occurrence.sql | Done. -2020-12-17 19:07:58.351540 | etl/etl/cdm_visit_detail.sql | Done. -2020-12-17 19:08:23.006803 | etl/etl/cdm_condition_occurrence.sql | Done. -2020-12-17 19:09:26.078770 | etl/etl/cdm_procedure_occurrence.sql | Done. -2020-12-17 19:09:50.742243 | etl/etl/lk_meas_labevents.sql | Done. -2020-12-17 19:10:28.343214 | etl/etl/lk_meas_specimen.sql | Done. -2020-12-17 19:10:38.002598 | etl/etl/lk_meas_chartevents.sql | Done. -2020-12-17 19:10:47.011433 | etl/etl/lk_meas_waveform.sql | Done. -2020-12-17 19:10:51.970375 | etl/etl/cdm_specimen.sql | Done. -2020-12-17 19:11:18.305917 | etl/etl/cdm_measurement.sql | Done. -2020-12-17 19:11:57.707589 | etl/etl/cdm_drug_exposure.sql | Done. -2020-12-17 19:12:02.751585 | etl/etl/cdm_device_exposure.sql | Done. -2020-12-17 19:12:27.754805 | etl/etl/cdm_observation.sql | Done. -2020-12-17 19:13:02.827783 | etl/etl/cdm_observation_period.sql | Done. -2020-12-17 19:13:19.081134 | etl/etl/cdm_fact_relationship.sql | Done. -2020-12-17 19:13:50.858918 | etl/etl/cdm_condition_era.sql | Done. -2020-12-17 19:15:14.874007 | etl/etl/cdm_drug_era.sql | Done. -2020-12-17 19:16:00.205091 | etl/etl/cdm_dose_era.sql | Done. -2020-12-17 19:16:05.035603 | etl/etl/cdm_cdm_source.sql | Done. -(~9.5 min) - --- Demo, Staging - -2020-12-17 18:38:35.890201 | start... | Done. -2020-12-17 18:38:45.221926 | etl/staging/st_core.sql | Done. -2020-12-17 18:39:21.801790 | etl/staging/st_hosp.sql | Done. -2020-12-17 18:39:44.467059 | etl/staging/st_icu.sql | Done. -2020-12-17 18:40:00.154143 | etl/staging/st_waveform.sql | Done. -2020-12-17 18:42:43.423947 | etl/staging/voc_copy_to_target_dataset.sql | Done. - --- Demo, DDL - -2020-12-17 18:33:54.526555 | start... | Done. -2020-12-17 18:34:14.484415 | etl/ddl/ddl_voc_5_3_1.sql | Done. -2020-12-17 18:35:09.203978 | etl/ddl/ddl_cdm_5_3_1.sql | Done. - - --- 2020-12-07 - --- Full, ETL - -2020-12-07 19:31:37.212251 | start... | Done. -2020-12-07 19:33:03.010728 | test/metrics_gen/me_totals_from_conf.sql | Done. -2020-12-07 19:33:37.045907 | test/metrics_gen/me_persons_visits.sql | Done. -2020-12-07 19:52:55.223610 | test/metrics_gen/me_top100_from_conf.sql | Done. -2020-12-07 19:58:35.608506 | test/metrics_gen/me_mapping_rate_from_conf.sql | Done. -2020-12-07 20:00:02.258063 | test/metrics_gen/me_tops_together_from_conf.sql | Done. -(30 min???) - --- Full, ETL - -2020-12-07 19:02:04.422818 | start... | Done. -2020-12-07 19:03:53.319370 | etl/etl/cdm_observation.sql | Done. -2020-12-07 19:05:27.370936 | etl/etl/cdm_observation_period.sql | Done. -2020-12-07 19:05:59.089848 | etl/etl/cdm_fact_relationship.sql | Done. -2020-12-07 19:09:04.014248 | etl/etl/cdm_condition_era.sql | Done. -2020-12-07 19:16:28.171653 | etl/etl/cdm_drug_era.sql | Done. -2020-12-07 19:18:29.999482 | etl/etl/cdm_dose_era.sql | Done. -2020-12-07 19:18:35.475699 | etl/etl/cdm_cdm_source.sql | Done. -(~16 min) - -2020-12-07 18:45:55.980741 | start... | Done. -2020-12-07 18:49:08.655358 | etl/etl/cdm_drug_exposure.sql | Done. -2020-12-07 18:49:16.747375 | etl/etl/cdm_device_exposure.sql | Done. -(4 min) - -2020-12-07 18:10:58.922181 | start... | Done. -2020-12-07 18:11:05.447952 | etl/etl/cdm_location.sql | Done. -2020-12-07 18:11:19.742011 | etl/etl/cdm_care_site.sql | Done. -2020-12-07 18:11:42.062463 | etl/etl/cdm_person.sql | Done. -2020-12-07 18:11:52.298663 | etl/etl/cdm_death.sql | Done. -2020-12-07 18:13:56.181460 | etl/etl/lk_vis_adm_transfers.sql | Done. -2020-12-07 18:14:17.473804 | etl/etl/cdm_visit_occurrence.sql | Done. -2020-12-07 18:15:01.848981 | etl/etl/cdm_visit_detail.sql | Done. -2020-12-07 18:17:18.769995 | etl/etl/cdm_condition_occurrence.sql | Done. -2020-12-07 18:20:31.838979 | etl/etl/cdm_procedure_occurrence.sql | Done. -2020-12-07 18:25:08.556831 | etl/etl/lk_meas_labevents.sql | Done. -2020-12-07 18:26:31.474257 | etl/etl/lk_meas_specimen.sql | Done. -2020-12-07 18:28:02.937824 | etl/etl/lk_meas_chartevents.sql | Done. -2020-12-07 18:28:09.799680 | etl/etl/lk_meas_waveform.sql | Done. -2020-12-07 18:28:18.284535 | etl/etl/cdm_specimen.sql | Done. -2020-12-07 18:30:14.450944 | etl/etl/cdm_measurement.sql | Done. -2020-12-07 18:33:16.785919 | etl/etl/cdm_drug_exposure.sql | Done. -2020-12-07 18:33:25.051489 | etl/etl/cdm_device_exposure.sql | Error: See query No 2 -(~24 min) - --- Demo, Unload - -2020-12-07 18:13:04.555612 | start... | Done. -2020-12-07 18:17:24.310792 | etl/unload/unload_to_atlas_gen.sql | Done. -(~4 min) - --- Demo, ETL - -2020-12-07 17:52:21.077727 | start... | Done. -2020-12-07 17:52:25.985666 | etl/etl/cdm_location.sql | Done. -2020-12-07 17:52:37.842696 | etl/etl/cdm_care_site.sql | Done. -2020-12-07 17:52:53.894965 | etl/etl/cdm_person.sql | Done. -2020-12-07 17:53:03.211364 | etl/etl/cdm_death.sql | Done. -2020-12-07 17:53:28.444292 | etl/etl/lk_vis_adm_transfers.sql | Done. -2020-12-07 17:53:34.815964 | etl/etl/cdm_visit_occurrence.sql | Done. -2020-12-07 17:53:44.322791 | etl/etl/cdm_visit_detail.sql | Done. -2020-12-07 17:54:06.257835 | etl/etl/cdm_condition_occurrence.sql | Done. -2020-12-07 17:55:09.243618 | etl/etl/cdm_procedure_occurrence.sql | Done. -2020-12-07 17:55:36.940027 | etl/etl/lk_meas_labevents.sql | Done. -2020-12-07 17:56:09.525831 | etl/etl/lk_meas_specimen.sql | Done. -2020-12-07 17:56:18.223890 | etl/etl/lk_meas_chartevents.sql | Done. -2020-12-07 17:56:26.052817 | etl/etl/lk_meas_waveform.sql | Done. -2020-12-07 17:56:32.292292 | etl/etl/cdm_specimen.sql | Done. -2020-12-07 17:56:56.336967 | etl/etl/cdm_measurement.sql | Done. -2020-12-07 17:57:35.004334 | etl/etl/cdm_drug_exposure.sql | Done. -2020-12-07 17:57:40.135100 | etl/etl/cdm_device_exposure.sql | Done. -2020-12-07 17:58:07.692068 | etl/etl/cdm_observation.sql | Done. -2020-12-07 17:58:44.732982 | etl/etl/cdm_observation_period.sql | Done. -2020-12-07 17:58:59.291317 | etl/etl/cdm_fact_relationship.sql | Done. -2020-12-07 17:59:29.892637 | etl/etl/cdm_condition_era.sql | Done. -2020-12-07 18:00:57.175484 | etl/etl/cdm_drug_era.sql | Done. -2020-12-07 18:01:51.194249 | etl/etl/cdm_dose_era.sql | Done. -2020-12-07 18:01:56.237400 | etl/etl/cdm_cdm_source.sql | Done. -(~10 min) - --- Demo, ETL - -2020-12-07 17:38:06.532486 | start... | Done. -2020-12-07 17:38:13.229217 | etl/etl/cdm_location.sql | Done. -2020-12-07 17:38:25.632340 | etl/etl/cdm_care_site.sql | Done. -2020-12-07 17:38:41.395910 | etl/etl/cdm_person.sql | Done. -2020-12-07 17:38:50.070199 | etl/etl/cdm_death.sql | Done. -2020-12-07 17:39:21.740361 | etl/etl/lk_vis_adm_transfers.sql | Done. -2020-12-07 17:39:27.811185 | etl/etl/cdm_visit_occurrence.sql | Done. -2020-12-07 17:39:37.454064 | etl/etl/cdm_visit_detail.sql | Done. -2020-12-07 17:40:02.458178 | etl/etl/cdm_condition_occurrence.sql | Done. -2020-12-07 17:41:04.323727 | etl/etl/cdm_procedure_occurrence.sql | Done. -2020-12-07 17:41:30.748272 | etl/etl/lk_meas_labevents.sql | Done. -2020-12-07 17:42:03.579935 | etl/etl/lk_meas_specimen.sql | Done. -2020-12-07 17:42:14.734223 | etl/etl/lk_meas_chartevents.sql | Done. -2020-12-07 17:42:21.300709 | etl/etl/lk_meas_waveform.sql | Done. -2020-12-07 17:42:27.647954 | etl/etl/cdm_specimen.sql | Done. -2020-12-07 17:42:49.937503 | etl/etl/cdm_measurement.sql | Done. -2020-12-07 17:43:28.469068 | etl/etl/cdm_drug_exposure.sql | Done. -2020-12-07 17:43:34.604793 | etl/etl/cdm_device_exposure.sql | Done. -2020-12-07 17:43:59.580918 | etl/etl/cdm_observation.sql | Done. -2020-12-07 17:44:34.213278 | etl/etl/cdm_observation_period.sql | Done. -2020-12-07 17:45:26.187851 | etl/etl/cdm_fact_relationship.sql | Done. -2020-12-07 17:45:58.925400 | etl/etl/cdm_condition_era.sql | Done. -2020-12-07 17:47:24.548323 | etl/etl/cdm_drug_era.sql | Done. -2020-12-07 17:48:13.212369 | etl/etl/cdm_dose_era.sql | Done. -2020-12-07 17:48:18.369771 | etl/etl/cdm_cdm_source.sql | Done. -(~10 min) - --- Demo, QA - -2020-12-07 15:24:20.156656 | start... | Done. -2020-12-07 15:24:57.210549 | test/qa/metric_total_row_count.sql | Done. -2020-12-07 15:25:00.792872 | test/qa/qa_start.sql | Done. -2020-12-07 15:25:05.423810 | test/qa/qa_care_site.sql | Done. -2020-12-07 15:25:08.823390 | test/qa/qa_person.sql | Done. -2020-12-07 15:25:12.627311 | test/qa/qa_visit_occurrence.sql | Done. -2020-12-07 15:25:24.064326 | test/qa/qa_visit_detail.sql | Done. - - --- 2020-12-04 - --- Demo, Metrics - -2020-12-05 01:56:44.933372 | start... | Done. -2020-12-05 01:58:09.593740 | test/metrics_gen/me_totals_from_conf.sql | Done. -2020-12-05 02:03:00.480584 | test/metrics_gen/me_mapping_rate_from_conf.sql | Done. -2020-12-05 02:17:28.905073 | test/metrics_gen/me_top100_from_conf.sql | Done. -2020-12-05 02:18:52.433699 | test/metrics_gen/me_tops_together_from_conf.sql | Done. -(all 22:08, top100 14:28, MR 4:51) - -2020-12-05 00:51:21.821751 | start... | Done. -2020-12-05 00:56:43.645901 | test/metrics_gen/me_mapping_rate_from_conf.sql | Done. -2020-12-05 01:14:46.731921 | test/metrics_gen/me_top100_from_conf.sql | Done. -2020-12-05 01:16:11.255781 | test/metrics_gen/me_tops_together_from_conf.sql | Done. -(all 24:50, top100 18:03, MR 5:22) - - --- 2020-12-02 - --- Demo, Metrics - -2020-12-03 03:19:53.664279 | test/metrics_gen/me_totals_from_conf.sql | Done. -2020-12-03 03:21:03.499813 | test/metrics_gen/me_persons_visits.sql | Done. -2020-12-03 03:37:01.949889 | test/metrics_gen/me_top100_from_conf.sql | Done. -2020-12-03 03:43:39.145102 | test/metrics_gen/me_mapping_rate_from_conf.sql | Done. -2020-12-03 03:45:01.449340 | test/metrics_gen/me_tops_together_from_conf.sql | Done. -(all ~25 min, top100 ~16 min, MR ~7 min) - --- Demo, UT - -2020-12-03 03:14:44.857165 | test/ut/ut_start.sql | Done. -2020-12-03 03:15:00.543907 | test/ut/ut_care_site.sql | Done. -2020-12-03 03:15:34.827997 | test/ut/ut_person.sql | Done. -2020-12-03 03:15:57.567634 | test/ut/ut_cdm_visit_occurrence.sql | Done. -2020-12-03 03:16:44.564206 | test/ut/ut_cdm_visit_detail.sql | Done. -(2 min) - --- Demo, ETL - -2020-12-03 03:02:14.309126 | etl/etl/cdm_location.sql | Done. -2020-12-03 03:02:30.099628 | etl/etl/cdm_care_site.sql | Done. -2020-12-03 03:02:47.675185 | etl/etl/cdm_person.sql | Done. -2020-12-03 03:02:57.711920 | etl/etl/cdm_death.sql | Done. -2020-12-03 03:03:29.138501 | etl/etl/lk_vis_adm_transfers.sql | Done. -2020-12-03 03:03:35.617203 | etl/etl/cdm_visit_occurrence.sql | Done. -2020-12-03 03:03:46.604226 | etl/etl/cdm_visit_detail.sql | Done. -2020-12-03 03:04:16.132708 | etl/etl/cdm_condition_occurrence.sql | Done. -2020-12-03 03:05:31.793818 | etl/etl/cdm_procedure_occurrence.sql | Done. -2020-12-03 03:06:02.068299 | etl/etl/lk_meas_labevents.sql | Done. -2020-12-03 03:06:38.139914 | etl/etl/lk_meas_specimen.sql | Done. -2020-12-03 03:06:49.644002 | etl/etl/lk_meas_chartevents.sql | Done. -2020-12-03 03:06:56.461640 | etl/etl/lk_meas_waveform.sql | Done. -2020-12-03 03:07:02.155629 | etl/etl/cdm_specimen.sql | Done. -2020-12-03 03:07:26.947619 | etl/etl/cdm_measurement.sql | Done. -2020-12-03 03:08:11.389124 | etl/etl/cdm_drug_exposure.sql | Done. -2020-12-03 03:08:16.736200 | etl/etl/cdm_device_exposure.sql | Done. -2020-12-03 03:08:52.017788 | etl/etl/cdm_observation.sql | Done. -2020-12-03 03:09:37.194630 | etl/etl/cdm_observation_period.sql | Done. -2020-12-03 03:09:57.319868 | etl/etl/cdm_fact_relationship.sql | Done. -2020-12-03 03:10:33.953524 | etl/etl/cdm_condition_era.sql | Done. -2020-12-03 03:12:07.044815 | etl/etl/cdm_drug_era.sql | Done. -2020-12-03 03:12:56.178168 | etl/etl/cdm_dose_era.sql | Done. -2020-12-03 03:13:01.103325 | etl/etl/cdm_cdm_source.sql | Done. -(~11 min) - --- Demo, Metrics - -2020-12-03 01:54:33.301789 | test/metrics_gen/me_totals_from_conf.sql | Done. -2020-12-03 01:55:08.820033 | test/metrics_gen/me_persons_visits.sql | Done. -2020-12-03 02:09:22.140762 | test/metrics_gen/me_top100_from_conf.sql | Done. -2020-12-03 02:14:45.236543 | test/metrics_gen/me_mapping_rate_from_conf.sql | Done. -2020-12-03 02:16:04.834676 | test/metrics_gen/me_tops_together_from_conf.sql | Done. -(all ~20 min, top100 ~14min, MR ~5 min) - --- Demo, UT - -2020-12-03 01:43:50.844507 | test/ut/ut_start.sql | Done. -2020-12-03 01:44:03.484421 | test/ut/ut_care_site.sql | Done. -2020-12-03 01:44:31.221541 | test/ut/ut_person.sql | Done. -2020-12-03 01:44:52.240972 | test/ut/ut_cdm_visit_occurrence.sql | Done. -2020-12-03 01:45:33.193454 | test/ut/ut_cdm_visit_detail.sql | Done. -(~2 min) - --- Demo, ETL - -2020-12-03 01:00:33.027013 | etl/etl/cdm_location.sql | Done. -2020-12-03 01:00:45.261413 | etl/etl/cdm_care_site.sql | Done. -2020-12-03 01:01:02.677052 | etl/etl/cdm_person.sql | Done. -2020-12-03 01:01:11.577053 | etl/etl/cdm_death.sql | Done. -2020-12-03 01:01:40.154344 | etl/etl/lk_vis_adm_transfers.sql | Done. -2020-12-03 01:01:46.854374 | etl/etl/cdm_visit_occurrence.sql | Done. -2020-12-03 01:01:56.577294 | etl/etl/cdm_visit_detail.sql | Done. -2020-12-03 01:02:21.799467 | etl/etl/cdm_condition_occurrence.sql | Done. -2020-12-03 01:03:21.746930 | etl/etl/cdm_procedure_occurrence.sql | Done. -2020-12-03 01:03:54.032926 | etl/etl/lk_meas_labevents.sql | Done. -2020-12-03 01:04:23.530683 | etl/etl/lk_meas_specimen.sql | Done. -2020-12-03 01:04:32.450700 | etl/etl/lk_meas_chartevents.sql | Done. -2020-12-03 01:04:40.632715 | etl/etl/lk_meas_waveform.sql | Done. -2020-12-03 01:04:45.848219 | etl/etl/cdm_specimen.sql | Done. -2020-12-03 01:05:07.072320 | etl/etl/cdm_measurement.sql | Done. -2020-12-03 01:05:42.877597 | etl/etl/cdm_drug_exposure.sql | Done. -2020-12-03 01:05:48.015787 | etl/etl/cdm_device_exposure.sql | Done. -2020-12-03 01:06:16.146768 | etl/etl/cdm_observation.sql | Done. -2020-12-03 01:06:53.344285 | etl/etl/cdm_observation_period.sql | Done. -2020-12-03 01:07:08.280825 | etl/etl/cdm_fact_relationship.sql | Done. -2020-12-03 01:07:41.522680 | etl/etl/cdm_condition_era.sql | Done. -2020-12-03 01:09:03.753540 | etl/etl/cdm_drug_era.sql | Done. -2020-12-03 01:09:54.246213 | etl/etl/cdm_dose_era.sql | Done. -2020-12-03 01:09:59.468184 | etl/etl/cdm_cdm_source.sql | Done. -(10 min) - - - --- Demo, Metrics_gen - -2020-12-02 14:49:46.186673 | test/metrics_gen/me_totals_from_conf.sql | Done. -2020-12-02 14:50:21.889959 | test/metrics_gen/me_persons_visits.sql | Done. -2020-12-02 15:05:28.521098 | test/metrics_gen/me_top100_from_conf.sql | Done. -2020-12-02 15:12:24.960984 | test/metrics_gen/me_mapping_rate_from_conf.sql | Done. -2020-12-02 15:14:05.699537 | test/metrics_gen/me_tops_together_from_conf.sql | Done. -(all ~30 min, top100 ~15 min, MR ~7 min) - --- Demo, UT - -2020-12-02 08:09:04.388096 | test/ut/ut_start.sql | Done. -2020-12-02 08:09:18.507882 | test/ut/ut_care_site.sql | Done. -2020-12-02 08:09:46.419874 | test/ut/ut_person.sql | Done. -2020-12-02 08:10:07.920790 | test/ut/ut_cdm_visit_occurrence.sql | Done. -2020-12-02 08:10:53.145396 | test/ut/ut_cdm_visit_detail.sql | Done. -report_id report_starttime table_id test_type field_name criteria_json test_passed - 2020-12-02 05:10:48 cdm_visit_detail foreign key preceding_visit_detail_id false - --- Demo, ETL - -2020-12-02 07:54:54.068203 | etl/etl/cdm_location.sql | Done. -2020-12-02 07:55:05.784760 | etl/etl/cdm_care_site.sql | Done. -2020-12-02 07:55:23.970933 | etl/etl/cdm_person.sql | Done. -2020-12-02 07:55:33.514225 | etl/etl/cdm_death.sql | Done. -2020-12-02 07:56:04.442111 | etl/etl/lk_vis_adm_transfers.sql | Done. -2020-12-02 07:56:10.719905 | etl/etl/cdm_visit_occurrence.sql | Done. -2020-12-02 07:56:20.393553 | etl/etl/cdm_visit_detail.sql | Done. -2020-12-02 07:56:48.236077 | etl/etl/cdm_condition_occurrence.sql | Done. -2020-12-02 07:57:52.411296 | etl/etl/cdm_procedure_occurrence.sql | Done. -2020-12-02 07:58:19.155634 | etl/etl/lk_meas_labevents.sql | Done. -2020-12-02 07:58:51.348281 | etl/etl/lk_meas_specimen.sql | Done. -2020-12-02 07:59:00.016570 | etl/etl/lk_meas_chartevents.sql | Done. -2020-12-02 07:59:06.852441 | etl/etl/lk_meas_waveform.sql | Done. -2020-12-02 07:59:11.589604 | etl/etl/cdm_specimen.sql | Done. -2020-12-02 07:59:35.236726 | etl/etl/cdm_measurement.sql | Done. -2020-12-02 08:00:12.786783 | etl/etl/cdm_drug_exposure.sql | Done. -2020-12-02 08:00:19.161196 | etl/etl/cdm_device_exposure.sql | Done. -2020-12-02 08:00:49.208805 | etl/etl/cdm_observation.sql | Done. -2020-12-02 08:01:33.683648 | etl/etl/cdm_observation_period.sql | Done. -2020-12-02 08:01:50.625117 | etl/etl/cdm_fact_relationship.sql | Done. -2020-12-02 08:02:28.271282 | etl/etl/cdm_condition_era.sql | Done. -2020-12-02 08:04:01.810171 | etl/etl/cdm_drug_era.sql | Done. -2020-12-02 08:04:53.399940 | etl/etl/cdm_dose_era.sql | Done. -2020-12-02 08:04:59.578403 | etl/etl/cdm_cdm_source.sql | Done. -(10 min) - - --- 2020-11-30 - --- Full, Metrics - -2020-11-30 14:05:19.695627 | test/metrics_gen/me_totals_from_conf.sql | Done. -2020-11-30 14:05:55.159937 | test/metrics_gen/me_persons_visits.sql | Done. -2020-11-30 17:08:09.586477 | test/metrics_gen/me_top100_from_conf.sql | Done. -2020-11-30 17:15:18.937162 | test/metrics_gen/me_mapping_rate_from_conf.sql | Done. -2020-11-30 17:17:08.466653 | test/metrics_gen/me_tops_together_from_conf.sql | Done. -(3:20:00) -(top 100 - 3 hours!) - --- Full, Unload - -2020-11-30 11:38:24.708656 | etl/unload/unload_to_atlas_gen.sql | Done. -(~20 min) - --- Full, ETL (expected: 45 min) - -2020-11-30 11:04:25.865613 | etl/etl/cdm_observation.sql | Done. -2020-11-30 11:06:09.984182 | etl/etl/cdm_observation_period.sql | Done. -2020-11-30 11:06:43.140302 | etl/etl/cdm_fact_relationship.sql | Done. -2020-11-30 11:09:45.147232 | etl/etl/cdm_condition_era.sql | Done. -2020-11-30 11:17:18.616060 | etl/etl/cdm_drug_era.sql | Done. -2020-11-30 11:19:24.438191 | etl/etl/cdm_dose_era.sql | Done. -2020-11-30 11:19:30.984810 | etl/etl/cdm_cdm_source.sql | Done. -(~18 min) - -2020-11-30 10:20:26.583522 | etl/etl/cdm_location.sql | Done. -2020-11-30 10:20:39.571305 | etl/etl/cdm_care_site.sql | Done. -2020-11-30 10:21:04.990988 | etl/etl/cdm_person.sql | Done. -2020-11-30 10:21:49.907182 | etl/etl/cdm_visit_occurrence.sql | Done. -2020-11-30 10:24:43.326156 | etl/etl/cdm_visit_detail.sql | Done. -2020-11-30 10:24:52.622860 | etl/etl/cdm_death.sql | Done. -2020-11-30 10:27:09.602383 | etl/etl/cdm_condition_occurrence.sql | Done. -2020-11-30 10:31:23.872452 | etl/etl/cdm_procedure_occurrence.sql | Done. -2020-11-30 10:35:26.047448 | etl/etl/lk_meas_labevents.sql | Done. -2020-11-30 10:36:55.978183 | etl/etl/lk_meas_specimen.sql | Done. -2020-11-30 10:38:40.898001 | etl/etl/lk_meas_chartevents.sql | Done. -2020-11-30 10:38:59.608780 | etl/etl/lk_meas_waveform.sql | Done. -2020-11-30 10:39:08.135171 | etl/etl/cdm_specimen.sql | Done. -2020-11-30 10:41:48.884722 | etl/etl/cdm_measurement.sql | Done. -2020-11-30 10:44:48.025826 | etl/etl/cdm_drug_exposure.sql | Done. -2020-11-30 10:44:55.263334 | etl/etl/cdm_device_exposure.sql | Error: See query No 2 -(24 min) - wrong quantity = '0.1667' - target_concept_id = 44940675 - code = '00019481604', 'MD-GASTROVIEW 66%-10% SOLN', 'Device' - SKIP DEVICE_EXPOSURE - -2020-11-30 09:40:49.064112 | etl/etl/cdm_location.sql | Done. -2020-11-30 09:41:02.050333 | etl/etl/cdm_care_site.sql | Done. -2020-11-30 09:41:30.427342 | etl/etl/cdm_person.sql | Done. -2020-11-30 09:42:12.130665 | etl/etl/cdm_visit_occurrence.sql | Done. -2020-11-30 09:45:22.188970 | etl/etl/cdm_visit_detail.sql | Done. -2020-11-30 09:45:32.742107 | etl/etl/cdm_death.sql | Done. -2020-11-30 09:47:50.550622 | etl/etl/cdm_condition_occurrence.sql | Done. -2020-11-30 09:51:06.959922 | etl/etl/cdm_procedure_occurrence.sql | Done. -2020-11-30 09:55:11.691765 | etl/etl/lk_meas_labevents.sql | Done. -2020-11-30 09:56:57.958007 | etl/etl/lk_meas_specimen.sql | Done. -2020-11-30 09:58:28.736297 | etl/etl/lk_meas_chartevents.sql | Done. -2020-11-30 09:58:36.657840 | etl/etl/lk_meas_waveform.sql | Done. -2020-11-30 09:58:45.218805 | etl/etl/cdm_specimen.sql | Done. -2020-11-30 10:00:52.573962 | etl/etl/cdm_measurement.sql | Done. -2020-11-30 10:03:52.637403 | etl/etl/cdm_drug_exposure.sql | Error: See query No 7 -(23 min) - --- Full, Staging - -2020-11-30 09:17:32.083983 | etl/staging/st_core.sql | Done. -2020-11-30 09:22:11.445505 | etl/staging/st_hosp.sql | Done. -2020-11-30 09:27:02.183644 | etl/staging/st_icu.sql | Done. -2020-11-30 09:27:19.817566 | etl/staging/st_waveform.sql | Done. -2020-11-30 09:30:10.870924 | etl/staging/voc_copy_to_target_dataset.sql | Error: See query No 10 -.......................... | etl/staging/voc_copy_to_target_dataset.sql | Done (minus stcm) -(~15 minutes) - - --- 2020-11-30 - --- Demo, Unload - -2020-11-30 13:12:23.159183 | etl/unload/unload_to_atlas_gen.sql | Done. - --- Demo, Metrics - -2020-11-30 12:44:54.756028 | test/metrics_gen/me_totals_from_conf.sql | Done. -2020-11-30 12:45:27.944277 | test/metrics_gen/me_persons_visits.sql | Done. -2020-11-30 13:02:54.934130 | test/metrics_gen/me_top100_from_conf.sql | Done. -2020-11-30 13:11:02.973057 | test/metrics_gen/me_mapping_rate_from_conf.sql | Done. -2020-11-30 13:12:56.539557 | test/metrics_gen/me_tops_together_from_conf.sql | Done. -(~30 min) - --- Demo, ETL - -2020-11-30 12:27:22.521201 | etl/etl/cdm_location.sql | Done. -2020-11-30 12:27:35.511346 | etl/etl/cdm_care_site.sql | Done. -2020-11-30 12:27:55.526172 | etl/etl/cdm_person.sql | Done. -2020-11-30 12:28:21.499040 | etl/etl/cdm_visit_occurrence.sql | Done. -2020-11-30 12:29:00.924581 | etl/etl/cdm_visit_detail.sql | Done. -2020-11-30 12:29:08.909711 | etl/etl/cdm_death.sql | Done. -2020-11-30 12:29:34.355589 | etl/etl/cdm_condition_occurrence.sql | Done. -2020-11-30 12:30:52.441593 | etl/etl/cdm_procedure_occurrence.sql | Done. -2020-11-30 12:31:21.131420 | etl/etl/lk_meas_labevents.sql | Done. -2020-11-30 12:31:55.754988 | etl/etl/lk_meas_specimen.sql | Done. -2020-11-30 12:32:05.642919 | etl/etl/lk_meas_chartevents.sql | Done. -2020-11-30 12:32:17.253812 | etl/etl/lk_meas_waveform.sql | Done. -2020-11-30 12:32:22.314991 | etl/etl/cdm_specimen.sql | Done. -2020-11-30 12:32:48.269325 | etl/etl/cdm_measurement.sql | Done. -2020-11-30 12:33:33.297546 | etl/etl/cdm_drug_exposure.sql | Done. -2020-11-30 12:33:39.647618 | etl/etl/cdm_device_exposure.sql | Done. -2020-11-30 12:34:13.408479 | etl/etl/cdm_observation.sql | Done. -2020-11-30 12:35:00.867018 | etl/etl/cdm_observation_period.sql | Done. -2020-11-30 12:35:21.563853 | etl/etl/cdm_fact_relationship.sql | Done. -2020-11-30 12:36:02.975324 | etl/etl/cdm_condition_era.sql | Done. -2020-11-30 12:37:53.395985 | etl/etl/cdm_drug_era.sql | Done. -2020-11-30 12:38:49.111865 | etl/etl/cdm_dose_era.sql | Done. -2020-11-30 12:38:55.683362 | etl/etl/cdm_cdm_source.sql | Done. -(11 min) - --- Demo, Staging - -2020-11-30 12:20:11.267449 | etl/staging/st_core.sql | Done. -2020-11-30 12:20:46.636200 | etl/staging/st_hosp.sql | Done. -2020-11-30 12:21:11.061613 | etl/staging/st_icu.sql | Done. -2020-11-30 12:21:30.457229 | etl/staging/st_waveform.sql | Done. -2020-11-30 12:24:22.644376 | etl/staging/voc_copy_to_target_dataset.sql | Done. -(~17 min) - --- Demo, metrics_gen - -2020-11-30 11:28:31.399800 | test/metrics_gen/me_persons_visits.sql | Done. -2020-11-30 11:30:14.071521 | test/metrics_gen/me_totals_from_conf.sql | Done. -2020-11-30 11:45:15.524515 | test/metrics_gen/me_top100_from_conf.sql | Done. -2020-11-30 11:51:47.020595 | test/metrics_gen/me_mapping_rate_from_conf.sql | Done. -2020-11-30 11:53:21.401308 | test/metrics_gen/me_tops_together_from_conf.sql | Done. -(~30 min) - -2020-11-30 11:03:27.811561 | me_top100_from_conf.sql | Done. - - -2020-11-30 10:03:22.938078 | me_totals_from_conf.sql | Done. - -+--------------------------------------+-------+-------+ -| category | name | count | -+--------------------------------------+-------+-------+ -| Number of persons | Total | 100 | -| Number of persons by Ethnicity | NULL | 100 | -| Number of persons by Race | NULL | 100 | -| Number of visits | Total | 275 | -| Number of visits by visit_concept_id | NULL | 275 | -+--------------------------------------+-------+-------+ - -2020-11-30 09:58:14.589598 | me_persons_visits.sql | Done. - - --- Demo ETL -2020-11-30 08:21:50.771434 | etl/etl/cdm_location.sql | Done. -2020-11-30 08:21:58.662773 | etl/etl/cdm_care_site.sql | Done. -2020-11-30 08:22:16.048542 | etl/etl/cdm_person.sql | Done. -2020-11-30 08:22:34.075105 | etl/etl/cdm_visit_occurrence.sql | Done. -2020-11-30 08:23:17.691201 | etl/etl/cdm_visit_detail.sql | Done. -2020-11-30 08:23:27.054452 | etl/etl/cdm_death.sql | Done. -2020-11-30 08:23:48.981310 | etl/etl/cdm_condition_occurrence.sql | Done. -2020-11-30 08:24:32.272272 | etl/etl/cdm_procedure_occurrence.sql | Done. -2020-11-30 08:24:51.865046 | etl/etl/lk_meas_labevents.sql | Done. -2020-11-30 08:25:15.977499 | etl/etl/lk_meas_specimen.sql | Done. -2020-11-30 08:25:25.484967 | etl/etl/lk_meas_chartevents.sql | Done. -2020-11-30 08:25:29.477739 | etl/etl/lk_meas_waveform.sql | Done. -2020-11-30 08:25:34.167512 | etl/etl/cdm_specimen.sql | Done. -2020-11-30 08:25:56.497927 | etl/etl/cdm_measurement.sql | Done. -2020-11-30 08:26:23.597644 | etl/etl/cdm_drug_exposure.sql | Done. -2020-11-30 08:26:28.285715 | etl/etl/cdm_device_exposure.sql | Done. -2020-11-30 08:26:50.541554 | etl/etl/cdm_observation.sql | Done. -2020-11-30 08:27:24.306771 | etl/etl/cdm_observation_period.sql | Done. -2020-11-30 08:27:39.202912 | etl/etl/cdm_fact_relationship.sql | Done. -2020-11-30 08:28:11.729508 | etl/etl/cdm_condition_era.sql | Done. -2020-11-30 08:29:19.867220 | etl/etl/cdm_drug_era.sql | Done. -2020-11-30 08:30:03.310365 | etl/etl/cdm_dose_era.sql | Done. -2020-11-30 08:30:08.057818 | etl/etl/cdm_cdm_source.sql | Done. - - --- 2020-11-24 - --- Demo ETL -2020-11-24 18:51:52.278224 | etl/etl/cdm_location.sql | Done. -2020-11-24 18:52:01.165001 | etl/etl/cdm_care_site.sql | Done. -2020-11-24 18:52:14.519685 | etl/etl/cdm_person.sql | Done. -2020-11-24 18:52:30.658725 | etl/etl/cdm_visit_occurrence.sql | Done. -2020-11-24 18:53:08.290012 | etl/etl/cdm_visit_detail.sql | Done. -2020-11-24 18:53:17.141000 | etl/etl/cdm_death.sql | Done. -2020-11-24 18:53:33.705937 | etl/etl/cdm_condition_occurrence.sql | Done. -2020-11-24 18:54:21.332463 | etl/etl/cdm_procedure_occurrence.sql | Done. -2020-11-24 18:54:39.365473 | etl/etl/lk_meas_labevents.sql | Done. -2020-11-24 18:55:03.427177 | etl/etl/lk_meas_specimen.sql | Done. -2020-11-24 18:55:07.629952 | etl/etl/lk_meas_waveform.sql | Done. -2020-11-24 18:55:13.774623 | etl/etl/cdm_specimen.sql | Done. -2020-11-24 18:55:30.961223 | etl/etl/cdm_measurement.sql | Done. -2020-11-24 18:55:54.412075 | etl/etl/cdm_drug_exposure.sql | Done. -2020-11-24 18:55:59.426987 | etl/etl/cdm_device_exposure.sql | Done. -2020-11-24 18:56:22.355180 | etl/etl/cdm_observation.sql | Done. -2020-11-24 18:56:54.803470 | etl/etl/cdm_observation_period.sql | Done. -2020-11-24 18:57:26.286481 | etl/etl/cdm_condition_era.sql | Done. -2020-11-24 18:58:34.693624 | etl/etl/cdm_drug_era.sql | Done. -2020-11-24 18:59:17.750609 | etl/etl/cdm_dose_era.sql | Done. -2020-11-24 18:59:23.728694 | etl/etl/cdm_cdm_source.sql | Done. - - --- 2020-11-12 - --- Demo QA -2020-11-13 02:02:45.509244 | test/qa/metric_total_row_count.sql | Done. -2020-11-13 02:02:48.759588 | test/qa/qa_start.sql | Done. -2020-11-13 02:02:52.046115 | test/qa/qa_care_site.sql | Done. -2020-11-13 02:02:55.339774 | test/qa/qa_person.sql | Done. -2020-11-13 02:02:58.470056 | test/qa/qa_visit_occurrence.sql | Done. - --- Demo UT -2020-11-13 01:59:49.736596 | test/ut/ut_start.sql | Done. -2020-11-13 02:00:02.238901 | test/ut/ut_care_site.sql | Done. -2020-11-13 02:00:27.837299 | test/ut/ut_person.sql | Done. -2020-11-13 02:00:27.855772 | test/ut/ut_visit_occurrence.sql | Done. - --- Demo ETL -2020-11-13 01:07:08.281358 | etl/etl/cdm_location.sql | Done. -2020-11-13 01:07:18.240974 | etl/etl/cdm_care_site.sql | Done. -2020-11-13 01:07:36.110694 | etl/etl/cdm_person.sql | Done. -2020-11-13 01:08:01.040708 | etl/etl/cdm_visit_occurrence.sql | Done. -2020-11-13 01:08:49.396504 | etl/etl/cdm_visit_detail.sql | Done. -2020-11-13 01:08:59.101201 | etl/etl/cdm_death.sql | Done. -2020-11-13 01:09:11.192721 | etl/etl/cdm_condition_occurrence.sql | Done. - Name icd_code not found inside src at [22:65] -2020-11-13 01:10:23.300039 | etl/etl/cdm_procedure_occurrence.sql | Done. -2020-11-13 01:10:30.684015 | etl/etl/lk_meas_labevents.sql | Done. -2020-11-13 01:10:39.211141 | etl/etl/lk_meas_specimen.sql | Done. -2020-11-13 01:10:43.972608 | etl/etl/lk_meas_waveform.sql | Done. -2020-11-13 01:10:47.833993 | etl/etl/cdm_specimen.sql | Done. -2020-11-13 01:10:51.451284 | etl/etl/cdm_measurement.sql | Done. -2020-11-13 01:11:34.244863 | etl/etl/cdm_drug_exposure.sql | Done. -2020-11-13 01:11:39.406375 | etl/etl/cdm_device_exposure.sql | Done. -2020-11-13 01:12:10.678063 | etl/etl/cdm_observation.sql | Done. -2020-11-13 01:12:16.625711 | etl/etl/cdm_observation_period.sql | Done. -2020-11-13 01:12:22.784533 | etl/etl/cdm_cdm_source.sql | Done. - --- Demo Staging --- - - --- 2020-10-21 --- Demo - -2020-10-21 20:52:45.517820 | etl/cdm_location.sql | Done. -2020-10-21 20:52:55.731269 | etl/cdm_care_site.sql | Done. -2020-10-21 20:53:15.450426 | etl/cdm_person.sql | Done. -2020-10-21 20:53:43.870170 | etl/cdm_visit_occurrence.sql | Done. -2020-10-21 20:54:38.693243 | etl/cdm_visit_detail.sql | Done. -2020-10-21 20:54:46.895288 | etl/cdm_observation_period.sql | Done. -2020-10-21 20:54:48.621447 | etl/cdm_condition_occurrence.sql | Done. -2020-10-21 20:56:57.944292 | etl/cdm_procedure_occurrence.sql | Done. -2020-10-21 20:56:59.898837 | etl/lk_meas_labevents.sql | Done. -2020-10-21 20:57:01.965428 | etl/lk_meas_waveform.sql | Done. -2020-10-21 20:57:05.682034 | etl/cdm_measurement.sql | Done. -2020-10-21 20:57:46.581866 | etl/cdm_drug_exposure.sql | Done. -2020-10-21 20:58:14.404026 | etl/cdm_observation.sql | Done. -2020-10-21 20:58:19.385489 | etl/cdm_cdm_source.sql | Done. -2020-10-21 21:01:33.477368 | etl/unload_to_atlas_gen.sql | Done. +2021-02-01 11:14:56.121986 | start... | Done. +2021-02-01 11:15:15.585768 | etl/ddl/ddl_voc_5_3_1.sql | Done. +2021-02-01 11:16:10.799645 | etl/ddl/ddl_cdm_5_3_1.sql | Done. +Run time: 0:01:14.678023