Code to transform CDS data.
To run the python script for transforming CDS data, the user should first place the raw CDS data file in the s3 bucket
python3 cds-transformation.py cds_config_example.yaml
To run the python script for transforming CDS data with the data format v1.2, the user should first place all the raw CDS data files in ./0-Raw-Data-Files/cds_raw_data_files_v1.2 folder than use the command to extrat raw data dictionary
python3 1-Transformation-Scripts/cds-transformation_v1.2.py --config_file 2-Config-Files/cds_config_v1.2/cds_config_example_v1.2.yaml --extract_raw_data_dictionary
After extracting the raw data dictionary, the user can use the command to transform raw data.
python3 1-Transformation-Scripts/cds-transformation_v1.2.py --config_file 2-Config-Files/cds_config_v1.2/cds_config_example_v1.2.yaml
If the user want to upload the transformed data to the s3 folder, the user can use the command below.
python3 1-Transformation-Scripts/cds-transformation_v1.2.py --config_file 2-Config-Files/cds_config_v1.2/cds_config_example_v1.2.yaml --upload_s3
To run the python script for transforming CDS data with the data format v1.3, the user should first place all the raw CDS data files in ./0-Raw-Data-Files/cds_raw_data_files_v1.3 folder than use the command
python3 1-Transformation-Scripts/cds-transformation_v1.3.py --config_file 2-Config-Files/cds_config_v1.3/cds_config_example_v1.3.yaml --extract_raw_data_dictionary
After extracting the raw data dictionary, the user can use the command to transform raw data.
python3 1-Transformation-Scripts/cds-transformation_v1.3.py --config_file 2-Config-Files/cds_config_v1.3/cds_config_example_v1.3.yaml
If the user want to download data from s3, the user can use the command to transform raw data.
python3 1-Transformation-Scripts/cds-transformation_v1.3.py --config_file 2-Config-Files/cds_config_v1.3/cds_config_example_v1.3.yaml --download_s3
If the user want to upload the transformed data to the s3 folder, the user can use the command below.
python3 1-Transformation-Scripts/cds-transformation_v1.3.py --config_file 2-Config-Files/cds_config_v1.3/cds_config_example_v1.3.yaml --upload_s3
-
Notifications
You must be signed in to change notification settings - Fork 0
CBIIT/cds-etl
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
About
No description or website provided.
Topics
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published