Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create GHA to injest the ZXS data #40

Open
Tracked by #10
franTarkenton opened this issue Jul 10, 2023 · 2 comments
Open
Tracked by #10

Create GHA to injest the ZXS data #40

franTarkenton opened this issue Jul 10, 2023 · 2 comments

Comments

@franTarkenton
Copy link
Member

Pull this data from the federal government datamart and push it up to object store. For location in object store, try to emulate the existing directory structure used on prem.

The trigger for this will be a cron based trigger, using the same timing as the existing job. Subsequent tasks will transition it to an event based trigger using the same patter created for the CMC pipeline

@franTarkenton
Copy link
Member Author

franTarkenton commented Aug 3, 2023

source for the ZXS data:
http://hpfx.collab.science.gc.ca/%YYYY%%MT%%DD%/WXO-DD/vertical_profile/observation/csv/

example (not templated) source dir:
http://hpfx.collab.science.gc.ca/20230801/WXO-DD/vertical_profile/observation/csv/

Script should pull the file ObsTephi_12_CZXS.csv down and rename it:
ObsTephi_12_CZXS-2013-04-09.csv

Where the date is the date the file is downloaded.

@franTarkenton
Copy link
Member Author

pipeline is now working, PR will deploy to dev, merge tears down and deploys to prod. Process is idempotent... If run consecutively should pickup where previous process left off.

Closing for now, but am creating a new ticket to create an opensearch query to report out on sucess of jobs, and potentially glue together with a kibana dashboard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Review / QA
Development

No branches or pull requests

1 participant