Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add ability to execute post actions for deploy-data script #110

Merged
merged 7 commits into from
Dec 17, 2020
Merged

Add ability to execute post actions for deploy-data script #110

merged 7 commits into from
Dec 17, 2020

Conversation

tlvu
Copy link
Collaborator

@tlvu tlvu commented Dec 17, 2020

Script deploy-data was previously introduced in PR #72 to deploy any files from any git repos to the local host it runs.

Now it grows the ability to run commands from the git repo it just pulls.

Being able to run commands open new possibilities:

Combining this deploy-data with the scheduler component means we have a way for cronjobs to automatically always execute the most up-to-date version of any scripts from any git repos.

@tlogan2000, @huard FYI this was required for the GEPS forecasts project.

@tlvu tlvu requested a review from matprov December 17, 2020 03:11
@tlvu tlvu mentioned this pull request Dec 17, 2020
Merged
@tlvu tlvu force-pushed the add-ability-to-execute-post-actions-for-deploy-data branch from 82f3129 to df01c4b Compare December 17, 2020 19:19
matprov
matprov approved these changes Dec 17, 2020
View reviewed changes
Copy link
Collaborator

@matprov matprov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@tlvu tlvu merged commit b650cf6 into master Dec 17, 2020
@tlvu tlvu deleted the add-ability-to-execute-post-actions-for-deploy-data branch December 17, 2020 20:26
tlvu added a commit to bird-house/birdhouse-deploy-ouranos that referenced this pull request Dec 18, 2020
@tlvu
Merge pull request #2 from bird-house/add-geps-forecasts-scheduler-jobs
b5e3d05 
Add geps forecasts scheduler jobs.

Matching PR to automate deployment and future updates of the PR Ouranosinc/pavics-vdb#21.  Requires PR bird-house/birdhouse-deploy#110.  Part of issue Ouranosinc/pavics-vdb#19 and issue Ouranosinc/raven#313.

Add a cronjob (`retrieve_geps_forecasts.env`) and a config file (`retrieve_geps_forecasts.yml`) that defines which git repo to pull from and what command to run to handle the geps forecasts.

Leverage the script I did earlier that can deploy any files from any git repos to a PAVICS host.

Activating the cronjob via `env.local` is like so https://github.com/bird-house/birdhouse-deploy/blob/2bb9ce6756cb317c47e28a3a65d95d62312c02fd/birdhouse/env.local.example#L156-L158

@tlogan2000 If you change the code in the `pavics-vdb` repo, it'll be live the next time the cronjob runs.  If you want to change the cronjob frequency, edit `retrieve_geps_forecasts.env` and it autodeploy itself.

Deployed to our staging env Medus: http://medus.ouranos.ca:8083/twitcher/ows/proxy/thredds/catalog/birdhouse/eccc/forecasts/geps/catalog.html.

@huard I'll apply this exact same recipe for your Intake catalog project.

@matprov This is an example showing you how CRIM can have custom cronjobs, all source controlled, centrally tracked by `env,local`, with automated deployment and update.

Sample cronjob logs from Medus:
[retrieve_geps_forecasts.log](https://github.com/bird-house/birdhouse-deploy-ouranos/files/5716267/retrieve_geps_forecasts.log)

```
==========
datadeploy START_TIME=2020-12-18T05:37:02+0000
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml config.checkout_cache
+ docker run --rm --name deploy_data_yq_2020-12-18T05_37_02p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml config.checkout_cache
+ CHECKOUT_CACHE=
+ '[' -z  ]
+ CHECKOUT_CACHE=/data/deploy_data_cache/retrieve_geps_forecasts
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml config.git_ssh_identity_file
+ docker run --rm --name deploy_data_yq_2020-12-18T05_37_02p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml config.git_ssh_identity_file
+ GIT_SSH_IDENTITY_FILE=
+ '[' -z  ]
+ GIT_SSH_IDENTITY_FILE=/home/mourad/.ssh/id_rsa_pavics-vdb-ro
+ '[' '!' -z /home/mourad/.ssh/id_rsa_pavics-vdb-ro ]
+ export 'GIT_SSH_COMMAND=ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentityFile=/home/mourad/.ssh/id_rsa_pavics-vdb-ro'
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[*].repo_url'
+ docker run --rm --name deploy_data_yq_2020-12-18T05_37_02p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[*].repo_url'
+ GIT_REPO_URLS=git@github.com:Ouranosinc/pavics-vdb.git
+ ensure_not_empty git@github.com:Ouranosinc/pavics-vdb.git
+ '[' -z git@github.com:Ouranosinc/pavics-vdb.git ]
+ REPO_NUM=0
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml --defaultValue origin/master 'deploy[0].branch'
+ docker run --rm --name deploy_data_yq_2020-12-18T05_37_02p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml --defaultValue origin/master 'deploy[0].branch'
+ GIT_BRANCH=origin/minor-enhancements-to-convert_grib2_to_nc.py
+ ensure_not_empty origin/minor-enhancements-to-convert_grib2_to_nc.py
+ '[' -z origin/minor-enhancements-to-convert_grib2_to_nc.py ]
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].checkout_name'
+ docker run --rm --name deploy_data_yq_2020-12-18T05_37_02p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].checkout_name'
+ GIT_CHECKOUT_NAME=pavics-vdb-for-geps-forecasts
+ ensure_not_empty pavics-vdb-for-geps-forecasts
+ '[' -z pavics-vdb-for-geps-forecasts ]
+ CLONE_DEST=/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts
+ '[' '!' -d /data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts ]
+ echo 'refresh repo '"'"'/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts'"'"' on branch '"'"'origin/minor-enhancements-to-convert_grib2_to_nc.py'"'"
refresh repo '/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts' on branch 'origin/minor-enhancements-to-convert_grib2_to_nc.py'
+ cd /data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts
+ git remote -v
origin	git@github.com:Ouranosinc/pavics-vdb.git (fetch)
origin	git@github.com:Ouranosinc/pavics-vdb.git (push)
+ git clean -fdx
+ git fetch --prune --all
Fetching origin
Warning: Permanently added 'github.com,140.82.114.3' (RSA) to the list of known hosts.
From github.com:Ouranosinc/pavics-vdb
 + 3e94dfb...a64a444 minor-enhancements-to-convert_grib2_to_nc.py -> origin/minor-enhancements-to-convert_grib2_to_nc.py  (forced update)
   a6e4c2b..a64a444  self-contain-geps-forecasts-runtime -> origin/self-contain-geps-forecasts-runtime
+ git checkout --force origin/minor-enhancements-to-convert_grib2_to_nc.py
Warning: you are leaving 1 commit behind, not connected to
any of your branches:

  3e94dfb convert_grib2_to_nc.py: toogle dask progress bar to avoid noise in logs

If you want to keep it by creating a new branch, this may be a good time
to do so with:

 git branch <new-branch-name> 3e94dfb

HEAD is now at a64a444 convert_grib2_to_nc.py: toogle dask progress bar to avoid noise in logs
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].dir_maps[*].source_dir'
+ docker run --rm --name deploy_data_yq_2020-12-18T05_37_02p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].dir_maps[*].source_dir'
+ SRC_DIRS=
+ DIR_NUM=0
+ yq r /home/mourad/PROJECTS/retrieve_geps_forecasts.yml --length 'deploy[0].post_actions'
+ docker run --rm --name deploy_data_yq_2020-12-18T05_37_02p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r /home/mourad/PROJECTS/retrieve_geps_forecasts.yml --length 'deploy[0].post_actions'
+ POST_ACTIONS_LENGTH=1
+ '[' -z 1 ]
+ expr 1 - 1
+ POST_ACTIONS_LENGTH_FOR_SEQ=0
+ seq 0 0
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].post_actions[0].action'
+ docker run --rm --name deploy_data_yq_2020-12-18T05_37_02p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].post_actions[0].action'
+ POST_ACTION='RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc'
+ ensure_not_empty 'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc'
+ '[' -z 'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc' ]
+ echo 'executing post_action '"'"'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc'"'"
executing post_action 'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc'
+ eval 'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH}' 'RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH}' 'RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH}' 'CONVERT_GRIB2_TO_NC_PROGRESSBAR=false' ECCC-datamart_sync/run_convert_grib2_to_nc
+ RUN_CONVERT_GRIB2_TO_NC_INPATH= RUN_CONVERT_GRIB2_TO_NC_OUTPATH= RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=/data/datasets/eccc/forecasts/geps CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc
==========
run_convert_grib2_to_nc START_TIME=2020-12-18T05:37:25+0000
+ id -u
+ USER_ID=0
+ id -g
+ GROUP_ID=0
+ mkdir -p /data/tmp/geps_forecast/grib2
+ mkdir -p /data/tmp/geps_forecast/netcdf
+ docker run --rm --name run_convert_grib2_to_nc --volume /data/tmp/geps_forecast/grib2:/data/tmp/geps_forecast/grib2:rw --volume /data/tmp/geps_forecast/netcdf:/data/tmp/geps_forecast/netcdf:rw --volume /data/datasets/eccc/forecasts/geps:/data/datasets/eccc/forecasts/geps:rw --volume /data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py:/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py:ro --env 'CONVERT_GRIB2_TO_NC_INPATH=/data/tmp/geps_forecast/grib2' --env 'CONVERT_GRIB2_TO_NC_OUTPATH=/data/tmp/geps_forecast/netcdf' --env 'CONVERT_GRIB2_TO_NC_THREDDSPATH=/data/datasets/eccc/forecasts/geps' --env CONVERT_GRIB2_TO_NC_PROGRESSBAR --user 0:0 pavics/workflow-tests:201214 python /data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py
Downloading grib2 files : attempt 1 of 3
Checking for updated GEPS files : Forecast 20201218 00
Done. Found 193 new files
Checking for updated GEPS files : Forecast 20201217 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 12
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 12
Done. Found 0 new files
Downloading grib2 files : attempt 2 of 3
Checking for updated GEPS files : Forecast 20201218 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 12
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 12
Done. Found 0 new files
Downloading grib2 files : attempt 3 of 3
Checking for updated GEPS files : Forecast 20201218 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 12
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 12
Done. Found 0 new files
coverting to netcdf ....
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P159_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P162_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P165_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P168_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P036_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P039_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P042_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P045_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P096_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P099_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P102_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P105_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P099_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P102_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P105_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P108_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P294_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P300_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P306_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P312_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P240_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P246_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P252_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P258_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P027_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P030_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P033_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P036_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P222_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P228_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P234_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P240_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P168_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P171_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P174_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P177_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P051_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P054_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P057_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P060_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P048_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P051_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P054_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P057_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P120_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P123_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P126_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P129_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P312_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P318_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P324_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P330_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P135_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P138_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P141_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P144_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P060_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P063_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P066_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P069_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P132_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P135_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P138_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P141_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P360_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P366_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P372_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P378_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P087_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P090_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P093_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P096_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P342_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P348_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P354_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P360_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P288_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P294_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P300_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P306_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P063_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P066_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P069_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P072_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P318_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P324_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P330_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P336_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P192_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P198_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P204_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P210_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P111_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P114_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P117_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P120_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P366_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P372_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P378_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P384_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P264_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P270_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P276_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P282_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P171_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P174_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P177_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P180_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P000_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P003_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P006_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P009_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P084_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P087_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P090_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P093_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P384_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P147_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P150_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P153_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P156_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P012_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P015_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P018_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P021_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P108_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P111_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P114_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P117_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P123_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P126_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P129_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P132_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P024_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P027_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P030_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P033_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P072_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P075_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P078_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P081_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P336_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P342_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P348_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P354_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P075_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P078_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P081_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P084_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P270_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P276_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P282_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P288_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P216_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P222_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P228_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P234_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P015_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P018_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P021_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P024_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P198_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P204_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P210_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P216_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P144_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P147_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P150_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P153_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P039_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P042_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P045_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P048_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P246_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P252_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P258_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P264_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P180_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P183_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P186_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P189_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P003_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P006_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P009_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P012_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P183_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P186_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P189_allmbrs.grib2
converting  CMC_geps-raw_APCP_SFC_0_latlon0p5x0p5_2020121800_P192_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P156_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P159_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P162_allmbrs.grib2
converting  CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121800_P165_allmbrs.grib2
done coverting  GEPS . It took :  222.2973051071167 seconds
2020121400 : no action needed for combining variables and timesteps
2020121412 : no action needed for combining variables and timesteps
2020121500 : no action needed for combining variables and timesteps
2020121512 : no action needed for combining variables and timesteps
2020121600 : no action needed for combining variables and timesteps
2020121612 : no action needed for combining variables and timesteps
2020121700 : no action needed for combining variables and timesteps
2020121712 : no action needed for combining variables and timesteps
2020121800 : combining variables and timesteps ...
CMC_geps-raw_latlon0p5x0p5_2020121800_allP_allmbrs.nc
Traceback (most recent call last):
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 153, in validate_ncml
    assert ds.reftime.values == pd.to_datetime(start_date, format='%Y%m%d%H')
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 336, in <module>
    main()
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 141, in main
    if validate_ncml(opendap_latest, latest_date):
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 157, in validate_ncml
    raise Exception("can't read ncml opendap link")
Exception: can't read ncml opendap link
+ cleanup_on_exit
+ set +x

run_convert_grib2_to_nc finished START_TIME=2020-12-18T05:37:25+0000
run_convert_grib2_to_nc finished   END_TIME=2020-12-18T05:47:11+0000
+ expr 0 + 1
+ REPO_NUM=1
+ cleanup_on_exit
+ set +x

datadeploy finished START_TIME=2020-12-18T05:37:02+0000
datadeploy finished   END_TIME=2020-12-18T05:47:12+0000
==========
datadeploy START_TIME=2020-12-18T06:37:06+0000
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml config.checkout_cache
+ docker run --rm --name deploy_data_yq_2020-12-18T06_37_06p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml config.checkout_cache
+ CHECKOUT_CACHE=
+ '[' -z  ]
+ CHECKOUT_CACHE=/data/deploy_data_cache/retrieve_geps_forecasts
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml config.git_ssh_identity_file
+ docker run --rm --name deploy_data_yq_2020-12-18T06_37_06p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml config.git_ssh_identity_file
+ GIT_SSH_IDENTITY_FILE=
+ '[' -z  ]
+ GIT_SSH_IDENTITY_FILE=/home/mourad/.ssh/id_rsa_pavics-vdb-ro
+ '[' '!' -z /home/mourad/.ssh/id_rsa_pavics-vdb-ro ]
+ export 'GIT_SSH_COMMAND=ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentityFile=/home/mourad/.ssh/id_rsa_pavics-vdb-ro'
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[*].repo_url'
+ docker run --rm --name deploy_data_yq_2020-12-18T06_37_06p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[*].repo_url'
+ GIT_REPO_URLS=git@github.com:Ouranosinc/pavics-vdb.git
+ ensure_not_empty git@github.com:Ouranosinc/pavics-vdb.git
+ '[' -z git@github.com:Ouranosinc/pavics-vdb.git ]
+ REPO_NUM=0
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml --defaultValue origin/master 'deploy[0].branch'
+ docker run --rm --name deploy_data_yq_2020-12-18T06_37_06p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml --defaultValue origin/master 'deploy[0].branch'
+ GIT_BRANCH=origin/minor-enhancements-to-convert_grib2_to_nc.py
+ ensure_not_empty origin/minor-enhancements-to-convert_grib2_to_nc.py
+ '[' -z origin/minor-enhancements-to-convert_grib2_to_nc.py ]
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].checkout_name'
+ docker run --rm --name deploy_data_yq_2020-12-18T06_37_06p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].checkout_name'
+ GIT_CHECKOUT_NAME=pavics-vdb-for-geps-forecasts
+ ensure_not_empty pavics-vdb-for-geps-forecasts
+ '[' -z pavics-vdb-for-geps-forecasts ]
+ CLONE_DEST=/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts
+ '[' '!' -d /data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts ]
+ echo 'refresh repo '"'"'/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts'"'"' on branch '"'"'origin/minor-enhancements-to-convert_grib2_to_nc.py'"'"
refresh repo '/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts' on branch 'origin/minor-enhancements-to-convert_grib2_to_nc.py'
+ cd /data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts
+ git remote -v
origin	git@github.com:Ouranosinc/pavics-vdb.git (fetch)
origin	git@github.com:Ouranosinc/pavics-vdb.git (push)
+ git clean -fdx
+ git fetch --prune --all
Fetching origin
Warning: Permanently added 'github.com,140.82.114.4' (RSA) to the list of known hosts.
+ git checkout --force origin/minor-enhancements-to-convert_grib2_to_nc.py
HEAD is now at a64a444 convert_grib2_to_nc.py: toogle dask progress bar to avoid noise in logs
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].dir_maps[*].source_dir'
+ docker run --rm --name deploy_data_yq_2020-12-18T06_37_06p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].dir_maps[*].source_dir'
+ SRC_DIRS=
+ DIR_NUM=0
+ yq r /home/mourad/PROJECTS/retrieve_geps_forecasts.yml --length 'deploy[0].post_actions'
+ docker run --rm --name deploy_data_yq_2020-12-18T06_37_06p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r /home/mourad/PROJECTS/retrieve_geps_forecasts.yml --length 'deploy[0].post_actions'
+ POST_ACTIONS_LENGTH=1
+ '[' -z 1 ]
+ expr 1 - 1
+ POST_ACTIONS_LENGTH_FOR_SEQ=0
+ seq 0 0
+ yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].post_actions[0].action'
+ docker run --rm --name deploy_data_yq_2020-12-18T06_37_06p0000 -v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml:/home/mourad/PROJECTS/retrieve_geps_forecasts.yml:ro mikefarah/yq:3.3.4 yq r -p v /home/mourad/PROJECTS/retrieve_geps_forecasts.yml 'deploy[0].post_actions[0].action'
+ POST_ACTION='RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc'
+ ensure_not_empty 'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc'
+ '[' -z 'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc' ]
+ echo 'executing post_action '"'"'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc'"'"
executing post_action 'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH} RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH} RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH} CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc'
+ eval 'RUN_CONVERT_GRIB2_TO_NC_INPATH=${RUN_CONVERT_GRIB2_TO_NC_INPATH}' 'RUN_CONVERT_GRIB2_TO_NC_OUTPATH=${RUN_CONVERT_GRIB2_TO_NC_OUTPATH}' 'RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=${RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH}' 'CONVERT_GRIB2_TO_NC_PROGRESSBAR=false' ECCC-datamart_sync/run_convert_grib2_to_nc
+ RUN_CONVERT_GRIB2_TO_NC_INPATH= RUN_CONVERT_GRIB2_TO_NC_OUTPATH= RUN_CONVERT_GRIB2_TO_NC_THREDDSPATH=/data/datasets/eccc/forecasts/geps CONVERT_GRIB2_TO_NC_PROGRESSBAR=false ECCC-datamart_sync/run_convert_grib2_to_nc
==========
run_convert_grib2_to_nc START_TIME=2020-12-18T06:37:30+0000
+ id -u
+ USER_ID=0
+ id -g
+ GROUP_ID=0
+ mkdir -p /data/tmp/geps_forecast/grib2
+ mkdir -p /data/tmp/geps_forecast/netcdf
+ docker run --rm --name run_convert_grib2_to_nc --volume /data/tmp/geps_forecast/grib2:/data/tmp/geps_forecast/grib2:rw --volume /data/tmp/geps_forecast/netcdf:/data/tmp/geps_forecast/netcdf:rw --volume /data/datasets/eccc/forecasts/geps:/data/datasets/eccc/forecasts/geps:rw --volume /data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py:/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py:ro --env 'CONVERT_GRIB2_TO_NC_INPATH=/data/tmp/geps_forecast/grib2' --env 'CONVERT_GRIB2_TO_NC_OUTPATH=/data/tmp/geps_forecast/netcdf' --env 'CONVERT_GRIB2_TO_NC_THREDDSPATH=/data/datasets/eccc/forecasts/geps' --env CONVERT_GRIB2_TO_NC_PROGRESSBAR --user 0:0 pavics/workflow-tests:201214 python /data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py
Downloading grib2 files : attempt 1 of 3
Checking for updated GEPS files : Forecast 20201218 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 12
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 12
Done. Found 0 new files
Downloading grib2 files : attempt 2 of 3
Checking for updated GEPS files : Forecast 20201218 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 12
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 12
Done. Found 0 new files
Downloading grib2 files : attempt 3 of 3
Checking for updated GEPS files : Forecast 20201218 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201217 12
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 12
Done. Found 0 new files
coverting to netcdf ....
done coverting  GEPS . It took :  2.776029586791992 seconds
2020121400 : no action needed for combining variables and timesteps
2020121412 : no action needed for combining variables and timesteps
2020121500 : no action needed for combining variables and timesteps
2020121512 : no action needed for combining variables and timesteps
2020121600 : no action needed for combining variables and timesteps
2020121612 : no action needed for combining variables and timesteps
2020121700 : no action needed for combining variables and timesteps
2020121712 : no action needed for combining variables and timesteps
2020121800 : no action needed for combining variables and timesteps
Traceback (most recent call last):
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 153, in validate_ncml
    assert ds.reftime.values == pd.to_datetime(start_date, format='%Y%m%d%H')
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 336, in <module>
    main()
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 141, in main
    if validate_ncml(opendap_latest, latest_date):
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 157, in validate_ncml
    raise Exception("can't read ncml opendap link")
Exception: can't read ncml opendap link
+ cleanup_on_exit
+ set +x

run_convert_grib2_to_nc finished START_TIME=2020-12-18T06:37:30+0000
run_convert_grib2_to_nc finished   END_TIME=2020-12-18T06:37:48+0000
+ expr 0 + 1
+ REPO_NUM=1
+ cleanup_on_exit
+ set +x

datadeploy finished START_TIME=2020-12-18T06:37:06+0000
datadeploy finished   END_TIME=2020-12-18T06:37:48+0000
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@matprov matprov matprov approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@tlvu @matprov