Sync Raven testdata to Thredds for Raven tutorial notebooks

[tlvu]

Part of Ouranosinc/raven#185

Leveraging the cron daemon of the scheduler component, sync Raven testdata to Thredds for Raven tutorial notebooks.

Activation of the pre-configured cronjob is via env.local as usual for infra-as-code.

New generic deploy-data script can clone any number of git repos, sync any number of folders in the git repo to any number of local folders, with ability to cherry-pick just the few files needed (Raven testdata has many types of files, we only need to sync .nc files to Thredds, to avoid polluting Thredds storage /data/datasets/testdata/raven).

Limitation of the first version of this deploy-data script:

• Do not handle re-organizing file layout, this is a pure sync only with very limited rsync filtering for now (tutorial notebooks deploy from multiple repos, need re-organizing the file layout)

So the script has room to grow. I see it as a generic solution to the repeated problem "take files from various git repos and deploy them somewhere automatically". If we need to deploy another repo, juste write a new config file, stop writing boilerplate code again.

Minor unrelated change in this PR:

• README update to reference the new birdhouse-deploy-ouranos.
• Make sourcing the various pre-configured cronjob backward-compat with older version of the repo where those cronjob did not exist yet.

[huard]

May want a second review from someone more competent in bash.

[matprov]

LGTM

[tlvu]

Tagged 1.11.4.

Autodeployed to prod:

triggerdeploy finished START_TIME=2020-10-15T05:07:03+0000
triggerdeploy finished   END_TIME=2020-10-15T05:08:45+0000

The following raven testdata are now available at https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/catalog/birdhouse/testdata/raven/catalog.html

created directory /data/datasets/testdata/raven
cd+++++++++ ./
cd+++++++++ XSS_forecast_data/
>f+++++++++ XSS_forecast_data/XSS_fcst_det.nc
>f+++++++++ XSS_forecast_data/XSS_fcst_ens.nc
>f+++++++++ XSS_forecast_data/XSS_obs.nc
cd+++++++++ cmip5/
>f+++++++++ cmip5/tas_Amon_CanESM2_rcp85_r1i1p1_200601-210012_subset.nc
cd+++++++++ gr4j_cemaneige/
>f+++++++++ gr4j_cemaneige/evap.nc
>f+++++++++ gr4j_cemaneige/pr.nc
>f+++++++++ gr4j_cemaneige/tas.nc
cd+++++++++ hydro_simulations/
>f+++++++++ hydro_simulations/raven-gr4j-cemaneige-sim_gr4jcn-0_Hydrographs.nc
>f+++++++++ hydro_simulations/raven-gr4j-cemaneige-sim_hmets-0_Hydrographs.nc
cd+++++++++ input2d/
>f+++++++++ input2d/input2d.nc
cd+++++++++ ostrich-gr4j-cemaneige/
>f+++++++++ ostrich-gr4j-cemaneige/Salmon-River-Near-Prince-George_meteo_daily.nc
cd+++++++++ ostrich-hbv-ec/
>f+++++++++ ostrich-hbv-ec/Salmon-River-Near-Prince-George_meteo_daily.nc
cd+++++++++ ostrich-hmets/
>f+++++++++ ostrich-hmets/Salmon-River-Near-Prince-George_meteo_daily.nc
cd+++++++++ ostrich-mohyse/
>f+++++++++ ostrich-mohyse/Salmon-River-Near-Prince-George_meteo_daily.nc
cd+++++++++ raven-gr4j-cemaneige/
>f+++++++++ raven-gr4j-cemaneige/Salmon-River-Near-Prince-George_meteo_daily.nc
>f+++++++++ raven-gr4j-cemaneige/Salmon-River-Near-Prince-George_meteo_daily_2d.nc
>f+++++++++ raven-gr4j-cemaneige/Salmon-River-Near-Prince-George_meteo_daily_3d.nc
cd+++++++++ ts_stats_outputs/
>f+++++++++ ts_stats_outputs/out.nc

sent 18.72M bytes  received 446 bytes  37.44M bytes/sec
total size is 18.71M  speedup is 1.00
+ expr 0 + 1
+ DIR_NUM=1
+ expr 0 + 1
+ REPO_NUM=1
+ cleanup_on_exit
+ set +x

datadeploy finished START_TIME=2020-10-15T05:30:02+0000
datadeploy finished   END_TIME=2020-10-15T05:30:59+0000