DM-32992: Allow the use of Docker images in GAR #5
Conversation
timj
changed the title
|
Hi @MichelleGower , I updated the file config/bps_idf.yaml with the lsst Docker image location in GAR, and added a new rst file doc/changes/DM-32992.misc.rst with an explanatory example. Please help review. Many thanks, Shuwei |
There was a problem hiding this comment.
Shouldn't the yaml in python/lsst/ctrl/bps/panda/conf_example also be updated? (And since the plugin has been split into its own package maybe move the conf_example up into the main directory instead of it being hidden under python?)
And there are a couple comments about how to set it to use docker hub (I didn't test what I asked in the question, so please test it to make sure the empty string does actually work.)
doc/changes/DM-32992.misc.rst
Outdated
| @@ -0,0 +1,10 @@ | |||
|
|
|||
| Introduced a new parameter **dockerImageLocation** in the PanDA IDF configuration yaml file to pull lsst release containes from **GAR (Google Artifact Registry)**. This parameter is trailed with **'/'**, so it could be used in *sw_image* path in the following example. And the *sw_image* will still refer to the **Docker hub**, if the parameter **dockerImageLocation** is not defined, to make the *sw_image* to back compatible with previous PanDA IDF configuration yaml files. | |||
There was a problem hiding this comment.
By putting dockerImageLocation in the predefined config/bps_idf.yaml, there's not a way to make it "not defined" other than copying the file and editing the value in the copy. Instead, can't the user override just that one value in their submit yaml with the empty string?
There was a problem hiding this comment.
Just added another example to use images from the Docker hub.
And changed the description:
if the parameter dockerImageLocation is empty or not defined, to make the sw_image backward compatible with previous PanDA IDF configuration yaml files.
That means that users can start with this syntax now without waiting for the update of the config/bps_idf.yaml.
There was a problem hiding this comment.
Just added an example of using the prefix {dockerImageLocation} into the file python/lsst/ctrl/bps/panda/conf_example/pipelines_check_idf.yaml.
Yes, I did test it with an undefined prefix, and it did pull that image from the Docker hub.
There was a problem hiding this comment.
Found a couple typos and made a request to make how to get Docker hub images consistent. Looks like the branch needs to be rebased (https://developer.lsst.io/work/flow.html?highlight=rebase#pushing-code). Merge approved after rebasing and fixing these couple things.
|
|
||
| Introduced a new parameter **dockerImageLocation** in the PanDA IDF configuration yaml file to pull lsst release containes from **GAR (Google Artifact Registry)**. This parameter is trailed with **'/'**, so it could be used in *sw_image* path in the following example. And the *sw_image* will still refer to the **Docker hub**, if the parameter **dockerImageLocation** is empty or not defined, to make the *sw_image* backward compatible with previous PanDA IDF configuration yaml files. | ||
|
|
||
| In the user bps submition yaml file, just prepend this parameter to the sw_image path, that is: |
There was a problem hiding this comment.
containes -> containers
submition -> submission
|
|
||
| Please note that there is no any extra character(s) between *{dockerImageLocation}* and *lsstsqre*. | ||
|
|
||
| In case you have to use images from the Docker hub instead, you just take out the prefix **{dockerImageLocation}** in the path, that is: |
There was a problem hiding this comment.
This works. I wasn't expecting it though because in the bps_idf.yaml file it mentions setting dockerImageLocation to the empty string. So I would make it consistent throughout.. either setting it to empty string or removing the prefix in the path.
There was a problem hiding this comment.
Hi @MichelleGower ,
The users can not modify config/bps_idf.yaml, so they have to remove the prefix in the path in order to use images from the Docker hub for their individual jobs. While on the system-wide side, the prefix can be set empty or undefined to force all the users to pull images from the Docker hub.
yesw2000
force-pushed
the
tickets/DM-32992
branch
from
742cecf
to
b5bbf52
Compare
Checklist
doc/changes