-
Notifications
You must be signed in to change notification settings - Fork 106
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor docker build/push workflow action #11653
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -15,6 +15,8 @@ on: | |
jobs: | ||
build_images_from_template: | ||
runs-on: ubuntu-latest | ||
env: | ||
SERVICE_NAME: ${{inputs.wmcore_component}} | ||
steps: | ||
- name: Get the Ref | ||
id: get-ref | ||
|
@@ -23,37 +25,27 @@ jobs: | |
ref: ${{ github.ref }} | ||
head_ref: ${{ github.head_ref }} | ||
|
||
- name: Build image | ||
env: | ||
PYPI_TAG: ${{steps.get-ref.outputs.tag}} | ||
run: | | ||
echo "Building service: ${{inputs.wmcore_component}}, with tag: ${PYPI_TAG}" | ||
svn checkout https://github.com/dmwm/CMSKubernetes/trunk/docker/pypi/${{inputs.wmcore_component}} | ||
cd ${{inputs.wmcore_component}} | ||
cat Dockerfile | ||
echo "Sleeping 5min to ensure that PyPi packages are available..." | ||
sleep 300 | ||
docker build --build-arg TAG=${PYPI_TAG} --tag registry.cern.ch/cmsweb/${{inputs.wmcore_component}}:${PYPI_TAG} . | ||
|
||
- name: Images | ||
run: | | ||
docker images | ||
|
||
- name: Login to registry.cern.ch | ||
uses: docker/login-action@v2 | ||
with: | ||
registry: registry.cern.ch | ||
username: ${{ secrets.cern_user }} | ||
password: ${{ secrets.cern_token }} | ||
|
||
- name: Publish image to registry.cern.ch | ||
uses: docker/build-push-action@v1 | ||
with: | ||
path: ${{inputs.wmcore_component}} | ||
build_args: | | ||
TAG=${{steps.get-ref.outputs.tag}} | ||
registry: registry.cern.ch | ||
username: ${{ secrets.cern_user }} | ||
password: ${{ secrets.cern_token }} | ||
repository: cmsweb/${{inputs.wmcore_component}} | ||
tag_with_ref: true | ||
- name: Build and publish docker image | ||
env: | ||
PYPI_TAG: ${{steps.get-ref.outputs.tag}} | ||
CERN_REGISTRY: registry.cern.ch | ||
run: | | ||
echo "Building service: ${SERVICE_NAME}, with tag: ${PYPI_TAG}" | ||
svn checkout https://github.com/dmwm/CMSKubernetes/trunk/docker/pypi/${SERVICE_NAME} | ||
cd ${SERVICE_NAME} | ||
echo "Retrieved Dockerfile with content:" | ||
cat Dockerfile | ||
echo "Sleeping 5min to ensure that PyPi packages are available..." | ||
sleep 300 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why do we need to sleep, what do you mean that PyPi packages are available? Once you download them they should be available and I don't see any needs for sleep. Why 300 and not any other number? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this we discussed when we were commissioning the build workflow. There is a delay between publishing a package to the PyPi repository and fetching it. This 5min is a fair enough commitment that we came up with in the past and which allows us to use the PyPi package when building the docker image. |
||
docker build --build-arg TAG=${PYPI_TAG} --tag ${CERN_REGISTRY}/cmsweb/${SERVICE_NAME}:${PYPI_TAG} . | ||
echo "Image build process completed. Current images are:" | ||
docker images | ||
echo "Now push new image to the CERN registry" | ||
docker push ${CERN_REGISTRY}/cmsweb/${SERVICE_NAME}:${PYPI_TAG} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need
svn
to checkout github repo? Why not to usegit
for that?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For this question, please refer to: #11639 for full context.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alan, even though ticket lists that you swap
curl
withsvn
it does not provide a reason, and neither address why you usecurl/svn
to get git repo files, why not to use (tool designed for that)git
? Please note that in all CMSKubernetes yaml file I always relies on git, e.g.RUN git checkout tags/$TAG -b build
see https://github.com/dmwm/CMSKubernetes/blob/master/docker/dbs2go/Dockerfile#L33There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@todor-ivanov has done the research on the
svn
util. My understanding is that withsvn
, we can fetch a sub-directory of the repository, whilegit
does not provide this capability and we would have to clone the whole repository for each of the 11(?) services that we have setup in the actions workflow.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, thanks for providing the reason. Said that, this line is not doing what the comment is saying, i.e. it will fetch latest subdir and not a particular tag. I do not know
svn
enough to answer if it can fetch proper tag of sub-dir, butcurl
cat fetch tarball of particular tag, andgit
indeed will fetch entire repo for that tag. So, we should fix eithersvn
or switch back tocurl
to fetch exact tag of the repo.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Valentin, the CMSKubernetes code that we check out isn't really tagged (we always fetch what is in HEAD).
The PYPI_TAG is indeed used, but then in the scope of WMCore (used for the tag in pypi and the docker build argument).
Having said that, I would say it works as expected and there is nothing to be changed here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @vkuznet @amaltaro
Yes this is the exact reason.
SVN is fully capable of fetching particular tag.