Add initial draft for building WMCore images via CI/CD #11339

vkuznet · 2022-10-11T16:31:01Z

Fixes #10921

Status

In development

Description

Add new CI/CD workflow to build WMCore images

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

#11318

External dependencies / deployment changes

GitHub Actions workflow

vkuznet · 2022-10-11T16:37:20Z

And, I adjusted CMSKubernetes area to use explicit TAG version in all WMCore Dockefiles, see dmwm/CMSKubernetes@942145b. The tag used in Dockefiles will be replaced by the tag used during a build, see relevant lines in this PR.

vkuznet · 2022-10-11T16:40:42Z

This PR also contains commented sections which we will enable later on once all images will be build. The commented sections require to add CERN registry credentials and we should decide if we'd like to automatically upload images to testbed/testX clusters via imagebot daemon. To do that, we should explicitly change imagebot daemon configuration to add section for appropriate services. If you'll inspect last section you'll see how it is done for all other services, e.g. DAS, WMArchive, etc., on testbed. This step is optional though and we can easily discard it if we do not want automatic deployment of images to testbed/testX clusters. I already instructed @muhammadimranfarooqi and @arooshap to install imagebot on testX clusters such that it will be available for automatic image deployment.

cmsdmwmbot · 2022-10-11T16:43:22Z

Jenkins results:

Python3 Unit tests: failed
- 25 new failures
- 1 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13635/artifact/artifacts/PullRequestReport.html

.github/workflows/images.yaml

cmsdmwmbot · 2022-10-13T14:53:26Z

Jenkins results:

Python3 Unit tests: succeeded
- 4 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13654/artifact/artifacts/PullRequestReport.html

amaltaro

@vkuznet thanks for providing this action, Valentin.

I have a few remarks to make:

creating a docker image depends on having the Pypi packages available. So there is a synchronization step that we need to consider when building both PyPi and Docker images on top of a freshly create tag.
I also left a comment along the code for the tags, where there is a pattern that needs to be excluded
on what concerns the images that we need to build, please check this list (some are missing here): https://github.com/dmwm/WMCore/wiki/Managing-WMCore-services-in-CMSWEB-kubernetes#list-of-services-and-images-that-need-to-be-provided-from-wmcore

Out of the scope of this PR, I think we should fist close the loop into these new Dockerfiles before throwing many things in parallel (there are many GH issues ongoing, and some should actually be preceding this one).

.github/workflows/images.yaml

cmsdmwmbot · 2022-10-14T17:21:03Z

Jenkins results:

Python3 Unit tests: succeeded
- 2 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13664/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-10-14T17:23:03Z

Jenkins results:

Python3 Unit tests: succeeded
- 1 tests no longer failing
- 3 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13665/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-10-14T18:16:33Z

Jenkins results:

Python3 Unit tests: failed
- 2 new failures
- 3 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13666/artifact/artifacts/PullRequestReport.html

vkuznet · 2022-10-28T18:00:24Z

@amaltaro , @khurtado I updated this PR and switched from Dockerfile which relies on pip install <pkg>==$TAG to Dockerfile.dist which uses pip install <pkg>-$TAR.tar.gz. This difference allows to use dist tar ball which is created before pushing to pypi in order to build a docker image with it. This will resolve multi-stage procedure Alan was complaining (i.e. we need to push to pypi in order to use pypi), and we can use this trick in CI/CD.

@khurtado , could you please try it out as separate step in CI/CD workflow on GitHub Action such that we can test this procedure.

For the record here is a diff between Dockerfile and Dockerfile.dist for reqmgr2 service:

diff -u pypi/reqmgr2/Dockerfile pypi/reqmgr2/Dockerfile.dist
--- pypi/reqmgr2/Dockerfile     2022-10-11 18:46:58.000000001 +0200
+++ pypi/reqmgr2/Dockerfile.dist        2022-10-28 19:51:43.000000001 +0200
@@ -4,7 +4,7 @@
 RUN apt-get install -y curl libcurl4 libcurl4-openssl-dev python3-pycurl pip
 RUN ln -s /usr/bin/python3 /usr/bin/python
 ENV TAG=2.1.4rc3
-RUN pip install reqmgr2==$TAG
+RUN pip install reqmgr2-$TAG.tar.gz
 ENV WDIR=/data
 WORKDIR $WDIR
 CMD ["python3"]

and you can find both files in this CMSKubernetes/docker/pypy/reqmgr2 area.

cmsdmwmbot · 2022-10-28T18:09:53Z

Jenkins results:

Python3 Unit tests: succeeded
- 2 tests no longer failing
- 2 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13693/artifact/artifacts/PullRequestReport.html

amaltaro

Valentin, there are a few comments along the code and one of them needs to be definitely addressed.

There is still something that I don't quite get. From the Dockerfile.dist:

RUN pip install reqmgr2ms-monitor-$TAG.tar.gz

how will it work if we do not have such tarball? Once we make a new tag, the only thing we have is:

https://github.com/dmwm/WMCore/archive/refs/tags/2.1.4.tar.gz

Is there something internally that does a magic mapping?

amaltaro · 2022-11-15T20:13:05Z

.github/workflows/images.yaml

+  push:
+    tags:
+      - '*'
+      - '!JENKINS*'


Given that we are currently struggling with this syntax in the PyPi workflow, I checked the documentation and it seems to be one of the ways to allow/disallow actions based on the tag name.
Another option would be:

tags-ignore: - 'JENKINS*'

as documented in: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onpushbranchestagsbranches-ignoretags-ignore

Just an FYI @khurtado

amaltaro · 2022-11-15T20:19:17Z

.github/workflows/images.yaml

+
+    - name: Build t0_reqmon image
+      run: |
+        curl -ksLO https://raw.githubusercontent.com/dmwm/CMSKubernetes/master/docker/pypi/t0_reqmon/Dockerfile.dist


We still do not have this t0_reqmon package in PyPi. I need to review it, but I think the Dockerfile will have to:

RUN pip install reqmon-$TAG.tar.gz

we need to investigate if:

t0_reqmon is even expected to be packaged in WMCore

if so, if the package is the same as reqmon. It could be that it's during service start up that we decide which CouchApps to use and whatnot.

amaltaro · 2022-11-15T20:19:50Z

.github/workflows/images.yaml

+
+    - name: Build reqmgr2 image
+      run: |
+        curl -ksLO https://raw.githubusercontent.com/dmwm/CMSKubernetes/master/docker/pypi/reqmgr2/Dockerfile.dist


This Dockerfile seems to be incorrect.

amaltaro · 2022-11-15T20:30:20Z

.github/workflows/images.yaml

+
+    - name: Build reqmgr2ms-unmerged image
+      run: |
+        curl -ksLO https://raw.githubusercontent.com/dmwm/CMSKubernetes/master/docker/pypi/reqmgr2ms-unmerged/Dockerfile.dist


For this MSUnmerged, I see that the Dockerfile actually depends on two images:

FROM registry.cern.ch/cmsweb/gfal:latest as gfal FROM python:3.8-bullseye

can you please point to this gfal Dockerfile?

vkuznet · 2022-11-16T00:03:19Z

Alan, of course there is no magic how tar ball should appear in Dockerfile.dist, there are few possibilities which I left out until we make a decision:

we can fetch a tarball from github, this can be done if someone will upload such tarball
we can use Dockerfile.dist within pypi build yaml file where this tar ball is created. In this case we need to add the following line into Dockerfile.dist ADD reqmgr2ms-monitor-$TAG.tar.gz reqmgr2ms-monitor-$TAG.tar.gz where concrete tarball will be first created
or, we can use git checkout of specific tag and invoke python setup.py sdist call to create tar ball

So far, I provided a skeleton Dockerfile.dist file for each package to show how we can build an image from a specific tar ball. The question how to get tarball depends on how we will decide create images. My preference is that concrete image should be created within pypi GitAction build, e.g. right after this section https://github.com/dmwm/WMCore/blob/master/.github/workflows/pypy_build_publish_template.yaml#L33 This line creates tarball and if we'll make a new section where we'll Dockerfile.dist with ADD line then we'll create an image out of newly created tarball.

amaltaro · 2022-11-16T14:20:09Z

My take on this is that we should not use tarball to deploy WMCore python-based applications. The reason is that we miss the logic embedded in the requirements.txt and setup scripts used to build the PyPi packages with the proper 3rd-party libraries dependencies.

Of course we could somehow do the same in the image built script/GH action, but then what is the point of uploading packages to PyPi if we are not using them to create our docker images and deploy services with kuberentes(?)

vkuznet · 2022-11-16T14:49:01Z

Alan, I'm not sure I understand your comment. The line https://github.com/dmwm/WMCore/blob/master/.github/workflows/pypy_build_publish_template.yaml#L33 creates a tarball. And, next section takes tar ball to upload it to pypi. So, during GH Action we create tarballs, and my suggestion to add section which will create images before or after we submit tarball to pypi. This is why everything can be automated within the same GH Action, i.e. it create new tarball based on give tag, it uploads tarball to pypi and it creates corresponding image and upload it to registry. If you need further clarification I suggest to discuss it via zoom chat. Please let me know.

amaltaro · 2022-11-16T15:00:20Z

I see. I guess I was mixing up the WMCore tag tarball from the tarball that gets published to PyPi. If we use it from PyPi, then we indeed get all the python required dependencies in place. I agree that this is what we should do.

I think it would be great if we had a way to easily separate these tasks into different GH actions. But expanding the current action template will do the job as well.

@khurtado since you wrote the PyPi action scripts, do you have any concerns or opinions on this subject?

vkuznet · 2022-11-27T19:09:24Z

Finally, I managed to build all WMCore MS images via CI/CD GH Action. Please see

I used separate repo to test GH Action, and in this repo I uploaded and modified pypy_build_publish.yml and pypy_build_images_template.yml to include step to build docker images. From GH Action page you can see that now all of them are successfully built. I suggest that we may close this issue and use yaml files I provided in https://github.com/vkuznet/wmcore-builds

vkuznet · 2022-11-28T16:15:11Z

We may close this issue in favor of #11374

amaltaro · 2022-11-29T02:36:28Z

@vkuznet Valentin, I am not sure I understand what you said.

We have been working on this issue: #10921
which was meant to be closed by this PR.

However, you created a new issue: #11374 and you say the changes provided in this PR are no longer relevant. Besides a possible deployment in k8s, what is the difference between #10921 and #11374 ?

vkuznet · 2022-11-29T12:19:45Z

@amaltaro , on Monday meeting you requested to create a new issue and even put an Action item for it: Valentin: create a PR with the latest CD changes (from Valentin’s own repository). So, here it is, this issue is about my CI/CD image creattion pipeline I created in separate repository, while #10921 is about full automation on k8s using CI/CD pipeline, i.e. it suppose that we'll also work and provide k8s manifest files. The latter step is required once we have images, but image creation has nothing to do with k8s manifest files.

amaltaro · 2022-11-29T13:38:21Z

So, the text says to create a PR - pull request - not a GH issue. The issue itself already existed.

vkuznet · 2022-11-29T15:11:49Z

@amaltaro , I followed the same pattern as with any other development. I created an issue and now you have a PR for it #11377 The issue described the logic of steps, e.g. separate repo, why it was created, etc. The PR now contains pull request to WMCore repo. Feel free to close #11374 if you think it is irrelevant.

vkuznet · 2022-12-01T20:28:46Z

I'm closing this now since it is no longer relevant

Add initial draft for building WMCore images via CI/CD

9f95715

vkuznet added deployment Issue related to deployment of the services containerization Kubernetes labels Oct 11, 2022

vkuznet requested review from amaltaro, khurtado and goughes October 11, 2022 16:31

vkuznet self-assigned this Oct 11, 2022

khurtado reviewed Oct 13, 2022

View reviewed changes

.github/workflows/images.yaml Outdated Show resolved Hide resolved

Fix steps.get-ref parts

0d1835e

amaltaro requested changes Oct 13, 2022

View reviewed changes

.github/workflows/images.yaml Outdated Show resolved Hide resolved

vkuznet added 2 commits October 14, 2022 13:02

Added workqueue and t0_reqmon

fa51a70

Added reqmgr2 service

82904e8

Adjust tagging convention to match other build workflows

07cb8e9

Switch from Dockerfile to Dockerfile.dist

18e98d5

vkuznet mentioned this pull request Nov 6, 2022

Automate docker image creation via GitHub Action CI/CD #10921

Closed

amaltaro self-requested a review November 14, 2022 15:41

amaltaro requested changes Nov 15, 2022

View reviewed changes

amaltaro requested a review from khurtado November 16, 2022 15:00

vkuznet mentioned this pull request Nov 28, 2022

Create WMCore images via CI/CD pipeline #11374

Closed

vkuznet mentioned this pull request Nov 29, 2022

Add new GH Action workflows to build WMCore images #11377

Merged

vkuznet closed this Dec 1, 2022

amaltaro mentioned this pull request Jul 10, 2023

Refactor docker build/push workflow action #11653

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add initial draft for building WMCore images via CI/CD #11339

Add initial draft for building WMCore images via CI/CD #11339

vkuznet commented Oct 11, 2022

vkuznet commented Oct 11, 2022

vkuznet commented Oct 11, 2022

cmsdmwmbot commented Oct 11, 2022

cmsdmwmbot commented Oct 13, 2022

amaltaro left a comment

cmsdmwmbot commented Oct 14, 2022

cmsdmwmbot commented Oct 14, 2022

cmsdmwmbot commented Oct 14, 2022

vkuznet commented Oct 28, 2022 •

edited

Loading

cmsdmwmbot commented Oct 28, 2022

amaltaro left a comment

amaltaro Nov 15, 2022

amaltaro Nov 15, 2022

amaltaro Nov 15, 2022

amaltaro Nov 15, 2022

vkuznet commented Nov 16, 2022

amaltaro commented Nov 16, 2022

vkuznet commented Nov 16, 2022

amaltaro commented Nov 16, 2022

vkuznet commented Nov 27, 2022

vkuznet commented Nov 28, 2022

amaltaro commented Nov 29, 2022

vkuznet commented Nov 29, 2022

amaltaro commented Nov 29, 2022

vkuznet commented Nov 29, 2022

vkuznet commented Dec 1, 2022

Add initial draft for building WMCore images via CI/CD #11339

Add initial draft for building WMCore images via CI/CD #11339

Conversation

vkuznet commented Oct 11, 2022

Status

Description

Is it backward compatible (if not, which system it affects?)

Related PRs

External dependencies / deployment changes

vkuznet commented Oct 11, 2022

vkuznet commented Oct 11, 2022

cmsdmwmbot commented Oct 11, 2022

cmsdmwmbot commented Oct 13, 2022

amaltaro left a comment

Choose a reason for hiding this comment

cmsdmwmbot commented Oct 14, 2022

cmsdmwmbot commented Oct 14, 2022

cmsdmwmbot commented Oct 14, 2022

vkuznet commented Oct 28, 2022 • edited Loading

cmsdmwmbot commented Oct 28, 2022

amaltaro left a comment

Choose a reason for hiding this comment

amaltaro Nov 15, 2022

Choose a reason for hiding this comment

amaltaro Nov 15, 2022

Choose a reason for hiding this comment

amaltaro Nov 15, 2022

Choose a reason for hiding this comment

amaltaro Nov 15, 2022

Choose a reason for hiding this comment

vkuznet commented Nov 16, 2022

amaltaro commented Nov 16, 2022

vkuznet commented Nov 16, 2022

amaltaro commented Nov 16, 2022

vkuznet commented Nov 27, 2022

vkuznet commented Nov 28, 2022

amaltaro commented Nov 29, 2022

vkuznet commented Nov 29, 2022

amaltaro commented Nov 29, 2022

vkuznet commented Nov 29, 2022

vkuznet commented Dec 1, 2022

vkuznet commented Oct 28, 2022 •

edited

Loading