Integrate into OpenMS repository?

[jpfeuffer]

@mwalzer @hroest @timosachsenberg @lukaszimmermann
I would like to hear opinions on integrating the dockerfiles into the OpenMS repository.
I think that is how the Dockerhub triggers are supposed to work. Whenever there is a change in the repo (on whitelisted branches, e.g. nightly, release*, qt5) it will use the dockerfiles in there to build it.
We don't have to care about maintaining folders for different branches, and have better tagging support for the dockerhub builds.

[lukaszimmermann]

Build are triggered upon pushing and branches are connected to tags. So probably it makes sense to:

• Register this repo on Dockerhub for automated build
• Match tags and branch names (I think ideally they are identical)

Then each push to a branch triggers the build of the tag on Docker hub (but the builds can also be triggered upon request).

[jpfeuffer]

The thing is, we have to push to this repository everytime we want to build and branches have to be hardcoded and changed in all the dockerfiles.

[jpfeuffer]

If we include it into OpenMS. It automatically checks out the sources of the branch where the Dockerfile lies. And a build is triggered whenever there is a push to the actual sources.

[lukaszimmermann]

So each branch contains one or multiple Dockerfiles, which all pull this very branch and build it in the respective environment. So a new push to that branch also automatically build and thus tests the branch on Dockerhub.

[jpfeuffer]

Yes. I do not see how this here would work without external triggers from e.g. our jenkins (this is how it is right now).

[jpfeuffer]

I am certainly up to suggestions.

[lukaszimmermann]

For a start, there could be the following files in the OpenMS repo:

• /dockerfiles/ubuntu
• /dockerfiles/centos
• /dockerfiles/archlinux
• /dockerfiles/debian

Which build the respective branch on the distribution. Does this make sense? Another question: Is building enough or should the Dockerfiles also run tests?

[timosachsenberg]

what about biocontainer base image?

[jpfeuffer]

yeah. I just merged the PR. Should be added too I agree.

[jpfeuffer]

Concerning tests: - I think we should maximally add a rudimentary test e.g. if an executable can be run. We are already close to the limit of dockerhub execution time I think.
That is why we e.g. added a specific pyopenms image

[hroest]

The alternative is to just set up a nightly push, docker allows you to query a specific URL with a key and then trigger the build. I think this is way easier and I feel that every push to develop is a bit too much given that it takes some time to build. I do this here: https://hub.docker.com/r/hroest/openms-lib-nightly/ and you have to get a "trigger token". Then I just have a cronjob on a server that triggers the build once per night. Same for the executables https://hub.docker.com/r/hroest/openms-executables-nightly/ we just have to make sure we dont run into a time limit with dockerhub

[tapaswenipathak]

Any actions required for this issue, now? or OpenMS/dockerfiles remains out of OpenMS ready to be consumed by users of OpenMS RTD documentation¹.

Footnotes

1. https://openms.readthedocs.io/en/develop/docs/index.html ↩

[jpfeuffer]

To be honest, I still think it is worth considering moving it (although with GH actions the triggers across repos are easier now). But don't worry. If we decide to do it, we just change the link in the docs.

[timosachsenberg]

yeah I think we should move.
Also check what is still needed from those:
2971026

[timosachsenberg]

For sprint:
move into contrib repository into dockerfiles/ subfolder
2. https://github.com/OpenMS/dockerfiles/tree/master/contrib

move into OpenMS repository into dockerfiles/ subfolder:
3. https://github.com/OpenMS/dockerfiles/tree/master/executables
4. https://github.com/OpenMS/dockerfiles/tree/master/library
5. https://github.com/OpenMS/dockerfiles/tree/master/pyOpenMS

Adapt actions to use them from their own repository instead of cloning the dockerfile repo.
E.g. https://github.com/OpenMS/OpenMS/blob/develop/.github/workflows/containerdeploy.yml#L36

[KyowonJeong]

What is the meaning of "move into contrib repository into dockerfiles/ subfolder" ?? move into xx into xx does not sound clear to me.

[timosachsenberg]

We just want to move the Dockerfile from https://github.com/OpenMS/dockerfiles/tree/master/contrib
into the OpenMS/contrib repository.

The OpenMS/contrib repository is also on github with all the XML, zlib, boost, stuff. Third-party dependencies.

So what you need to do:

1. make a pull-request that adds the Dockerfile from https://github.com/OpenMS/dockerfiles/tree/master/contrib to OpenMS/contrib (in a folder "dockerfiles" that you need to create)
2. make a pull-request that removes the folder https://github.com/OpenMS/dockerfiles/tree/master/contrib from the OpenMS/dockerfiles repository

[KyowonJeong]

I see. it seems it is a really easy one. I will definitely try it! Thanks for your lesson!

[KyowonJeong]

This will be Kyowon's next sprint, BTW.

[KyowonJeong]

I created 2 PRs: one in OpenMS/dockerfiles and the other OpenMS/contrib. I tested in my local machine but no error found yet. Please let me know if anything is not correct!