-
Notifications
You must be signed in to change notification settings - Fork 111
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: Docker-based CI #256
WIP: Docker-based CI #256
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a good start, however you should use build stages:
- in build stage "base" build mathcomp and push a docker image
math-comp:dev
etc... that is docked on top of the Coq one but has math-comp installed. You may need to push an image named with the hash of the commit to avoid races. - in the
test
stage, you can indeed then test the proposed CIs in paralell.
That would work well for merging IMHO; once done we could study how to factor the versioning. For now you could use a bash
array indexed by Coq version if you need version-specific pins.
See the docs at https://docs.travis-ci.com/user/build-stages/share-docker-image/ I am not sure if this would work for pull requests tho, I forgot how the permission model works for this. |
Yup indeed this won't work for pull requests :S I got used to gitlab where we workaround that. |
@anton-trunov thanks for working on this. (Afterwards, this could be complemented with some distribution of math-comp stable docker images for which I gave some suggestions in #243, but of course these two things are a bit orthogonal. Also I believe @CohenCyril was also interested in experimenting this stuff) As suggested by @ejgallego you should definitely use some build stages to implement this, cf. https://docs.travis-ci.com/user/build-stages, and indeed Travis CI does not have a Docker registry per se, so you could try to push to Docker Hub or so, but it also seems to me this wouldn't be possible for fork branches, due to the need for (encrypted) private tokens to login into the registry, which can only be decrypted from upstream branches builds, not fork builds. However I think this is not blocking to achieve what we want: you could have 2 stages "build" and "test":
Then, it is one of two things:
FYI there are some references on the net documenting ways to address point 2. above:
|
@ejgallego I guess one of your suggestions (copied below) was dealing with the naming convention of the "mathcomp:dev" image (if it is pushed on Docker Hub at some point) but just to be sure could you give a small example of what you meant here by "bash array indexed by Coq version"?
|
@erikmd thanks a lot for the links! Oh indeed using the cache could be great! It has to be done with care as to trigger the proper rebuild when it is needed, but indeed could work very well in the end. What I meant about the array is that for example imagine project |
git clone https://github.com/math-comp/odd-order | ||
cd odd-order | ||
make -j ${NJOBS} | ||
" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should also add math_comp/fourcolor
as that's also a stable (and sizeable) development.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will do. I added a TODO list to the original post trying to summarize all the suggestions. Feel free to edit the list adding new items or removing the ones you'd prefer not to test.
@ejgallego @erikmd If you really want to use stages and push a docker image for math-comp in the first stage, then I strongly encouraged ditching Travis here and using GitLab CI instead. The experience with Coq has been very positive. |
@Zimmi48 Does one need a Gitlab organization for this? |
@Zimmi48 thanks for your feedback! actually I guess the plan consists in both:
as point 2. doesn't require to push the first Docker image somewhere (the cache could be used to communicate inter-stages) this approach could be feasible using Travis. However it's true that GitLab CI's Docker support is excellent (the configuration file obtained in the end would certainly be simpler than with Travis, and point 2. above regarding PRs could also be extended with a |
Hi @ejgallego! you said about the approach of using staged builds with several docker images:
but out of curiosity, do you rely on some dedicated tool for that (related maybe to coqbot), or do you just use a standard GitLab CI feature mentioned in its documentation? |
Ok, the docs say that as the first step one should do the following:
Where do we do this? I guess (@CohenCyril or @gares or @amahboubi) could create a new GitLab organization for this? Or is there some generic org for Coq-related project? Any pointers would be much appreciated. |
@erikmd @anton-trunov @amahboubi @gares Should I create a |
@CohenCyril I don't know for sure, but it seemed to me that @Zimmi48's suggestion was to create that group on https://gitlab.com/ to benefit from its GitLab CI runners? (like https://gitlab.com/coq/coq) |
BTW (in anticipation of #243) you could create a |
Indeed, I had not thought of the possibility of using the cache of Travis. In the old days when we started using Travis (almost 2 years ago), it was not possible to share the cache between stages. |
Hi @Zimmi48
OK! Note anyway that I did not experimented the references mentioned in my comment and in particular if this approach would now be applicable between different stages of the same Travis build. In any case, relying on gitlab.com CI seems a good choice. BTW @Zimmi48 @ejgallego are you aware of specific limitations regarding the CI time of standard runners on gitlab.com CI? |
No time limitations that should be relevant for math-comp. Indeed I am not sure how caches do work wrt stages. |
Caches do seem to work well with stages; another note: gitlab's CI registry allows to get a deploy-only token for read-only access. |
Done: https://gitlab.inria.fr/math-comp EDIT: Sorry! I did not read carefully. Mistake corrected |
Regarding Emilio's answer to my question about GitLab CI time limitations
I've looked at https://about.gitlab.com/solutions/github/ and https://about.gitlab.com/pricing/#gitlab-com and it's not clear for me if we'd benefit from 2,000 CI pipeline minutes or 50,000 CI pipeline minutes… Out of curiosity, could @Zimmi48 or @ejgallego take a look at the following page to see if that info would be available for https://gitlab.com/coq/coq ? |
Open source projects have no limitations but no guaranteed shared runners either. In practice it works very well as the on-demand approach means that backlog is caught up quickly. Also it is fairly easy to add your own runners, in Coq we have a machine that helps in times of reduced worker availability. |
Just not to forget and to follow-up the discussion at the visio meeting, you may want to create an organization at https://gitlab.com/math-comp as well. (see also that docs link mentioned by @anton-trunov: Cc @CohenCyril |
Indeed, as explained by @erikmd the proposal was to create the organization on gitlab.com not on Inria's GitLab. |
Hi @anton-trunov! did you perform more experiments regarding the migration/setup of mathcomp's CI on GitLab? otherwise I can propose to look at this tonight (relying on the GitLab CI config example that I recently simplified to avoid the |
@erikmd I was intermittently reading the docs and experimenting with CI on IMDEA's internal Gitlab, since I have the necessary rights there, but I haven't finished yet. Feel free to take over this PR any time, though. |
Thanks @anton-trunov! so I resumed my experiments as well, diving in GitLab's doc and in some examples of CI setup including that of Coq. |
OK, I might not be very reactive though, but I'll try to help if I can. |
Superseded by #266 |
* Update INSTALL.md for installing mathcomp as local opam packages This patch documents the change 1046da9 that was required by the GitLab CI setup to be able to rely on opam 2.0. Close #251 * Propose to bump the required version of coq in coq-mathcomp-ssreflect.opam so the CI could use the Docker image coqorg/coq:8.9 (which is already available albeit it currently is a synonymous of coqorg/coq:8.9-beta1 with the opam repos coq-released + coq-core-dev + coq-extra-dev) * [ci] Add math-comp/odd-order and coq-community/lemma-overloading jobs Related: #245 and #256
* Update INSTALL.md for installing mathcomp as local opam packages This patch documents the change 1046da9 that was required by the GitLab CI setup to be able to rely on opam 2.0. Close math-comp#251 * Propose to bump the required version of coq in coq-mathcomp-ssreflect.opam so the CI could use the Docker image coqorg/coq:8.9 (which is already available albeit it currently is a synonymous of coqorg/coq:8.9-beta1 with the opam repos coq-released + coq-core-dev + coq-extra-dev) * [ci] Add math-comp/odd-order and coq-community/lemma-overloading jobs Related: math-comp#245 and math-comp#256
Trying to fix #245. I added https://github.com/math-comp/odd-order to CI. I did a quick hack, so I'd appreciate if someone reviewed this PR.
CC @erikmd @ejgallego (and just in case, @Zimmi48).
TODO: