CI auto trigger still not working properly

[dvojtise]

Our use of multibranch pipeline is quite convenient for testing branches before accepting PR

however, the auto trigger of the job still doesn't work properly. and we still have to start the build manually

When commiting on a branch of one of the sub repositories, this does not trigger the build.

apparently on internet they suggest to use one multibranch pipeline job per repo (each with its own jenkins file), this would work but this is not acceptable for us as the build is too big and would take too much space on disk.

One alternative to consider is:

• one main repo having the jenkins file and using git module to get the content. Then configure jenkins on this repo.

The question that remains is how to automate the creation/update/delete of the branches/modules in this main repo according to our workflow (all repo on the same branch name or on master branch as a fallback). (ie. for every sub repo make sure to move the module to the last commit (head) on every branch, also if a sub module as a branch create a branch with the same name, if no more branch exist with a given name, remove the corresponding branch on the main repo)

[combemale]

"on internet" ? is it a similar context to us?

It's really important to document.

[dvojtise]

If I remember well this was https://stackoverflow.com/questions/38821362/jenkins-pipeline-code-auto-trigger-with-multiple-repositories-through-github-org

This is due to several points we use together:

• we use several repositories
• each new feature is developed using a branch, a feature may require changes on only one repository, or on several ones. If a feature does not require changes on a repository, we consider that it uses the master branch of this repository.
• each branch is compiled (and tested) (using multibranch pipelines) in order to validate pull request before accepting them.
• having a jenkinsfile per repository, (and maintaining it to be the same on each repo) may help to trigger on any repo, but (If I remember well,) it also tends to duplicate job workspaces on disk depending on which repos was touched and triggerred, it can also be triggered twice and produce strange results
• the jenkinsfile hosted in the repos themselves are ok for build by eclipse organization, but not for build for gemoc organization, since this second works on forked repositories, the uri contained in the jenkinsfile are wrong (and the other jenkins may not be strictly be configured the same way (not the same tool installed, not the same credentials too) thus the "one jenkinsfile per repo" approach does not work for gemoc organization

The solution I propose (and I'm currently experimenting) works as follow:

• one root repository contain a jenkinsfile AND all considered GEMOC repositories as gitsubmodules
• a job (periodically triggered) is in charge to:
  • make sure that every branches in one of the gemoc repositories also exists with the same name in the root repository. (i.e the root repository branches then include all the submodules with the corresponding branch name or fallback to master branch)
  • make sure that all submodules for a given branch use the latest revision
• a job (the main one) is a simple multibranch pipeline connected to the root repository. each of its branches should be triggered by any changed in one of its git submodules (since they have been pushed by the other job)

If this works, this can be duplicated on both eclipse and gemoc organizations (ie. official commiter infrastructure + external contributor infrasructure for our phd student and other partners)

[dvojtise]

I recently worked on a better automatisation of GEMOC multibranch pipeline (cf. #142).

• previous situation:
  • branches were taken into account only if a branch with the same name exist in the repo https://github.com/eclipse/gemoc-studio for http://ci.eclipse.org/gemoc (or in https://github.com/gemoc/gemoc-studio-jenkins for http://ci.inria.fr/gemoc) the branch was required even if there were no commit in it.
    • Additionnaly, as the eclipse and the gemoc fork doesn't use the same ci and configuration the build architecture was slightly different (the additional gemoc/gemoc-studio-jenkins repo)
  • In many situation, a manual triggering was required:
  • push in branches of branched repo didn't trigger the build of the branches
  • push in master branch didn't trigger the build of branches where the repo to consider remains "master" (ex. branch "feature-x" in repo moccml, branch master for all the other repo, a commit in master in one of these other repo did not trigger the build of pipeline "feature-x")

So the new jobs try to have the same behavior as before while fixing these issues, without additional task for developpers.

Basically, it improves the "scan multibranch pipeline" action by adding a preliminary phase that automates branch collection into git modules in a dedicated repository.

Internally how it works:

• Gemoc maintainers @eclipse organization, use the repo https://github.com/gemoc/gemoc-studio-eclipse-integration , in the integration branch (master) they define a list of tracked git modules pointing to all the gemoc repositories to consider
• based on https://github.com/dvojtise/git-sync , a job periodically (5 min) scans these modules in order to detect branches in any of the git modules, for each branch name, it creates one in the eclipse-integration repo and make sure that its git modules follow the given branch or master branch. It also makes sure to update the git ref to point to the latest commit of all branches. (Obviously, it takes care of branch addition and deletion)
• a http://ci.eclipse.org/gemoc/job/gemoc-studio-integration job (replacing the old http://ci.eclipse.org/gemoc/job/gemoc-studio job) is then a simple multibranch pipeline that is triggered correctly

So, once pushed on github, your branches are supposed to be available automatically (after the polling delays)