-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add an inactive proof-of-concept GitLab-CI config file #411
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mkszuba
force-pushed
the
feature/gitlab-ci_poc
branch
from
August 22, 2019 16:13
4a9ebae
to
61f1c46
Compare
(in case it isn't obvious, it has been deactivated by inserting '.inactive' into the correct file name) Unlike Ubuntu-based Travis builds, on GitLab-CI we use official Perl images from Docker Hub. These are Debian-based so no major changes have been required, support both perl-5.14 and perl-5.26 without any fuss (and even if they eventually removed images for some version we could always grab their relevant Dockerfiles and rebuild them ourselves), and since they only contain core Perl they give us better control over our dependency trees (i.e. much less risk of forgetting to add a non-core module to cpanfile because it was already there as distro package on the developer's machine). What works: - the test suite runs and passes for both perl-5.14/sqlite and perl-5.26/mysql. What still needs to be done to achieve full feature parity with Travis builds: - enable Coveralls checks for 5.26/mysql jobs. Should work in theory but would require adjusting coveralls.io configuration to accept reports from EBI GitLab; - trigger dependent builds. According to GitLab-CI documentation this is quite simple to do but unsurprisingly, the relevant GitLab API differs from its Travis counterpart so we cannot simply re-use the existing script. Besides, for the time being we do not mirror any of the dependent repos on EBI GitLab; - disable API version checks in non-release builds. Left out on purpose because we might have the whole of Ensembl officially adopt the early-version-bump policy by the time this is put into use. Does not have to be implemented in GitLab-CI: - configuration of e-mail and Slack notifications - in GitLab this is done in projects, not at CI level. What could be done even though it isn't done in Travis: - create our own build images with all the common dependencies already installed. GitLab repositories can function as Docker-compatible container registries so it is quite simple to create a new repo with this feature enabled, shove a couple of Dockerfiles into it which install the necessary common dependencies onto base Perl images, and set up CI for that repo so that the base images are periodically rebuilt in order to account for possible updates. Result: simpler API-level CI configuration, faster runs, less bandwidth wasted; - process cpanfiles in ensembl-io and ensembl-variation. These are problematic for two reasons: * both depend on Bio::DB::BigFile, which requires messing with Kent libraries and linker flags in order to work properly, * ensembl-variation depends on Bio::DB::HTS, which pulls in BioPerl, which unless explicitly instrumented to install in minimal mode depends on known-to-be-broken GD module. At present we get away with not processing them because unlike ensembl-compara (which luckily does not demand any difficult modules), Core unit tests depending on -io and -variation code only import modules which do not use either of the above. This of course is a lucky co-incidence and cannot be relied upon to last. We have already got scripts for dealing with both Bio::DB::BigFile and GD, in Otter for instance, so we could in principle address this right away. Then again, in my opinion it would be better to delegate this to the aforementioned custom build images and keep repository-specific CI scripting as lean as possible; - setting passwords for MySQL users. Trivial to do but in order to avoid having those passwords in plain text in .gitlab-ci.yml, we should either set them as GitLab project variables or encrypt them somehow.
mkszuba
force-pushed
the
feature/gitlab-ci_poc
branch
from
August 23, 2019 12:51
61f1c46
to
aedc621
Compare
ens-bwalts
reviewed
Aug 27, 2019
ens-bwalts
approved these changes
Aug 29, 2019
Reminder: this should be "squash-and-merge"d |
s-mm
approved these changes
Sep 3, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add an inactive (in case it isn't obvious, it has been deactivated by inserting
'.inactive' into the correct file name) proof-of-concept GitLab-CI config file.
Unlike Ubuntu-based Travis builds, on GitLab-CI we use official Perl
images from Docker Hub. These are Debian-based so no major changes have
been required, support both perl-5.14 and perl-5.30 without any fuss
(and even if they eventually removed images for some version we could
always grab their relevant Dockerfiles and rebuild them ourselves), and
since they
only contain core Perlcontain only a few most commonnon-core modules in default images (which is what this uses now) and
only core Perl in case of slim ones, they give us better control over our
dependency trees (i.e. much less risk of forgetting to add a non-core
module to cpanfile because it was already there as distro package on the
developer's machine).
What works:
perl-5.30/mysql;
secret variable in GitLab CI configuration is set to the correct token from coveralls.io;
What still needs to be done to achieve full feature parity with Travis
builds:
Does not have to be implemented in GitLab-CI:
done in projects, not at CI level.
What could be done even though it isn't done in Travis:
installed. GitLab repositories can function as Docker-compatible
container registries so it is quite simple to create a new repo with
this feature enabled, shove a couple of Dockerfiles into it which
install the necessary common dependencies onto base Perl images, and
set up CI for that repo so that the base images are periodically
rebuilt in order to account for possible updates. Result: simpler
API-level CI configuration, faster runs, less bandwidth wasted;
problematic because they both depend on Bio::DB::BigFile, which
requires messing with Kent libraries and linker flags in order to
install properly,
At present we get away with not processing them because unlike
ensembl-compara (which luckily does not demand any difficult
modules), Core unit tests depending on -io and -variation code only
import modules which do not use either of the above. This of course
is a lucky co-incidence and cannot be relied upon to last.
We have already got scripts for dealing with both Bio::DB::BigFile
and GD, in Otter for instance, so we could in principle address this
right away. Then again, in my opinion it would be better to delegate
this to the aforementioned custom build images and keep
repository-specific CI scripting as lean as possible;
avoid having those passwords in plain text in .gitlab-ci.yml, we
should either set them as GitLab project variables or encrypt them
somehow.
Use case
Our Travis space has really been struggling to keep up with the load we impose on it, especially in ensembl-rest. Moreover, with Ubuntu 14.04 having reached end of life in April 2019 the "trusty" build environment is now deprecated, and perl-5.14 is not officially supported by Travis for Xenial and newer. Finally, with many features of Travis being reserved for paying customers we are forced do things not entirely efficiently: having to install all common dependencies every build, not running Travis on private repositories etc.
At the same time, we have at EBI an instance of Gitlab Enterprise Edition whose CI system (which has already been paid for as part of the Enterprise licence) offers more features than Travis CE, which integrates better with Jira than GitHub (to the best of my knowledge, GitHub Jira integration only works with the cloud edition of the latter) and which can either fully mirror GitHub repositories (there is also a possibility of mirroring whole projects - issues, PRs, you name it - but WebProd have told me that so far they have had problems enabling it without simultaneously letting people log in with GitHub account and therefore have decided to keep it off for now) or simply run CI jobs for repositories still located on GitHub, meaning we could use this without actually migrating off GH.
Benefits
Demonstrates that it is not only feasible but in fact quite straightforward to run Ensembl CI jobs on EBI GitLab.
Possible Drawbacks
Care should be taken to keep this file in sync with .travis.yml in the event of changes to the latter. For this reason, GitLab-CI configuration has been made as similar to Travis configuration as possible.
Testing
Have you added/modified unit tests to test the changes?
No.
If so, do the tests pass/fail?
N/A
Have you run the entire test suite and no regression was detected?
Neither the test suite itself nor Travis builds are affected by this. GitLab-CI builds succeed.