Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an inactive proof-of-concept GitLab-CI config file #411

Merged
merged 5 commits into from
Sep 3, 2019

Conversation

mkszuba
Copy link
Contributor

@mkszuba mkszuba commented Aug 22, 2019

Description

Add an inactive (in case it isn't obvious, it has been deactivated by inserting
'.inactive' into the correct file name) proof-of-concept GitLab-CI config file.

Unlike Ubuntu-based Travis builds, on GitLab-CI we use official Perl
images from Docker Hub. These are Debian-based so no major changes have
been required, support both perl-5.14 and perl-5.30 without any fuss
(and even if they eventually removed images for some version we could
always grab their relevant Dockerfiles and rebuild them ourselves), and
since they only contain core Perl contain only a few most common
non-core modules in default images (which is what this uses now) and
only core Perl in case of slim ones, they give us better control over our
dependency trees (i.e. much less risk of forgetting to add a non-core
module to cpanfile because it was already there as distro package on the
developer's machine).

What works:

  • the test suite runs and passes for both perl-5.14/sqlite and
    perl-5.30/mysql;
  • enable Coveralls reporting should work for 5.30/mysql jobs if an appropriate
    secret variable in GitLab CI configuration is set to the correct token from coveralls.io;
  • it is possible to trigger dependent jobs for downstream projects configured to run GitLab-CI.

What still needs to be done to achieve full feature parity with Travis
builds:

  • tweak the condition of triggering dependent builds so that they appropriately cover pull requests.

Does not have to be implemented in GitLab-CI:

  • configuration of e-mail and Slack notifications - in GitLab this is
    done in projects, not at CI level.

What could be done even though it isn't done in Travis:

  • consider supporting dependent jobs in repositories which do not support GitLab-CI;
  • create our own build images with all the common dependencies already
    installed. GitLab repositories can function as Docker-compatible
    container registries so it is quite simple to create a new repo with
    this feature enabled, shove a couple of Dockerfiles into it which
    install the necessary common dependencies onto base Perl images, and
    set up CI for that repo so that the base images are periodically
    rebuilt in order to account for possible updates. Result: simpler
    API-level CI configuration, faster runs, less bandwidth wasted;
  • process cpanfiles in ensembl-io and ensembl-variation. These are
    problematic because they both depend on Bio::DB::BigFile, which
    requires messing with Kent libraries and linker flags in order to
    install properly,
    At present we get away with not processing them because unlike
    ensembl-compara (which luckily does not demand any difficult
    modules), Core unit tests depending on -io and -variation code only
    import modules which do not use either of the above. This of course
    is a lucky co-incidence and cannot be relied upon to last.
    We have already got scripts for dealing with both Bio::DB::BigFile
    and GD, in Otter for instance, so we could in principle address this
    right away. Then again, in my opinion it would be better to delegate
    this to the aforementioned custom build images and keep
    repository-specific CI scripting as lean as possible;
  • setting passwords for MySQL users. Trivial to do but in order to
    avoid having those passwords in plain text in .gitlab-ci.yml, we
    should either set them as GitLab project variables or encrypt them
    somehow.

Use case

Our Travis space has really been struggling to keep up with the load we impose on it, especially in ensembl-rest. Moreover, with Ubuntu 14.04 having reached end of life in April 2019 the "trusty" build environment is now deprecated, and perl-5.14 is not officially supported by Travis for Xenial and newer. Finally, with many features of Travis being reserved for paying customers we are forced do things not entirely efficiently: having to install all common dependencies every build, not running Travis on private repositories etc.

At the same time, we have at EBI an instance of Gitlab Enterprise Edition whose CI system (which has already been paid for as part of the Enterprise licence) offers more features than Travis CE, which integrates better with Jira than GitHub (to the best of my knowledge, GitHub Jira integration only works with the cloud edition of the latter) and which can either fully mirror GitHub repositories (there is also a possibility of mirroring whole projects - issues, PRs, you name it - but WebProd have told me that so far they have had problems enabling it without simultaneously letting people log in with GitHub account and therefore have decided to keep it off for now) or simply run CI jobs for repositories still located on GitHub, meaning we could use this without actually migrating off GH.

Benefits

Demonstrates that it is not only feasible but in fact quite straightforward to run Ensembl CI jobs on EBI GitLab.

Possible Drawbacks

Care should be taken to keep this file in sync with .travis.yml in the event of changes to the latter. For this reason, GitLab-CI configuration has been made as similar to Travis configuration as possible.

Testing

Have you added/modified unit tests to test the changes?

No.

If so, do the tests pass/fail?

N/A

Have you run the entire test suite and no regression was detected?

Neither the test suite itself nor Travis builds are affected by this. GitLab-CI builds succeed.

(in case it isn't obvious, it has been deactivated by inserting
'.inactive' into the correct file name)

Unlike Ubuntu-based Travis builds, on GitLab-CI we use official Perl
images from Docker Hub. These are Debian-based so no major changes have
been required, support both perl-5.14 and perl-5.26 without any fuss
(and even if they eventually removed images for some version we could
always grab their relevant Dockerfiles and rebuild them ourselves), and
since they only contain core Perl they give us better control over our
dependency trees (i.e. much less risk of forgetting to add a non-core
module to cpanfile because it was already there as distro package on the
developer's machine).

What works:
 - the test suite runs and passes for both perl-5.14/sqlite and
   perl-5.26/mysql.

What still needs to be done to achieve full feature parity with Travis
builds:
 - enable Coveralls checks for 5.26/mysql jobs. Should work in theory but
   would require adjusting coveralls.io configuration to accept reports
   from EBI GitLab;
 - trigger dependent builds. According to GitLab-CI documentation this
   is quite simple to do but unsurprisingly, the relevant GitLab API
   differs from its Travis counterpart so we cannot simply re-use the
   existing script. Besides, for the time being we do not mirror any of
   the dependent repos on EBI GitLab;
 - disable API version checks in non-release builds. Left out on purpose
   because we might have the whole of Ensembl officially adopt the
   early-version-bump policy by the time this is put into use.

Does not have to be implemented in GitLab-CI:
 - configuration of e-mail and Slack notifications - in GitLab this is
   done in projects, not at CI level.

What could be done even though it isn't done in Travis:
 - create our own build images with all the common dependencies already
   installed. GitLab repositories can function as Docker-compatible
   container registries so it is quite simple to create a new repo with
   this feature enabled, shove a couple of Dockerfiles into it which
   install the necessary common dependencies onto base Perl images, and
   set up CI for that repo so that the base images are periodically
   rebuilt in order to account for possible updates. Result: simpler
   API-level CI configuration, faster runs, less bandwidth wasted;
 - process cpanfiles in ensembl-io and ensembl-variation. These are
   problematic for two reasons:
    * both depend on Bio::DB::BigFile, which requires messing
      with Kent libraries and linker flags in order to work properly,
    * ensembl-variation depends on Bio::DB::HTS, which pulls in BioPerl,
      which unless explicitly instrumented to install in minimal mode
      depends on known-to-be-broken GD module.
   At present we get away with not processing them because unlike
   ensembl-compara (which luckily does not demand any difficult
   modules), Core unit tests depending on -io and -variation code only
   import modules which do not use either of the above. This of course
   is a lucky co-incidence and cannot be relied upon to last.
   We have already got scripts for dealing with both Bio::DB::BigFile
   and GD, in Otter for instance, so we could in principle address this
   right away. Then again, in my opinion it would be better to delegate
   this to the aforementioned custom build images and keep
   repository-specific CI scripting as lean as possible;
 - setting passwords for MySQL users. Trivial to do but in order to
   avoid having those passwords in plain text in .gitlab-ci.yml, we
   should either set them as GitLab project variables or encrypt them
   somehow.
.gitlab-ci.inactive.yml Outdated Show resolved Hide resolved
@ens-bwalts
Copy link
Contributor

Reminder: this should be "squash-and-merge"d

@mkszuba mkszuba requested a review from s-mm September 3, 2019 08:44
@mkszuba mkszuba merged commit 93354a6 into master Sep 3, 2019
@mkszuba mkszuba deleted the feature/gitlab-ci_poc branch September 3, 2019 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants