New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retire jenkins.imagej.net #178

Open
ctrueden opened this Issue Oct 4, 2017 · 4 comments

Comments

Projects
None yet
4 participants
@ctrueden
Member

ctrueden commented Oct 4, 2017

Our faithful Jenkins server has served us well for several years. However, Jenkins is a complex piece of software with many security concerns and nuances. LOCI is making a concerted effort to reduce the number and complexity of services it runs, in favor of community- and cloud-based services whenever possible. E.g., rather than hosting the ImageJ source code in SCM on the LOCI server rack, all code is hosted here on GitHub. This is superior in pretty much every way, especially for community-oriented collaboration and development.

For continuous integration, the cloud-based services Travis CI (for Linux and macOS) and AppVeyor (for Windows) are freely available for OSS projects, and have become very popular and reliable. As such, the time of the ImageJ Jenkins draws to a close. This issue describes the work that needs to be done to fully migrate away from Jenkins in favor of Travis and Appyveyor for all ImageJ-related CI tasks.

  • Migrate pure-Java component builds to Travis. Most of our components are pure Java and build the same on every platform. So we can use Travis to build master whenever there is a push, and deploy the result to the ImageJ Maven repository. This replaces many individual Jenkins build jobs.
  • Use Travis to release components to ImageJ Maven repository. This partially replaces the private Release-Version Jenkins job.
  • Use Travis to release components to OSS Sonatype. This partially replaces the private Release-Version Jenkins job. Verify that cache invalidation still works, and/or is no longer necessary, such that released artifacts are immediately available from imagej.public group.
  • Use Travis to manage ImageJ1 repositories. There are several Jenkins jobs in a chain for detecting when ImageJ1 has changed and pushing those changes to ImageJA.
  • Multi-platform builds. We need to use both Travis and AppVeyor, and they need to detect when to auto-close (and when not to auto-close) the staging repository, such that all platform artifacts are successfully deployed.
  • GPG signing. We want to sign all artifacts when GPG credentials are available. Right now, Jenkins only signs things being deployed to OSS Sonatype. The username is scijava-ci with "real" name SciJava CI where applicable. Email can be scijava-ci@scijava.org or similar. The release-version.sh script is currently where GPG settings are validated, but release:perform on the cloud side is where it will actually be used. This must be reconciled.
  • Fiji distribution bundles. This includes Stable-Fiji-Java-8 and Stable-Fiji-MacOSX.
  • Convert remaining jobs to cloud CI. This includes:
    • SciJava Maven plugin – the catch with this one is that it currently deploys SNAPSHOT builds to OSS Sonatype snapshots repository – we need to decide whether to keep doing that (and if so, whether to generalize that to all OSS-Sonatype-based projects)
    • BoneJ2 (bonej-org/BoneJ2#65)
    • MorphoLibJ
    • BAR
    • H5J_Loader_Plugin
    • github-backup-osx
    • MPI-CBG jobs – my preference is to simply retire these jobs; when @fjug and co. need CI builds for Indago et. al they can switch to Travis at that time
    • Git mirror jobs – my preference, after much consideration, is to completely retire all of git.imagej.net – there are only a small handful of projects where I think the Git mirror is even useful, and really, we can resurrect them individually using GitHub + Travis if the need arises in the future
    • OpenSPIM jobs
    • SLIM-Curve jobs
    • Internal server jobs (some private jobs) – main things are: A) Drupal cron (which can be set up inside each Docker container using cron as normal); B) disk usage checks (which can be set up via cron); C) some public service validation such as the Validate-Javadoc (which needs a Travis analogue somehow)
  • Auto-update status.scijava.org using Travis. Replace obsolete Source-Code-Status job with (nightly?) Travis CI build of scijava/status.scijava.org.
@hadim

This comment has been minimized.

Show comment
Hide comment
@hadim

hadim Oct 13, 2017

I don't know if it fits here but would that make sense to have a Maven plugin that can push packages to IJ Update Sites?

If you point me to the right direction, I can try to have a look.

hadim commented Oct 13, 2017

I don't know if it fits here but would that make sense to have a Maven plugin that can push packages to IJ Update Sites?

If you point me to the right direction, I can try to have a look.

@stelfrich

This comment has been minimized.

Show comment
Hide comment
@stelfrich

stelfrich Jan 11, 2018

Member

Just noting one related point here, so we don't forget about it: can we use Travis CI to trigger the generation of the ComponentStats template for each plugin on http://imagej.net?

Member

stelfrich commented Jan 11, 2018

Just noting one related point here, so we don't forget about it: can we use Travis CI to trigger the generation of the ComponentStats template for each plugin on http://imagej.net?

@imagejan

This comment has been minimized.

Show comment
Hide comment
@imagejan

imagejan Jan 11, 2018

Member

@stelfrich wrote:

can we use Travis CI to trigger the generation of the ComponentStats template for each plugin on http://imagej.net?

According to fiji/fiji#121 (comment), it is mediawiki-maven-info that can achieve this.

Member

imagejan commented Jan 11, 2018

@stelfrich wrote:

can we use Travis CI to trigger the generation of the ComponentStats template for each plugin on http://imagej.net?

According to fiji/fiji#121 (comment), it is mediawiki-maven-info that can achieve this.

@ctrueden

This comment has been minimized.

Show comment
Hide comment
@ctrueden

ctrueden Jan 17, 2018

Member

Indeed, we can have a nightly Travis cron attached to mediawiki-maven-info that regenerates whenever something has changed. I finally hooked up a nightly to status.scijava.org and it is working OK; doing it for the sidebars too would be great. But I want to take care to only push changes when things have actually changed. Needs more work than I have time to do right now. In any case, it is not a blocker for retiring Jenkins, since Jenkins currently does not do this auto-sidebar-updating anyway.

Member

ctrueden commented Jan 17, 2018

Indeed, we can have a nightly Travis cron attached to mediawiki-maven-info that regenerates whenever something has changed. I finally hooked up a nightly to status.scijava.org and it is working OK; doing it for the sidebars too would be great. But I want to take care to only push changes when things have actually changed. Needs more work than I have time to do right now. In any case, it is not a blocker for retiring Jenkins, since Jenkins currently does not do this auto-sidebar-updating anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment