Support for `systemd` #266

basil · 2022-01-28T00:31:09Z

Downstream of jenkins-infra/docker-packaging#29. Beginnings of a common systemd.service(5) unit configuration for the deb, rpm, and suse targets.

Should fix most if not all of the following bugs:

Fixes #216

basil · 2022-01-29T00:48:48Z

First successful CI test run with systemd on all three targets: deb/rpm/suse.

We now have a distribution-agnostic systemd service unit (jenkins.service), as well as a distribution-agnostic launcher script (/usr/bin/jenkins) that accepts a well-defined set of environment variables (or no input at all, in which case it is the equivalent of java -jar jenkins.war). The systemd service unit defines sensible default options, and these options can be overridden with a systemd drop-in unit (which the tests do to customize -Xmx).

The main work that remains is the migration script to copy over old settings from /etc/{default,sysconfig} to the systemd drop-in unit, if the user has customized old defaults. I have the beginnings of such a script, but it needs more work. I haven't included it in the build yet or figured out how to invoke it, and it has a number of non-trivial TODOs remaining before I can even start testing.

basil · 2022-01-29T04:55:24Z

I implemented a migration routine for all three targets and did some light testing. It seems to work fine. I'll continue to do more extensive testing, but at this point I think this change is ready for review.

basil · 2022-01-29T05:35:09Z

Tested "Restart Jenkins when installation is complete and no jobs are running" with ExitLifecycle, worked as expected.

ekohl · 2022-02-03T11:16:56Z

@serianox you need to change the type to simple. notify requires 2.333. That's a much better workaround than an infinite timeout. "Fun" fact, a colleague just submitted systemd/systemd#22385 for a bug with TimeoutStartSec=infinity.

basil · 2022-02-03T21:24:15Z

@axmetishe To channel some of Kohsuke's energy:

I appreciate your code review, but let's focus on important things here.

Or to channel the energy of the late Roger Faulkner:

You think this is so easy? Why don't you try writing it? From scratch.

My main objective for this PR is not to cause a regression for the thousands of users with existing installations who will be running sudo {apt-get,dnf,zypper} upgrade and running these bits soon enough. Applying a configuration change of this magnitude to such a wide variety of legacy installations that are not even visible to me (let alone under my control) is a formidable challenge. The migration script alone is 452 lines long. This constraint, in my opinion, is the primary reason this issue has remained unresolved for the past five (5) years. What may work for one environment may not work in another. The Jenkins project enjoys a broad spectrum of adoption, but this comes with significant backward compatibility challenges. Compatibility matters, and this is a core value of the Jenkins project.

As @timja mentioned, this is a v1. There is room for improvement. People should feel free to propose new PRs with their ideas for improvement once this PR is merged. We now have CI build/test automation using Ansible, so this repository has become friendlier to first-time contributors. Yes, we can remove the old System V init scripts. Yes, we can optimize startup time with regard to networking. Yes, we can make the shell scripts squeaky clean or possibly eliminate them altogether. But we can't do any of this if we don't first ship this PR and migrate our existing install base to systemd(1).

Let's cross that bridge first and polish everything up on the other side. Join me in doing so.

wwuck · 2022-02-04T11:53:16Z

We are running multiple Jenkins LTS 2.319.2 instances on Devuan 4 and Debian 11 without SystemD. Both systems are running on sysvinit.

I installed the .deb file from build 36
During the install, the debconf dialog asked me if I would like to overwrite the configuration file at /etc/default/jenkins with the new copy from the .deb file.
I selected to keep the existing file with my customisations and this caused Jenkins to fail startup after the new package version was installed.

Configuration file '/etc/default/jenkins'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** jenkins (Y/I/N/O/D/Z) [default=N] ? n
Installing new version of config file /etc/init.d/jenkins ...
insserv: warning: script 'savecache' missing LSB tags
insserv: Default-Start undefined, assuming empty start runlevel(s) for script `savecache'
insserv: Default-Stop  undefined, assuming empty stop runlevel(s) for script `savecache'
Restarting Jenkins Automation Server: jenkinsCorrect java version found
Correct java version found
Setting up max open files limit to 8192
 failed!
invoke-rc.d: initscript jenkins, action "restart" failed.
dpkg: error processing package jenkins (--install):
 installed jenkins package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 jenkins

I then did a quick edit of /etc/default/jenkins to fix the updated line for JENKINS_WAR=.

# location of the jenkins war file
-JENKINS_WAR=/usr/share/$NAME/$NAME.war
+JENKINS_WAR=/usr/share/java/$NAME.war

After this edit, Jenkins started successfully with service jenkins start.

Yes, we can remove the old System V init scripts.

I would like to please request that this is not done in the future as we are surely not the only people running Jenkins on Debian without SystemD.

MarkEWaite · 2022-02-04T13:13:19Z

I installed the .deb file from build 36 During the install, the debconf dialog asked me if I would like to overwrite the configuration file at /etc/default/jenkins with the new copy from the .deb file. I selected to keep the existing file with my customisations and this caused Jenkins to fail startup after the new package version was installed.

Configuration file '/etc/default/jenkins'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** jenkins (Y/I/N/O/D/Z) [default=N] ? n
Installing new version of config file /etc/init.d/jenkins ...
insserv: warning: script 'savecache' missing LSB tags
insserv: Default-Start undefined, assuming empty start runlevel(s) for script `savecache'
insserv: Default-Stop  undefined, assuming empty stop runlevel(s) for script `savecache'
Restarting Jenkins Automation Server: jenkinsCorrect java version found
Correct java version found
Setting up max open files limit to 8192
 failed!
invoke-rc.d: initscript jenkins, action "restart" failed.
dpkg: error processing package jenkins (--install):
 installed jenkins package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 jenkins

I then did a quick edit of /etc/default/jenkins to fix the updated line for JENKINS_WAR=.

# location of the jenkins war file
-JENKINS_WAR=/usr/share/$NAME/$NAME.war
+JENKINS_WAR=/usr/share/java/$NAME.war

After this edit, Jenkins started successfully with service jenkins start.

Thanks for testing. I performed a similar test on my Debian 11 machine that was configured behind a reverse proxy. The Debian standard handling of the modified file /etc/default/jenkins worked the same for me as it did for you. I applied my changes to the configuration file /etc/default/jenkins and Jenkins restarted. In my case, there were more options that I needed to add (--prefix), but it was the same workflow.

I assume that modifying the /etc/default/xxx file is a common way to do things on Debian when using System V init. I assume that the override.conf file from systemd configuration avoids that type of problem by not being modified by the installer. Is that correct or is there a risk that future changes may require a similar conflict resolution process?

ekohl · 2022-02-04T13:24:34Z

Yes, we can remove the old System V init scripts.

I would like to please request that this is not done in the future as we are surely not the only people running Jenkins on Debian without SystemD.

My suggestion was limited to RPM or even just Red Hat where there is no sysvinit anymore. You are correct that Debian does give the choice.

basil · 2022-02-04T16:55:25Z

I assume that modifying the /etc/default/xxx file is a common way to do things on Debian when using System V init. I assume that the override.conf file from systemd configuration avoids that type of problem by not being modified by the installer. Is that correct or is there a risk that future changes may require a similar conflict resolution process?

@MarkEWaite Correct, there are no hard conflicts requiring mechanical merge resolution in systemd(1) drop-ins because these are separate files from the main service unit configuration. Of course, there may be soft conflicts (e.g., a user may have overridden an environment variable, the environment variable may get renamed in the main service unit configuration, and the user's customized override might remain under the old name and thus become ineffective).

The `--javahome` argument that was listed there is rejected by Jenkins when it is included in the JENKINS_ARGS variable. The `--javaHome` argument is accepted by Jenkins but does not seem to set the JAVA_HOME environment variable in Jenkins 2.330 on Debian 11. The JAVA_HOME environment variable is not shown in Jenkins 2.330 in the "Manage Jenkins" -> "System Information" page of Jenkins 2.330 and is not shown in the environment variable list reported by a job run on the Jenkins 2.330 controller. The `--javaHome` argument is accepted by Jenkins and sets the JAVA_HOME environment variable in Jenkins 2.333 when using pull request jenkinsci#266 (systemd support) and is shown in the environment variable list reported by a job run on the Jenkins 2.333 controller installed with the systemd support provided in pull request jenkinsci#266.

MarkEWaite · 2022-02-05T17:55:20Z

I've run a series of successful upgrade tests experimenting with:

Jenkins 2.330 on Debian 11 with prefix, custom Java installation, and custom Java garbage collection arguments upgraded to Jenkins 2.233 packaged in build 36 of this pull request. The JAVA_HOME environment variable was not visible in the Jenkins "Manage Jenkins" -> "System information" of Jenkins 2.330 when set with --javaHome in the JENKINS_ARGS of /etc/default/jenkins , but becomes visible after upgrade to Jenkins 2.333 packaged in build 36 when placed in the JENKINS_OPTS environment variable

There was an unexpected entry in the output of journalctl -u jenkins.service. The output was:

Feb 05 10:49:49 debian11-a systemd[1]: Started My Jenkins Continuous Integration Server.
Feb 05 10:49:49 debian11-a jenkins[4184952]: 2022-02-05 17:49:49.618+0000 [id=23]        WARNING        h.lifecycle.SystemdLifecycle#notify
Feb 05 10:49:49 debian11-a jenkins[4184952]: com.sun.jna.LastErrorException: [1] Operation not permitted
Feb 05 10:49:49 debian11-a jenkins[4184952]:         at com.sun.jna.Native.invokeInt(Native Method)
Feb 05 10:49:49 debian11-a jenkins[4184952]:         at com.sun.jna.Function.invoke(Function.java:426)
Feb 05 10:49:49 debian11-a jenkins[4184952]:         at com.sun.jna.Function.invoke(Function.java:361)
Feb 05 10:49:49 debian11-a jenkins[4184952]:         at com.sun.jna.Library$Handler.invoke(Library.java:265)
Feb 05 10:49:49 debian11-a jenkins[4184952]:         at hudson.lifecycle.$Proxy40.sd_notify(Unknown Source)
Feb 05 10:49:49 debian11-a jenkins[4184952]:         at hudson.lifecycle.SystemdLifecycle.notify(SystemdLifecycle.java:57)
Feb 05 10:49:49 debian11-a jenkins[4184952]:         at hudson.lifecycle.SystemdLifecycle.onReady(SystemdLifecycle.java:34)
Feb 05 10:49:49 debian11-a jenkins[4184952]:         at hudson.WebAppMain$3.run(WebAppMain.java:258)

I've also reviewed the documentation pull request and commented there.

MarkEWaite · 2022-02-05T18:45:11Z

I ran an upgrade test on Debian 10 that started with Jenkins 2.331 customized to use /opt/jdk-11 as JAVA_HOME, port 9090, custom garbage collection arguments, and /jenkins as the prefix.

In order to use a custom Java in 2.331, I set JAVA_HOME in /etc/default/jenkins and also had to set PATH=$JAVA_HOME/bin:$PATH in /etc/default/jenkins. Those changes to JAVA_HOME and PATH were not preserved during the upgrade process, while all the other settings were preserved (including the JENKINS_OPTS setting of --javaHome). I assume the JAVA_HOME technique that I used in the 2.331 configuration is non-standard and likely only works as a happy accident.

When I followed the instructions in the systemd configuration file to set JAVA_HOME, it behaved exactly as I expected. I'm running a Jenkins 2.333 on Debian 10 with custom Java installation, custom garbage collection settings, a non-standard port, and a URL prefix.

MarkEWaite · 2022-02-05T21:02:07Z

I ran an upgrade test on my Red Hat Enterprise Linux 8.5 installation with a custom prefix, custom Java installation, and environment variable settings for configuration as code. The systemd override file felt much easier to modify as needed than the /etc/sysconfig/jenkins file.

I'd like a little more time to use the list of fixed bugs as ideas for more exploration. I think that my exploratory tests could be completed as early as end of day Monday.

It would be nice to have a blog post that describes the changes and the ways that systemd makes things easier and simpler than the System V init process.

MarkEWaite · 2022-02-05T22:49:28Z

systemd/jenkins.sh

timja · 2022-02-08T07:39:47Z

Marking as on hold till after Wednesday security release.

basil · 2022-02-08T16:32:29Z

Marking as on hold till after Wednesday security release.

Not just the Wednesday security release, but this PR will have to wait until after this repository has branched for the stable March LTS, as the API changes in core for Type=notify are unlikely to be backported.

timja · 2022-02-08T17:02:17Z

Marking as on hold till after Wednesday security release.

Not just the Wednesday security release, but this PR will have to wait until after this repository has branched for the stable March LTS, as the API changes in core for Type=notify are unlikely to be backported.

we can cut that whenever, and just exclude this, I wouldn't hold off for that 😄

timja · 2022-02-13T09:36:17Z

Branch was created by the looks of it: https://github.com/jenkinsci/packaging/commits/stable-2.332

This PR is now ready for merge, after ~24 hours, we will merge it if there's no negative feedback.

Thanks!

serianox · 2022-02-13T21:03:21Z

@serianox you need to change the type to simple. notify requires 2.333. That's a much better workaround than an infinite timeout. "Fun" fact, a colleague just submitted systemd/systemd#22385 for a bug with TimeoutStartSec=infinity.

@ekohl Thanks. For the record, I think the correct type is exec; it seems to work, whereas simple indefinitely hang. (anwsering here in case someone is in the same predicament as I)

timja · 2022-02-14T18:29:27Z

Last couple of runs failed CI FYI @basil

Looks like it is centos-stream-9

Error:

Problem: package python3-devel-3.9.10-1.el9.x86_64 requires python3 = 3.9.10-1.el9,

but none of the providers can be installed

cannot install both python3-3.9.10-1.el9.x86_64 and python3-3.9.10-2.el9.x86_64

cannot install the best candidate for the job

(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)\

timja · 2022-02-14T22:51:12Z

There's a CI fix in #283 which is unrelated to this PR so will ship anyway

Jenkins versions >=2.332.1 ship the jenkins.war now as /usr/share/java/jenkins.war, and no longer as /usr/share/jenkins/jenkins.war. Change from upstream took place in jenkinsci/packaging#266 FTR, current upstream version of the jenkins systemd service unit: $ grep -v '^#' /lib/systemd/system/jenkins.service | sed '/^$/d' [Unit] Description=Jenkins Continuous Integration Server Requires=network.target After=network.target [Service] Type=notify NotifyAccess=main ExecStart=/usr/bin/jenkins Restart=on-failure SuccessExitStatus=143 User=jenkins Group=jenkins Environment="JENKINS_HOME=/var/lib/jenkins" WorkingDirectory=/var/lib/jenkins Environment="JENKINS_WEBROOT=%C/jenkins/war" Environment="JAVA_OPTS=-Djava.awt.headless=true" Environment="JENKINS_PORT=8080" [Install] WantedBy=multi-user.target Change-Id: I8349fdfc5028f5bcc9903a79f26e5a685643a618

Support for systemd

d0a0832

basil added the work-in-progress label Jan 28, 2022

basil added 9 commits January 27, 2022 23:34

Beginnings of migration script

28f645c

fixup

e48c601

Add a wrapper script

483823a

Check JENKINS_HOME

92ee186

Use @@ARTIFACTNAME@@

370b719

Fix quoting

7143902

More work on migration script

7b6dade

Fix tests

e63c6f5

Loosen wrapper script

19d8cc5

Use LoadCredential/SetCredential for keystore

a0af39e

basil mentioned this pull request Jan 29, 2022

Systemd support #216

Closed

basil added 5 commits January 28, 2022 17:32

Use systemd specifiers for paths

4e26b52

Squash some TODOs

592c691

Migration

39f9a5d

Amazon Linux doesn't support %S

fccdf6c

fixup

d6f9025

basil marked this pull request as ready for review January 29, 2022 04:55

basil requested a review from a team as a code owner January 29, 2022 04:55

basil added enhancement and removed work-in-progress labels Jan 29, 2022

fixups

cb83332

This was referenced Jan 29, 2022

Convert RPMs from sysvinit to systemd + other cleanups #224

Closed

Add jenkins.service for systemd which is in use in debian #93

Closed

basil mentioned this pull request Jan 29, 2022

Documentation updates for systemd transition jenkins-infra/jenkins.io#4875

Merged

fixups

2a2bfcb

timja requested a review from MarkEWaite February 4, 2022 17:09

MarkEWaite mentioned this pull request Feb 5, 2022

Fix javaHome arg in Debian default file comment #281

Merged

6 tasks

MarkEWaite reviewed Feb 6, 2022

View reviewed changes

systemd/jenkins.sh Show resolved Hide resolved

timja added the on-hold label Feb 8, 2022

timja removed the on-hold label Feb 13, 2022

timja mentioned this pull request Feb 14, 2022

Workaround broken centos stream 9 repo setup #283

Merged

timja merged commit cfa89d8 into jenkinsci:master Feb 14, 2022

basil mentioned this pull request Feb 16, 2022

Debian packaging puts the PID file in an odd place #254

Closed

MarkEWaite mentioned this pull request Mar 5, 2022

Use systemd and other updates on stable-2.332 branch #291

Merged

6 tasks

dbnicholson mentioned this pull request Apr 5, 2022

Default jenkins.war path wrong since 2.332.1 jenkinsci/plugin-installation-manager-tool#427

Closed

itsAftabAlam mentioned this pull request Dec 9, 2023

Updated jenkins.war default path for linux jenkinsci/plugin-installation-manager-tool#622

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for `systemd` #266

Support for `systemd` #266

basil commented Jan 28, 2022 •

edited

Loading

basil commented Jan 29, 2022

basil commented Jan 29, 2022

basil commented Jan 29, 2022

ekohl commented Feb 3, 2022

basil commented Feb 3, 2022

wwuck commented Feb 4, 2022

MarkEWaite commented Feb 4, 2022

ekohl commented Feb 4, 2022

basil commented Feb 4, 2022 •

edited

Loading

MarkEWaite commented Feb 5, 2022 •

edited

Loading

MarkEWaite commented Feb 5, 2022 •

edited

Loading

MarkEWaite commented Feb 5, 2022

MarkEWaite commented Feb 5, 2022 •

edited

Loading

timja commented Feb 8, 2022

basil commented Feb 8, 2022

timja commented Feb 8, 2022

timja commented Feb 13, 2022

serianox commented Feb 13, 2022

timja commented Feb 14, 2022 •

edited

Loading

timja commented Feb 14, 2022

Support for systemd #266

Support for systemd #266

Conversation

basil commented Jan 28, 2022 • edited Loading

basil commented Jan 29, 2022

basil commented Jan 29, 2022

basil commented Jan 29, 2022

ekohl commented Feb 3, 2022

basil commented Feb 3, 2022

wwuck commented Feb 4, 2022

MarkEWaite commented Feb 4, 2022

ekohl commented Feb 4, 2022

basil commented Feb 4, 2022 • edited Loading

MarkEWaite commented Feb 5, 2022 • edited Loading

MarkEWaite commented Feb 5, 2022 • edited Loading

MarkEWaite commented Feb 5, 2022

MarkEWaite commented Feb 5, 2022 • edited Loading

timja commented Feb 8, 2022

basil commented Feb 8, 2022

timja commented Feb 8, 2022

timja commented Feb 13, 2022

serianox commented Feb 13, 2022

timja commented Feb 14, 2022 • edited Loading

timja commented Feb 14, 2022

Support for `systemd` #266

Support for `systemd` #266

basil commented Jan 28, 2022 •

edited

Loading

basil commented Feb 4, 2022 •

edited

Loading

MarkEWaite commented Feb 5, 2022 •

edited

Loading

MarkEWaite commented Feb 5, 2022 •

edited

Loading

MarkEWaite commented Feb 5, 2022 •

edited

Loading

timja commented Feb 14, 2022 •

edited

Loading