PATH environment variable in jenkins user cannot see java

[jvz]

If I run this container as root, $JAVA_HOME/bin is added to the PATH. However, when I ssh in as the jenkins user, the PATH is not propagated, and therefore the ssh-slaves plugin can't find java in the PATH.

My current workaround is to manually set the javaPath configuration value to the expected openjdk path, though this only fixes the inability to launch an agent.

[LCerebo]

Same problem here. The tag jdk11 is not affected but I need to use java 8.

[GunArm]

Same issue for me today. Seeing the above I switched to jdk11 and it works. Since I only need java to run the jenkins agent, I'd rather use the jdk8 version which is less than half the size.

[GunArm]

Since the jdk11 tag was updated a couple days ago, jdk11 now has the same issue

[oleg-nenashev]

So it is a regression in recent versions. Would be great to have it fixed indeed

[GunArm]

Someone should check that my previous comment is correct, since I'm new to jenkins docker plugin, and most docker concepts generally. But I had an empty test project build working on jdk11 tag on dec 23, and after the holidays (same configuration/empty test project, unchanged except jenkins auto updated my image), now cannot find java. No problems on tag "latest-jdk11" which is a year old

[jvz]

Seems to be a change in where the JDK is installed. My guess is that a switch from Oracle or OpenJDK to AdoptOpenJDK or similar which had a slightly different installation procedure.

As I mentioned in the ticket, you can work around the issue by manually adding the agent's JAVA_HOME as an agent parameter.

[Virtlink]

What I think is happening here is that this line is incorrect:

env | grep _ >> /etc/environment

This will take from the current environment all lines with an underscore in it, anywhere, and append them to /etc/environment. So while this will copy your JAVA_HOME and MAVEN_OPTS, as they contain underscores in their names, it will only copy your PATH if there is some path in it with an underscore. That seems extremely arbitrary to me.

For my own uses I changed that line to the following:

env | egrep -v "^(HOME=|USER=|MAIL=|LC_ALL=|LS_COLORS=|LANG=|HOSTNAME=|PWD=|TERM=|SHLVL=|LANGUAGE=|_=)" >> /etc/environment

This takes all environment variables, except those listed, and appends then to /etc/environment, overriding any previously defined there.

[GunArm]

@Virtlink I also did a doubletake on that line but basically wrote it off as being beyond my comprehension. I am really curious about the intention of that. Still, as random as filtering the variables by underscore seems, it would catch JAVA_HOME so I figured it wasn't the problem.

You don't explicitly mention the effect of your change. Does it allow java to execute without manually specifying the path in the agent config? Does it also allow ENV declarations in the dockerfile to be available to builds (#46)?

I'm a docker newbie, but if anyone can explain what it is about this project that makes such a line like env | whatever > /etc/environment necessary, I would be very interested.

[Virtlink]

@GunArm Let's say your call to env prints this:

SHELL=/bin/bash
TERM=xterm
SSH_CLIENT=10.10.4.77 60394 22
LC_ALL=en_US.UTF-8
USER=root
JAVA_HOME=/usr/lib/jvm/openjdk/
PATH=/usr/lib/jvm/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAIL=/var/mail/root
HOME=/root
LANG=en_US.UTF-8
_=/usr/bin/env

Doing env | grep _ will return each line that has an underscore in it. This approach might be inspired by this Stackoverflow answer, where the author states it is meant to capture application environment variables such as PG_DBNAME or JAVA_HOME, which often contain an underscore. But in fact, it will capture all lines that contain an underscore, so in this example it will return:

SSH_CLIENT=10.10.4.77 60394 22
LC_ALL=en_US.UTF-8
JAVA_HOME=/usr/lib/jvm/openjdk/
LANG=en_US.UTF-8
_=/usr/bin/env

Note how, indeed, JAVA_HOME is one of the selected lines. Also note that LANG is also selected because the value happens to contain an underscore. It's kind of arbitrary. Finally, note how PATH is not selected, even though it's value contains the path to the JDK (/usr/lib/jvm/openjdk/bin) and it's required to have this set correctly for javac to be found on the path, which I guess is the problem the OP had.

The command env | grep _ >> /etc/environment appends the selected lines to the /etc/environment file, which is a system file with the initial values of environment variables for each user. This effectively sets JAVA_HOME and other selected environment variables for each user, overriding any existing values that where previously set in the /etc/environment file, if any.

---

Arguably an improvement would be to just append everything from env to /etc/environment:

env >> /etc/environment

This would include lines such as PATH=, allowing javac to be found. However, this would also include lines such as USER=root and HOME=/root, which definitely do not apply to all users.
Therefore, I want to exclude such lines.

---

So my command does essentially the same as the original command, but selects different lines.

egrep -v "^(HOME=|USER=|MAIL=|LC_ALL=|LS_COLORS=|LANG=|HOSTNAME=|PWD=|TERM=|SHLVL=|LANGUAGE=|_=)"

It selects, using egrep, any lines that does not match (-v) the pattern. The pattern matches the start of the line with ^, followed by one of the alternatives in the list. So it will exclude a line that starts with HOME=, or USER=, etc. Applied on the above example, it will reduce the list of environment variables to:

SSH_CLIENT=10.10.4.77 60394 22
JAVA_HOME=/usr/lib/jvm/openjdk/
PATH=/usr/lib/jvm/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

This looks like a sensible list. I think SSH_CLIENT doesn't have to be in there, but it probably doesn't hurt either. Now, just like before, we append these environment variables to /etc/environment, which means they'll be the default for any other user.

[GunArm]

Great answer. Given the existence of JAVA_HOME I (unthinkingly) assumed it somehow took the place of PATH but of course it couldn't.

Blacklisting known unwanted system variables (especially those known from considering the parent image) is a much better solution.

[getJv]

Hello all,

In fact the solution proposed by @Virtlink Works well to me too.

For my own uses I changed that line to the following:

env | egrep -v "^(HOME=|USER=|MAIL=|LC_ALL=|LS_COLORS=|LANG=|HOSTNAME=|PWD=|TERM=|SHLVL=|LANGUAGE=|_=)" >> /etc/environment

I had the same issue with version 8 and 11. What is the impeditive to fix the docker image with this? Just a PR or Is this solution not recommended?

[oleg-nenashev]

Just a PR, I'd guess

[GunArm]

FYI, my (similar) experiences noted above were due to a larger issue of losing all env variables as described in issue #46, which turned out to be because of issue #62, and resolved in PR #63.

[gclayburg]

I ran into this issue as well when running the agent from jenkins/ssh-agent:latest

After launching the image, I fixed it by just updating the .bashrc file for the jenkins user like this:

ssh -i ssh/jenkins_agent_key -p 1122 jenkins@localhost "cp .bashrc .bashrc.orig && echo 'PATH=/usr/local/openjdk-8/bin:\$PATH' > newbashrc && cat .bashrc >> newbashrc && cp newbashrc .bashrc"

Maybe the dockerfile just needs to update the .bashrc file with the PATH for java?

BTW, I also tried the alpine image and ran into the same issue

[GunArm]

@gclayburg our of curiosity, are you using jenkins' docker-plugin to launch ephemeral build agents in jenkins builds?

[gclayburg]

At this point, I was just following this jenkins guide. It just seemed to me the instructions were more complex than they needed to be for a simple issue like this. To me, the agent should just run without workarounds.

I actually just forked this project and created a working docker image. Should I create a PR?

[GunArm]

Ok yeah, FYI that guide is showing use of docker-plugin.

So, your solution might be a functional workaround. But there is a mechanism for this already: I believe PATH is set for the default user from the jdk base image. Getting it from the default user to the jenkins user is accounted for when the setup-sshd script "copys" the primary environment variables (all of them, not just PATH) to /etc/environment. This is not working because jenkins docker-plugin has a default of sending an sshd command to the image, and the way the script works, when it gets a command passed to it, it bypasses the part that makes the environment variables available to the jenkins user.

My PR #63 resolves this issue. But it has yet to be merged.

Additionally, the tweak described above, changing
env | grep _ >> /etc/environment to
egrep -v "^(HOME=|USER=|MAIL=|LC_ALL=|LS_COLORS=|LANG=|HOSTNAME=|PWD=|TERM=|SHLVL=|LANGUAGE=|_=)"
Is a good one and might also help with this, but I did not include it in my pull request (because I didn't want to go outside the scope of the PR and make it more complicated to review). Unfortunately my PR branch moves that line in the script so that it does not get bypassed when a command is sent. So a separate PR that just changes that line is going to cause a merge conflict.

I'd happily add it to my PR (if it didn't further delay a release).

[gclayburg]

yeah, that sounds reasonable, @GunArm
I tried to use the workaround discussed in the guide - it sounds like it was based off of comments on this issue - but I couldn't get it to work for me. Maybe I did it wrong, I don't know. What I do know is I was able to fix it by just slightly modifying the Dockerfile. In the end, it was just this one line change to Dockerfile:

However, I just looked at some of your work and it sounds like yours might be better in that should fix the issue for more than one docker image. I only changed the one for jdk 8 / debian buster.

[GunArm]

Right also, this is slightly complex because there are really two compounding issues causing the java path problem.

1. env | grep _ >> /etc/environment does not duplicate PATH (although it duplicates JAVA_HOME)
2. For those of us using docker-plugin for ephemeral environments (rather than hosting one of these containers as a long running agent more like a normal bare-metal jenkins agent host), my changes are needed to deal with the quirk of docker-plugin sending the sshd command by default

Also 2 is more broad of an issue than just paths, it affects all environment variables you might set in the dockerfile, like for example DOTNET_CLI_TELEMETRY_OPTOUT or anything else. And it breaks windows usage entirely.

By all means you should use your fix until this gets sorted out, but note that it papers over the underlying problem, which has effects beyond the java path.

[GunArm]

@gclayburg actually on looking at your linked docker reference more closely, my apologies, I'm not sure if it is infact using docker-plugin or not. 2 in my list above might not apply to you, and the fix
env | egrep -v "^(HOME=|USER=|MAIL=|LC_ALL=|LS_COLORS=|LANG=|HOSTNAME=|PWD=|TERM=|SHLVL=|LANGUAGE=|_=)" >> /etc/environment
might work for you equally well as the .bashrc approach

[Aschay]

I have the same issue @Virtlink workaround fixed it for me

[andrey3diq]

@Virtlink workaround works fine.
For those who came here from jenkins tutorial https://www.jenkins.io/doc/book/using/using-agents/.
The workaround doesn't work on alpine. Instead of jenkins/ssh-agent:alpine you should use jenkins/ssh-agent:jdk11

[Elwin00]

I use docker-compose and I wanted to have it fixed without the need to manually enter a command every time something gets restarted. Here is what I did.

I created a simple ./ssh-agent-fix/Dockerfile with this content:

FROM jenkins/ssh-agent:jdk11
RUN echo PATH=$PATH >> /etc/environment

Then I modified the docker-compose.yml like this:

 jenkins-agent-docker:
   #image: jenkins/ssh-agent:jdk11
   build: ./ssh-agent-fix

Last step is to build the new image

docker-compose build && docker-compose up -d

When Jenkins web refreshed, the agent was already online! :-)