No way to override bundled plugin when using Docker

[chriswhelix]

Issue Type

Bug Report

Summary

The latest GoCD server version (17.11.0) does not come bundled with the latest YAML config plugin version (0.6.0). When using the official GoCD Docker image to run the server, there is no way to override the bundled plugin.

Environment

Using a Dockerfile that looks like:

FROM gocd/gocd-server:v17.11.0

ENV GOCD_YAML_CONFIG_VERSION=0.6.0

RUN mkdir -p /godata/plugins/external && cd /godata/plugins/external \
	&& wget "https://github.com/tomzo/gocd-yaml-config-plugin/releases/download/${GOCD_YAML_CONFIG_VERSION}/yaml-config-plugin-${GOCD_YAML_CONFIG_VERSION}.jar"

Desired Results

GoCD uses plugin version 0.6.0.

Actual Results

GoCD uses bundled plugin version 0.5.0. Note, this is as per documentation; the behavior is expected, but deeply frustrating.

On a persistent server, I could remove the bundled plugin, or install the new version over it, then restart GoCD. But because we're using Docker to manage plugin installation, all plugin installation activities occur before the first run of the server. So whatever I might do in the Dockerfile, as soon as the server boots up, it writes out its bundled plugins, overriding my desired versions.

Possible Fix

My immediate problem would be solved by bundling the latest version of the plugin (CC/ @tomzo).

However, it also seems to me there should be some way, as a user of GoCD via Docker, to disable or replace bundled plugins. I would view it as a defect that, in the context of Docker, certain plugins have been made mandatory and can not be upgraded or disabled (unless I'm missing something).

[bdpiprava]

@chriswhelix --- You are right about the bundled plugin behaviour. On server boot up it first loads the bundled plugins and then the external plugin. Only way I can see is,

1. Start gocd/gocd-server:v17.11.0 with volume mount.

docker run --name go-server -d -p8153:8153 -p8154:8154 -v <PATH_TO_DATA_DIR>:/godata gocd/gocd-server:v17.11.0

2. Replace bundled plugin under <PATH_TO_DATA_DIR>/plugins/bundled

wget -O gocd-yaml-config-plugin.jar  https://github.com/tomzo/gocd-yaml-config-plugin/releases/download/0.6.0/yaml-config-plugin-0.6.0.jar

3. Restart go server

docker restart go-server

[tomzo]

I recall @arvindsv talking on gitter that this is legacy behavior that should be removed. It used to be that bundled plugins had move quality testing and hence their priority when loading.
I think that currently we should simply switch to preferring the latest version, regardless if it is in /bundled or /external folder.

[ketan]

I think that currently we should simply switch to preferring the latest version, regardless if it is in /bundled or /external folder.

I agree with this argument. I'll defer to @arvindsv for the final vote to confirm this behavior.

@bdpiparva just pointed me to this method that seems to be in the way, and it the current behavior (that bundled overrides external) seems by design. https://github.com/bdpiparva/gocd/blob/fcbfd6443f6e9edc24df7fb49ad651a78bc9dac3/plugin-infra/go-plugin-infra/src/com/thoughtworks/go/plugin/infra/listeners/DefaultPluginJarChangeListener.java#L120-L124. According to @bdpiparva, looking at the old "cruise" code does not provide any historical context either.

[arvindsv]

Yes, this needs to change. Currently, it's too dependent on the filesystem and the order the directories are considered. How about we use these principles?

1. (ID + version) becomes important.
2. Distinction between bundled and external becomes unimportant.
3. Upon startup all plugins, across bundled and external are considered and loaded. Only the latest version is loaded if ID is duplicated. Irrespective of where it is.
4. API to load plugins and override running plugins (to upgrade version, for example).
5. Hacky workaround to load plugins automatically during development goes away.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 90 days.
If you can still reproduce this error on the master branch using local development environment or on the latest GoCD Release, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.

[Stono]

This is a pain :(
Does anyone have a workaround that doesn't involve starting the server?
I want the plugins in my build process, not a run-time concern.

[chadlwilson]

Just curious @Stono - which bundled plugin are you looking to override?

[Stono]

The yum repo one. The version of repoquery we use now after upgrading the OS returns multiple results unless you specify --latest-limit=1. This breaks the plugin (with the way we use it at least) because we have multiple matches like filename-v1 filename-v2 etc.

I'll likely raise a PR to "fix" it and backwards compatible support older and newer repoquery at some point, but the next problem we hd is that we're currently still on 20.4.0, so I needed to compile the latest version of the source, with the additional --latest-limit=1, but targeting 20.4.0.

In the spirit of "cattle rather than pets" - I strive to have as much as possible in the Dockerfile/build process, so it is immutable as it progresses through environments, leaving only dynamic data to be what is truly relevant to the environment (for example; pipelines, artifacts etc). Because of this, the idea of "jumping into the container, overwriting the plugin, and restarting the server" doesn't work for me.

The "dirty" way i've got working is this in our Dockerfile:

# Download other static external plugins, and addons from Jfrog
RUN sh /download-plugins.sh

# Copy the build external plugins across
COPY --from=plugin-ldap /usr/local/autotrader/app/gocd-ldap-authorization-plugin.jar /godata-symlinks/plugins/external

# Start GoCD which will extract all bundled plugins
RUN sh /test-run.sh

# Overwrite the bundled yum repo plugin
COPY --from=plugin-yum /usr/local/autotrader/app/gocd-yum-repo-plugin.jar /godata-symlinks/plugins/bundled/gocd-yum-repository-poller-plugin.jar

In this example; you'll note a few things.

• download-plugins.sh, will fetch our external and addons and put them into /godata-symlinks/addons and /godata-symlinks/plugins/external
• we copy some external plugins we've compiled in previous layers
• we then "start" gocd-server, which generates the bundled plugins, the script is pretty noddy:

#!/bin/bash
set -e
/docker-entrypoint.sh &
PID=$!

MAX_WAIT_TIME=60
shutdown=0
until $(curl --output /dev/null --silent --head --fail http://127.0.0.1:8153); do
  printf '.'
  if [ $shutdown -ge "$MAX_WAIT_TIME" ]; then
    echo "GoCD did not start in time!"
    exit 1
  fi
  sleep 1
  shutdown=$((shutdown+1))
done

echo "GoCD started, stopping it..."
set +e
kill $PID
wait $PID

exit 0

• And then finally we overwrite the bundled yum plugin.

In our entrypoint script, which will run on the cluster, we symlink the /godata folders that are on the PersistentVolume to the godata-symlinks folder in our container:

# Symlink our godata plugins
shout "Configuring GoCD plugins and addons dirs"
rm -rf /godata/addons
ln -sf /godata-symlinks/addons /godata/addons

rm -rf /godata/plugins
ln -sf /godata-symlinks/plugins /godata/plugins

And this gives us what we want, plugins that are entirely populated as part of the build process, statically promoted through environments.

It would be awesome to not have to do all of this, because I'm sure you can agree, it's dirty!

[chadlwilson]

Thanks for sharing @Stono - yes that is indeed troublesome.

The yum repo one. The version of repoquery we use now after upgrading the OS returns multiple results unless you specify --latest-limit=1. This breaks the plugin (with the way we use it at least) because we have multiple matches like filename-v1 filename-v2 etc.

It doesn't get at the root of your problem, but it does occur to me that the yum repo poller is a little "different" to the other bundled plugins. While not having used it before, I wonder why it is bundled and perhaps whether it shouldn't be - given its coupling to behaviour of other tools (such as repoquery) with a different lifecycle to GoCD itself. Possibly could argue the same for the LDAP auth plugin. Hmm.

I'll likely raise a PR to "fix" it and backwards compatible support older and newer repoquery at some point, but the next problem we hd is that we're currently still on 20.4.0, so I needed to compile the latest version of the source, with the additional --latest-limit=1, but targeting 20.4.0.

If you are just talking about this line you possibly don't need to change this. It's a compileOnly dependency and I don't believe should affect ability of the built plugin to work with GoCD 20.4.0, especially if the plugin compiles and tests OK without other changes.

[Stono]

While not having used it before, I wonder why it is bundled and perhaps whether it shouldn't be - given its coupling to behaviour of other tools

I mean really, I don't think the concept of bundled plugins needs to exist at all. Let users add the plugins that are important to them. Appreciate this might be somewhat of a big change so perhaps it could be configurable, -Dgocd.bundled=false etc.

For example in our case we use the ldap-authorization plugin, which means the bundled ldap-authentication plugin is useless. No need to load it. I'd rather it not be a configurable option for our users.

If you are just talking about this line you possibly don't need to change this

I had to change that line to get it to compile against the version of java that our GoCD server runs (11 LTS). The newer target required Java 13, and we only use LTS versions in prod.

[chadlwilson]

Bundling the JSON and YAML config plugins made it a lot easier for people to adopt pipelines as code and get important features running out-of-the-box, including some options to improve security posture and allow source controlling secrets such as file-based secrets. Personally as a user I appreciated that for the config plugins. Less so yum or ldap authn. :) Also keep in mind ldap authz was commercial for a long time making its usage more fringe.

But I see where you are coming from.

I had to change that line to get it to compile against the version of java that our GoCD server runs (11 LTS). The newer target required Java 13, and we only use LTS versions in prod.

Ahh, yes. Personally I think it would have been better to only target LTS versions, in retrospect. Hope to have things working against Java 17 soon, but we can possibly drop the release target back to 11. I contemplated doing it for 21.4, but didn't get a chance to discuss with others.

[Stono]

Yeah, it's a bit of a problem for us upgrading GoCD at the moment! We produce hardened/signed off "base" images for core languages. They integrate quite tightly into our platform too - so the more versions we support the more overhead on the core platform team. As a result, there's little desire for us to support non-LTS versions.

But like anything, it's a balancing act. Not being able to upgrade GoCD means we have issues like the one above, and CVEs like this.

It'd be great if you support 17 in the next release, or failing that, 11.

[chadlwilson]

Hi @Stono - just to follow up on this, while it doesn't resolve this bundled plugin issue, nor your dnf issue on the yum-repository-poller-plugin...

• GoCD 22.1.0 is out with Java 17 support
• Source/target compatibility was also dropped back to Java 11 LTS (from 13)
• and if you build your custom plugin version off the plugin's master it should work fine against Java 11 now since the plugin-api version has been dropped back to Java 11 target consistent with the server itself.

[chadlwilson]

It doesn't solve the root issue here, but the yum repo plugin is no longer bundled as of 23.1.0 (now released), so for this plugin there should be no need for weird hacks to workaround the bundling.