GoCD 23.2/23.3 breaks agent mTLS connectivity when private key is encrypted/passphrase protected #11866

Mai-Khattab · 2023-08-14T12:51:00Z

Issue Type

Bug Report

Summary

Environment

Basic environment details

Go Version: 23.2.0
JAVA Version: 17.0.8
OS: Linux 3.10.0-1160.83.1.el7.x86_64
Browser vendor and version (if relevant): Edge

Additional Environment Details

Steps to Reproduce

Update GoCD server/agents to v23.2
Ensure that wrapper-properties have client certificate along with passphrase configurations as below:
wrapper.app.parameter.104=-sslCertificateFile
wrapper.app.parameter.105= /var/go/.vfclientcert/client.pem
wrapper.app.parameter.106=-sslPrivateKeyFile
wrapper.app.parameter.107=/var/go/.vfclientcert/client.key
wrapper.app.parameter.108=-sslPrivateKeyPassphraseFile
wrapper.app.parameter.109=/usr/share/go-agent/wrapper-config/test
wrapper.app.parameter.110=-sslVerificationMode
wrapper.app.parameter.111=FULL

Expected Results

Actual Results

Log snippets

2023-08-14 13:51:49,176 ERROR [WrapperJarAppMain] ServerBinaryDownloader:93 - Couldn't update admin/agent-launcher.jar. Sleeping for 60s. Error:
org.bouncycastle.openssl.PEMException: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available
        at org.bouncycastle.openssl.jcajce.PEMUtilities.getKey(Unknown Source)
        at org.bouncycastle.openssl.jcajce.PEMUtilities.getKey(Unknown Source)
        at org.bouncycastle.openssl.jcajce.PEMUtilities.crypt(Unknown Source)
        at org.bouncycastle.openssl.jcajce.JcePEMDecryptorProviderBuilder$1$1.decrypt(Unknown Source)
        at org.bouncycastle.openssl.PEMEncryptedKeyPair.decryptKeyPair(Unknown Source)
        at com.thoughtworks.go.agent.common.ssl.GoAgentServerClientBuilder.getPrivateKey(GoAgentServerClientBuilder.java:111)
        at com.thoughtworks.go.agent.common.ssl.GoAgentServerClientBuilder.keyStoreFromPem(GoAgentServerClientBuilder.java:93)
        at com.thoughtworks.go.agent.common.ssl.GoAgentServerClientBuilder.agentKeystore(GoAgentServerClientBuilder.java:83)
        at com.thoughtworks.go.agent.common.ssl.GoAgentServerHttpClientBuilder.build(GoAgentServerHttpClientBuilder.java:67)
        at com.thoughtworks.go.agent.launcher.ServerBinaryDownloader.fetchUpdateCheckHeaders(ServerBinaryDownloader.java:108)
        at com.thoughtworks.go.agent.launcher.ServerBinaryDownloader.downloadIfNecessary(ServerBinaryDownloader.java:83)
        at com.thoughtworks.go.agent.launcher.AgentLauncherImpl.doLaunch(AgentLauncherImpl.java:86)
        at com.thoughtworks.go.agent.launcher.AgentLauncherImpl.lambda$launch$0(AgentLauncherImpl.java:68)
        at com.thoughtworks.go.logging.LogConfigurator.runWithLogger(LogConfigurator.java:62)
        at com.thoughtworks.go.agent.launcher.AgentLauncherImpl.launch(AgentLauncherImpl.java:68)
        at com.thoughtworks.go.agent.bootstrapper.AgentBootstrapper.go(AgentBootstrapper.java:76)
        at com.thoughtworks.go.agent.bootstrapper.AgentBootstrapper.lambda$main$0(AgentBootstrapper.java:57)
        at com.thoughtworks.go.logging.LogConfigurator.runWithLogger(LogConfigurator.java:53)
        at com.thoughtworks.go.agent.bootstrapper.AgentBootstrapper.main(AgentBootstrapper.java:57)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at com.thoughtworks.gocd.Boot.run(Boot.java:90)
        at com.thoughtworks.gocd.Boot.main(Boot.java:56)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.tanukisoftware.wrapper.WrapperJarApp.run(WrapperJarApp.java:451)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.security.NoSuchAlgorithmException: PBKDF-OpenSSL SecretKeyFactory not available
        at java.base/javax.crypto.SecretKeyFactory.<init>(Unknown Source)
        at java.base/javax.crypto.SecretKeyFactory.getInstance(Unknown Source)
        at org.bouncycastle.jcajce.util.DefaultJcaJceHelper.createSecretKeyFactory(Unknown Source)
        ... 31 common frames omitted

The text was updated successfully, but these errors were encountered:

chadlwilson · 2023-08-14T13:40:20Z

Sorry, can you please describe the problem and your environment more clearly?

You haven't described what actually happened, what you expected to happen, how you install the server, what the previous version you were using (which presumably worked?) was - or any of the rest of the basic details requested by the issue template.

Mai-Khattab · 2023-08-14T18:35:54Z

Hello Chad,
Sorry, seems some details were removed due to formating.
So, I have a GoCD server/agents which were on version v22.1, recently I have upgraded them to v23.2 following the standard Linux based version instruction (yum update, restart) (https://docs.gocd.org/current/installation/install/server/linux.html).
They used to have the end-to-end transport security verification (client certificate verification) enabled as per (https://docs.gocd.org/current/installation/ssl_tls/end_to_end_transport_security.html).
However, after the upgrade to v23.2 the Agents converted from IDLE to LOST CONTACT with the below error message in the launcher logs.

Note when updating to v23.1 instead it worked normally.

2023-08-14 13:51:49,176 ERROR [WrapperJarAppMain] ServerBinaryDownloader:93 - Couldn't update admin/agent-launcher.jar. Sleeping for 60s. Error:
org.bouncycastle.openssl.PEMException: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available
        at org.bouncycastle.openssl.jcajce.PEMUtilities.getKey(Unknown Source)
        at org.bouncycastle.openssl.jcajce.PEMUtilities.getKey(Unknown Source)
        at org.bouncycastle.openssl.jcajce.PEMUtilities.crypt(Unknown Source)
        at org.bouncycastle.openssl.jcajce.JcePEMDecryptorProviderBuilder$1$1.decrypt(Unknown Source)
        at org.bouncycastle.openssl.PEMEncryptedKeyPair.decryptKeyPair(Unknown Source)
        at com.thoughtworks.go.agent.common.ssl.GoAgentServerClientBuilder.getPrivateKey(GoAgentServerClientBuilder.java:111)
        at com.thoughtworks.go.agent.common.ssl.GoAgentServerClientBuilder.keyStoreFromPem(GoAgentServerClientBuilder.java:93)
        at com.thoughtworks.go.agent.common.ssl.GoAgentServerClientBuilder.agentKeystore(GoAgentServerClientBuilder.java:83)
        at com.thoughtworks.go.agent.common.ssl.GoAgentServerHttpClientBuilder.build(GoAgentServerHttpClientBuilder.java:67)
        at com.thoughtworks.go.agent.launcher.ServerBinaryDownloader.fetchUpdateCheckHeaders(ServerBinaryDownloader.java:108)
        at com.thoughtworks.go.agent.launcher.ServerBinaryDownloader.downloadIfNecessary(ServerBinaryDownloader.java:83)
        at com.thoughtworks.go.agent.launcher.AgentLauncherImpl.doLaunch(AgentLauncherImpl.java:86)
        at com.thoughtworks.go.agent.launcher.AgentLauncherImpl.lambda$launch$0(AgentLauncherImpl.java:68)
        at com.thoughtworks.go.logging.LogConfigurator.runWithLogger(LogConfigurator.java:62)
        at com.thoughtworks.go.agent.launcher.AgentLauncherImpl.launch(AgentLauncherImpl.java:68)
        at com.thoughtworks.go.agent.bootstrapper.AgentBootstrapper.go(AgentBootstrapper.java:76)
        at com.thoughtworks.go.agent.bootstrapper.AgentBootstrapper.lambda$main$0(AgentBootstrapper.java:57)
        at com.thoughtworks.go.logging.LogConfigurator.runWithLogger(LogConfigurator.java:53)
        at com.thoughtworks.go.agent.bootstrapper.AgentBootstrapper.main(AgentBootstrapper.java:57)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at com.thoughtworks.gocd.Boot.run(Boot.java:90)
        at com.thoughtworks.gocd.Boot.main(Boot.java:56)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.tanukisoftware.wrapper.WrapperJarApp.run(WrapperJarApp.java:451)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.security.NoSuchAlgorithmException: PBKDF-OpenSSL SecretKeyFactory not available
        at java.base/javax.crypto.SecretKeyFactory.<init>(Unknown Source)
        at java.base/javax.crypto.SecretKeyFactory.getInstance(Unknown Source)
        at org.bouncycastle.jcajce.util.DefaultJcaJceHelper.createSecretKeyFactory(Unknown Source)
        ... 31 common frames omitted

chadlwilson · 2023-08-14T23:39:59Z

Thanks for the additional details.

Ugh, I think I might have an idea why this is.

To clarify, do things work fine if the server is on 23.2 or 23.3 but the agents are left on 23.1?

chadlwilson · 2023-08-15T01:01:19Z

Just to clarify, there is no major harm (or loss of functionality) with continuing to use the 'installed' agent 23.1.0 with a later GoCD Server so if this works fine for you, that's an acceptable workaround. The agent you 'install' is actually just a bootstrapper which connects to the server and downloads the 'real' agent binaries. Of course it needs to be able to connect first, so if using mtls 9n your reverse proxy, API GW or load balancer this issue might be a blocker.

If this was caused by a change (which seems likely) I'll still need to fix it though so would appreciate help narrowing it down.

Mai-Khattab · 2023-08-15T09:26:25Z

Thanks a lot for your feedback, Chad :)
So, below are the performed scenarios:

1- GoCD server on v22.1 and agents upgraded to v23.1 (Working).
2- GoCD server on v22.1 and agents upgraded to v23.2 (Not Working)
3- GoCD server on v23.2 and agents on v22.1 (Not Working)
4- GoCD server on v23.2 and agents on v23.2 (Not Working)

Hope this would help narrowing the issue.
Thanks again,

chadlwilson · 2023-08-15T09:45:07Z

Thanks for this.

Are you sure about this as "not working":

3- GoCD server on v23.2 and agents on v22.1 (Not Working)

If it's definitely not working in that combination, can you please share the agent-side log and stack trace? (appropriately redacted)

That combination should work, or at least any problem is likely to have a different root cause (off top of my head, I can't think of any other major incompatibility between agent bootstrappers and server between 22.1 and 23.2 but I can confirm if I can see the error experienced by agents with this configuration)

As a general rule, it's not wise to have agent bootstrappers and server differ by more than ~12 months due to the somewhat unwritten compatibility guarantees we try to adhere to (22.1 March 2022 vs 23.2 July 2023).

Mai-Khattab · 2023-08-15T19:40:16Z

Hello Chad,

Apologies , a correction in this scenario versions were (GoCD server on v23.2 and agents on v21.4) and it was not working with the same error message.

Actually, I got all agents updated o v23.2. I will try to check if I can rollback any and provide you wiht the logs.

chadlwilson · 2023-08-16T01:35:58Z

Its very unlikely to me that the error message (a BouncyCastle PEM parsing error) was the same on earlier agents such as 22.4 or 22.1, however as I said above its possible there is a separate compatibility issue that prevents agents from starting or talking to the server.

What did you do to get 23.2 agents to work? From what I can see that would not be possible unless you disable mTLS (I e stop using a client cert and key to authenticate with your reverse proxy/API GW/load balancer) or perform much deeper hacks to the JVM configuration used by the agent bootstrapper.

If the mTLS configuration is not actually being used or validated by your reverse proxy (perhaps it is old configuration) in front of your server removing that config would be fine, but if you are relying on mTLS for security/policy reasons I think you're best to stay with 23.1 agents.

The problem indicated by the specific BouncyCastle error here was introduced in 23.2.0 and I can replicate locally (I also have a fix, but I am trying to add regression tests to avoid such a problem accidentally being introduced again in future).

chadlwilson · 2023-08-21T01:52:29Z

Hi @Mai-Khattab - ive closed this now as I've addressed the root problem for 23.4, regardless of mix-and-matched agent/server versions.

After thinking further, you might be right that the GoCD server version of 23.2 or 23.3 would also have had issues, since the mTLS capability is not just required to initially bootstrap, but also to continue talking to the server.

It may not be acceptable to you, but you may find that another workaround to make mTLS possible with 23.2 and 23.3 is if the private key is not encrypted (i.e not protected with a passphrase).

Mai-Khattab · 2023-08-22T21:20:18Z

Hello Chad,

Thank you so much for the updates. I have a workaround currently in place. I will check once GoCD 23.4 released.

Kind Regards,

chadlwilson · 2023-08-22T23:49:51Z

In the spirit of open source, it would perhaps be useful to know what that workaround is, for the benefit of the wider community - rather than keeping it to yourself?

Mai-Khattab · 2024-01-16T08:55:36Z

Hello Chad,

Really apologies, I have just seen this comment right now.
Actually, it's not a code related workaround, rather than using some other agents which aren't using passphrase value set.

Regards,

chadlwilson · 2024-01-16T09:33:30Z

Ok thanks. Anyway, this issue should be resolved with 23.4.0 or 23.5.0 in any case.

Mai-Khattab changed the title ~~GoCD Server/Agent 23.2 || Client Certificate using Passphrase~~ GoCD Server/Agent 23.2 || Agents using client certificate along with passphrase lost connections. Aug 14, 2023

chadlwilson added agents bug labels Aug 15, 2023

chadlwilson added this to the Release 23.4.0 milestone Aug 15, 2023

chadlwilson self-assigned this Aug 15, 2023

chadlwilson mentioned this issue Aug 16, 2023

Correct agent connectivity with mTLS and arbitrary agent-side private key formats #11870

Merged

chadlwilson closed this as completed in #11870 Aug 20, 2023

chadlwilson changed the title ~~GoCD Server/Agent 23.2 || Agents using client certificate along with passphrase lost connections.~~ GoCD 23.2/23.3 breaks agent mTLS connectivity when private key is encrypted/passphrase protected Aug 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GoCD 23.2/23.3 breaks agent mTLS connectivity when private key is encrypted/passphrase protected #11866

GoCD 23.2/23.3 breaks agent mTLS connectivity when private key is encrypted/passphrase protected #11866

Mai-Khattab commented Aug 14, 2023 •

edited by chadlwilson

chadlwilson commented Aug 14, 2023 •

edited

Mai-Khattab commented Aug 14, 2023 •

edited by chadlwilson

chadlwilson commented Aug 14, 2023

chadlwilson commented Aug 15, 2023 •

edited

Mai-Khattab commented Aug 15, 2023

chadlwilson commented Aug 15, 2023 •

edited

Mai-Khattab commented Aug 15, 2023

chadlwilson commented Aug 16, 2023 •

edited

chadlwilson commented Aug 21, 2023

Mai-Khattab commented Aug 22, 2023

chadlwilson commented Aug 22, 2023 •

edited

Mai-Khattab commented Jan 16, 2024

chadlwilson commented Jan 16, 2024

GoCD 23.2/23.3 breaks agent mTLS connectivity when private key is encrypted/passphrase protected #11866

GoCD 23.2/23.3 breaks agent mTLS connectivity when private key is encrypted/passphrase protected #11866

Comments

Mai-Khattab commented Aug 14, 2023 • edited by chadlwilson

Issue Type

Summary

Environment

Basic environment details

Additional Environment Details

Steps to Reproduce

Expected Results

Actual Results

Log snippets

chadlwilson commented Aug 14, 2023 • edited

Mai-Khattab commented Aug 14, 2023 • edited by chadlwilson

chadlwilson commented Aug 14, 2023

chadlwilson commented Aug 15, 2023 • edited

Mai-Khattab commented Aug 15, 2023

chadlwilson commented Aug 15, 2023 • edited

Mai-Khattab commented Aug 15, 2023

chadlwilson commented Aug 16, 2023 • edited

chadlwilson commented Aug 21, 2023

Mai-Khattab commented Aug 22, 2023

chadlwilson commented Aug 22, 2023 • edited

Mai-Khattab commented Jan 16, 2024

chadlwilson commented Jan 16, 2024

Mai-Khattab commented Aug 14, 2023 •

edited by chadlwilson

chadlwilson commented Aug 14, 2023 •

edited

Mai-Khattab commented Aug 14, 2023 •

edited by chadlwilson

chadlwilson commented Aug 15, 2023 •

edited

chadlwilson commented Aug 15, 2023 •

edited

chadlwilson commented Aug 16, 2023 •

edited

chadlwilson commented Aug 22, 2023 •

edited