Secret Variable Deleted on GoCD Startup Config Migration

[Evesy]

Issue Type

• Bug Report

Summary

During a GoCD server restart, a particular secret variable in a pipeline appears to have been essentially deleted by GoCD during startup

Environment

Basic environment details

• Go Version: 23.1.0
• JAVA Version: 17.0.8
• OS: Rocky 8 (Linux 5.15.107+)

Additional Environment Details

Steps to Reproduce

Unable to reproduce as of yet

Expected Results

Secret variables should not be affected by server restarts

Actual Results

Secret variable was made non-secure during restart, making it unavaiable in pipelines.

Possible Fix

Log snippets

commit 22d5f0b25551194d8c9b4eb2c26e41bff86e6a28
Author: anonymous <go-cd-dev@googlegroups.com>
Date:   Fri Aug 11 16:49:43 2023 +0000

    user:anonymous|timestamp:1691772583558|schema_version:139|go_edition:OpenSource|go_version:23.1.0 (16079-21e78c998e1eb35d8d489c1d3e3e9813dc18233a)|md5:316b2419a68e7091c5ab2e99fd9d80e0

diff --git a/cruise-config.xml b/cruise-config.xml
index 6e154f03..32392741 100644
--- a/cruise-config.xml
+++ b/cruise-config.xml
@@ -10739,7 +10739,7 @@ spec:
     </pipeline>
     <pipeline name="shippr-cli.rate-of-sale-model-service" lockBehavior="unlockWhenFinished" template="shippr-cli_build_preprod_prod_cd">
       <environmentvariables>
-        <variable name="AWS_ACCESS_KEY_ID" secure="true">
+        <variable name="AWS_ACCESS_KEY_ID">
           <encryptedValue>AES:RbJiPlCY9loPDDscovN9jA==:BoYdhzt4YexgEf2indobBiS9IKBuPs99R+wMn5hxDqc=</encryptedValue>
         </variable>
         <variable name="AWS_SECRET_ACCESS_KEY" secure="true">
@@ -10847,7 +10847,7 @@ spec:
         </variable>
       </environmentvariables>
       <materials>
-        <git url="https://github.atcloud.io/AutoTrader/atvaluations.git" autoUpdate="false">
+        <git url="https://github.atcloud.io/AutoTrader/atvaluations.git">
           <filter>
             <ignore pattern="README.*" />
             <ignore pattern=".editorconfig" />

---
commit ce8898193596198c53372d50416c9ebe66ca3b2d
Author: Upgrade <go-cd-dev@googlegroups.com>
Date:   Fri Aug 11 16:49:40 2023 +0000

    user:Upgrade|timestamp:1691772580766|schema_version:139|go_edition:OpenSource|go_version:23.1.0 (16079-21e78c998e1eb35d8d489c1d3e3e9813dc18233a)|md5:cb9f95e6b2f8d1db2b4d8566befcba32

diff --git a/cruise-config.xml b/cruise-config.xml
index 32392741..6e154f03 100644
--- a/cruise-config.xml
+++ b/cruise-config.xml
@@ -10739,7 +10739,7 @@ spec:
     </pipeline>
     <pipeline name="shippr-cli.rate-of-sale-model-service" lockBehavior="unlockWhenFinished" template="shippr-cli_build_preprod_prod_cd">
       <environmentvariables>
-        <variable name="AWS_ACCESS_KEY_ID">
+        <variable name="AWS_ACCESS_KEY_ID" secure="true">
           <encryptedValue>AES:RbJiPlCY9loPDDscovN9jA==:BoYdhzt4YexgEf2indobBiS9IKBuPs99R+wMn5hxDqc=</encryptedValue>
         </variable>
         <variable name="AWS_SECRET_ACCESS_KEY" secure="true">
@@ -10847,7 +10847,7 @@ spec:
         </variable>
       </environmentvariables>
       <materials>
-        <git url="https://github.atcloud.io/AutoTrader/atvaluations.git">
+        <git url="https://github.atcloud.io/AutoTrader/atvaluations.git" autoUpdate="false">
           <filter>
             <ignore pattern="README.*" />
             <ignore pattern=".editorconfig" />

---
commit 90222157399c2346673966ad05749b0d182edbdf
Author: Filesystem <go-cd-dev@googlegroups.com>
Date:   Fri Aug 11 16:49:40 2023 +0000

    user:Filesystem|timestamp:1691772580736|schema_version:139|go_edition:OpenSource|go_version:23.1.0 (16079-21e78c998e1eb35d8d489c1d3e3e9813dc18233a)|md5:316b2419a68e7091c5ab2e99fd9d80e0

diff --git a/cruise-config.xml b/cruise-config.xml
index 6e154f03..32392741 100644
--- a/cruise-config.xml
+++ b/cruise-config.xml
@@ -10739,7 +10739,7 @@ spec:
     </pipeline>
     <pipeline name="shippr-cli.rate-of-sale-model-service" lockBehavior="unlockWhenFinished" template="shippr-cli_build_preprod_prod_cd">
       <environmentvariables>
-        <variable name="AWS_ACCESS_KEY_ID" secure="true">
+        <variable name="AWS_ACCESS_KEY_ID">
           <encryptedValue>AES:RbJiPlCY9loPDDscovN9jA==:BoYdhzt4YexgEf2indobBiS9IKBuPs99R+wMn5hxDqc=</encryptedValue>
         </variable>
         <variable name="AWS_SECRET_ACCESS_KEY" secure="true">
@@ -10847,7 +10847,7 @@ spec:
         </variable>
       </environmentvariables>
       <materials>
-        <git url="https://github.atcloud.io/AutoTrader/atvaluations.git" autoUpdate="false">
+        <git url="https://github.atcloud.io/AutoTrader/atvaluations.git">
           <filter>
             <ignore pattern="README.*" />
             <ignore pattern=".editorconfig" />

Any other info

As can be seen from the git history of the config.xml above, AWS_ACCESS_KEY_ID in the above pipeline had secure="true" removed by Filesystem re-added by Upgrade, and then removed again by anonymous.
This essentially turned the variable into a plaintext one with no value (even though the secure value was still persisted in the XML). Eventually when an unrelated change was made to the pipeline through the GUI (material change), the encrypted value was also deleted from the XML.

Unfortunately this was only spotted a couple of weeks later as it was a seldom used pipeline, so the go server logs have since been rotated away.

[chadlwilson]

This is probably the same as #11868

Without server logs I will close as a duplicate as there is no way to confirm the root cause is the same.

If you can look back through config history and see the oldest GoCD version where you see this flip-flopping happening it might help.

git log -S "secure=" -p

There is no current solid explanation for why it might be more commonly happening now, or on 23.1.0 (based on the confirmed bug fixed), and why a few people have suddenly reported the same thing, as the code here is very old.

[chadlwilson]

Would love to get that detail (or analysis) on how far back in your config repo (and with which GoCD versions) you've seen similar boolean flip-flopping as I understand you're relatively long time GoCD users - or any pattern you see in increase in frequency, but up to you 🙏

If you'd rather share privately, feel free to do so via chadw@thoughtworks.com

[chadlwilson]

Sorry for the delay, but wanted to add that this change is now out in GoCD 23.4.0. Fingers crossed that we've nailed this one! https://www.gocd.org/download/