Unable to run commands with /bin/bash -c in v19.12.0

[bsodmike]

Issue Type

• Bug Report

Summary

GoCD v19.12.0

Steps to Reproduce

Run a job with the following command

              <exec command="/bin/bash">
                <arg>-c</arg>
                <arg>echo $HOME</arg>
                <runif status="passed" />
              </exec>

Expected Results

You should see the current path stored in the $HOME env var.

Actual Results

[go] Task: /bin/bash -ctook: 0.103sexited: 2
 "echo $HOME"
/bin/bash: -
: invalid option
Usage:	/bin/bash [GNU long option] [option] ...
	/bin/bash [GNU long option] [option] script-file ...
GNU long options:
	--debug
	--debugger
	--dump-po-strings
	--dump-strings
	--help
	--init-file
	--login
	--noediting
	--noprofile
	--norc
	--posix
	--rcfile
	--restricted
	--verbose
	--version
Shell options:
	-ilrsD or -c command or -O shopt_option		(invocation only)
	-abefhkmnptuvxBCHP or -o option
[go] Task status: failed, took: 0.103s, exited: 2

[bsodmike]

Here's something rather strange, this fails (as detailed above)

              <exec command="/bin/bash">
                <arg>-c</arg>
                <arg>echo $HOME</arg>
                <runif status="passed" />
              </exec>

However, this works just fine

              <exec command="/bin/bash">
                <arg>-c</arg>
                <arg>echo $HOME</arg>
              </exec>

Removing <runif status="passed" /> solved it for me. 😲

[kritika-singh3]

Can you share the agent logs from the time this task was failing?

[bsodmike]

Is this what you're after, or is it the log on the agent itself?

[go] Task: /bin/bash -ctook: 0.103sexited: 2
 "echo $HOME"
/bin/bash: -
: invalid option
Usage:	/bin/bash [GNU long option] [option] ...
	/bin/bash [GNU long option] [option] script-file ...
GNU long options:
	--debug
	--debugger
	--dump-po-strings
	--dump-strings
	--help
	--init-file
	--login
	--noediting
	--noprofile
	--norc
	--posix
	--rcfile
	--restricted
	--verbose
	--version
Shell options:
	-ilrsD or -c command or -O shopt_option		(invocation only)
	-abefhkmnptuvxBCHP or -o option
[go] Task status: failed, took: 0.103s, exited: 2
[go] Current job status: failed

[kritika-singh3]

No log on the agent itself.

[bsodmike]

 echo $HOME14:01:53,686 INFO  [scheduler-2] CommandLine:371 - Running command: /bin/bash -c
2020-01-08 14:01:53,689 WARN  [Thread-8606] CommandLine:48 - /bin/bash: -
2020-01-08 14:01:53,689 WARN  [Thread-8606] CommandLine:48 - : invalid option
2020-01-08 14:01:53,689 WARN  [Thread-8606] CommandLine:48 - Usage:	/bin/bash [GNU long option] [option] ...
2020-01-08 14:01:53,689 WARN  [Thread-8606] CommandLine:48 - 	/bin/bash [GNU long option] [option] script-file ...
2020-01-08 14:01:53,689 WARN  [Thread-8606] CommandLine:48 - GNU long options:
2020-01-08 14:01:53,689 WARN  [Thread-8606] CommandLine:48 - 	--debug
2020-01-08 14:01:53,689 WARN  [Thread-8606] CommandLine:48 - 	--debugger
2020-01-08 14:01:53,689 WARN  [Thread-8606] CommandLine:48 - 	--dump-po-strings
2020-01-08 14:01:53,689 WARN  [Thread-8606] CommandLine:48 - 	--dump-strings
2020-01-08 14:01:53,689 WARN  [Thread-8606] CommandLine:48 - 	--help
2020-01-08 14:01:53,689 WARN  [Thread-8606] CommandLine:48 - 	--init-file
2020-01-08 14:01:53,689 WARN  [Thread-8606] CommandLine:48 - 	--login
2020-01-08 14:01:53,689 WARN  [Thread-8606] CommandLine:48 - 	--noediting
2020-01-08 14:01:53,689 WARN  [Thread-8606] CommandLine:48 - 	--noprofile
2020-01-08 14:01:53,689 WARN  [Thread-8606] CommandLine:48 - 	--norc
2020-01-08 14:01:53,689 WARN  [Thread-8606] CommandLine:48 - 	--posix
2020-01-08 14:01:53,689 WARN  [Thread-8606] CommandLine:48 - 	--rcfile
2020-01-08 14:01:53,690 WARN  [Thread-8606] CommandLine:48 - 	--restricted
2020-01-08 14:01:53,690 WARN  [Thread-8606] CommandLine:48 - 	--verbose
2020-01-08 14:01:53,690 WARN  [Thread-8606] CommandLine:48 - 	--version
2020-01-08 14:01:53,690 WARN  [Thread-8606] CommandLine:48 - Shell options:
2020-01-08 14:01:53,690 WARN  [Thread-8606] CommandLine:48 - 	-ilrsD or -c command or -O shopt_option		(invocation only)
2020-01-08 14:01:53,690 WARN  [Thread-8606] CommandLine:48 - 	-abefhkmnptuvxBCHP or -o option
2020-01-08 14:01:53,789 WARN  [scheduler-2] BaseCommandBuilder:92 - return code is 2
2020-01-08 14:01:53,793 INFO  [scheduler-2] HttpService:129 - Got back 200 from server

[kritika-singh3]

@bsodmike I was able to reproduce the issue. As part of performance improvement to our pipeline config pages we started utilizing the Pipeline Config API to update tasks as part of #7400 .
While updating the config, the task contains a \r in the args which is treated as a newline by the command line. So the config which is saved is

<exec command="/bin/bash">
                <arg>-c\r</arg>
                <arg>echo $HOME</arg>
                <runif status="passed" />
 </exec>

Since \r is a non-printable character, it is not visible on any UI. The logs I asked was to confirm that the command is actually not utilizing the echo part.
Hence, the actual command which ran was /bin/bash -c which is an incorrect. Hence the error.

Now, the fix you mentioned was not due to removing the said line. It was because the whole config was overwritten by the arg without \r in it.
Thanks for reporting the issue. I will take a deeper look into it later in this week.

[sfuhrm]

Unfortunately I did not find any \r sequences in the cruise-config.xml of our setup.

For all of you needing a workaround I have found out this:

• After changing the pipeline configuration, restart the server (systemctl restart go-server)
• Now the config seems to be consistent again and GoCD is usable

Not the best workaround for larger teams, but I hope it helps at least some people.

[maheshp]

The other workaround is to save the cruise_config.xml without making any changes.

1. Go to https://<gocd_server_url>/go/admin/config_xml
2. Click Edit and Save

[kritika-singh3]

There is another workaround but it will effect the performance of the pipeline config save page.
The changes around performance is behind a toggle. Super admins can turn that toggle off via:

curl -k 'https://go-server-url/go/api/admin/feature_toggles/fast_pipeline_save' 
       -H 'Accept: application/vnd.go.cd+json' 
       -uusername:password 
       -X PUT 
       -d '{ "toggle_value": "off"}'

WARNING: This will revert the changes made for faster save of the pipeline configs (as it will make a full config save for every change).

[rajiesh]

Verified on 20.1.0 (11055-ef4fcdcffd933ed6a319262c2ac112b6e56c7936)