[Flaky Test]: TestStandaloneUpgrade – failed to detect remote snapshot repo

[rdner]

Failing test case

TestStandaloneUpgrade

Error message

failed to detect remote snapshot repo

Build

• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/451#018ff3ac-f265-4717-a37e-bff0e09f453b
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/725#01902a87-5bfa-47a3-ad7f-59f50dc73577
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/815#019036a0-33c4-4b4a-894b-646e83ecd8ee
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1013#01905f51-1574-47e9-b460-78d3d378b147
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1099
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1190
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1554
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1720
• https://buildkite.com/elastic/elastic-agent-extended-testing/builds/2087#01917b0e-94c4-495a-a21a-e027d743f289

Observed only when updating from 7.17.x to 8.x.

OS

Linux

Stacktrace and notes

upgrade_standalone_test.go:77: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/upgrade_standalone_test.go:77
        	            				/home/ubuntu/agent/testing/integration/upgrade_standalone_test.go:43
        	Error:      	Received unexpected error:
        	            	failed to start agent upgrade to version "8.15.0": exit status 1
        	            	Error: Failed trigger upgrade of daemon: initiating fetcher: failed to detect remote snapshot repo, proceeding with configured: invalid character '<' looking for beginning of value
        	            	For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/7.17/fleet-troubleshooting.html
        	Test:       	TestStandaloneUpgrade/Upgrade_7.17.23-SNAPSHOT_to_8.15.0-SNAPSHOT_(privileged)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[rdner]

@ycombinator we might want to prioritize this based on the amount of failed builds.

[cmacknz]

I suspect the < is the beginning of an XML response that would probably tell us we are being rate limited if we could see it. We should update the code to dump out the raw response when we can't parse it to confirm.

[ycombinator]

Another build that failed on this test: https://buildkite.com/elastic/elastic-agent-extended-testing/builds/2087#01917b0e-94c4-495a-a21a-e027d743f289. Added to list in issue description.