Fix cloude2e tests

[michel-laterman]

Fix issue that prevented cloud e2e test from running.

Output on previous tests had an error:

make[2]: Entering directory '/opt/buildkite-agent/builds/bk-agent-prod-gcp-1700554025504218876/elastic/fleet-server/dev-tools/cloud'
""
make[2]: Leaving directory '/opt/buildkite-agent/builds/bk-agent-prod-gcp-1700554025504218876/elastic/fleet-server/dev-tools/cloud'
FLEET_SERVER_URL=make[1]: Entering directory '/opt/buildkite-agent/builds/bk-agent-prod-gcp-1700554025504218876/elastic/fleet-server/dev-tools/cloud' "" make[1]: Leaving directory '/opt/buildkite-agent/builds/bk-agent-prod-gcp-1700554025504218876/elastic/fleet-server/dev-tools/cloud' go test -v -tags=cloude2e -count=1 -race -p 1 ./testing/cloude2e
bash: Entering: command not found
make[1]: *** [Makefile:396: test-cloude2e-set] Error 127
make[1]: Leaving directory '/opt/buildkite-agent/builds/bk-agent-prod-gcp-1700554025504218876/elastic/fleet-server'
make: [Makefile:389: test-cloude2e] Error 2 (ignored)

This change correctly runs the test

[nchaulet]

FIX LGTM 🚀
Anything we can do to make sure the build fail if something like happen again?

[michel-laterman]

Not sure, I thought that the -@set -o pipefail we use in test-cloude2e would catch the failure. Maybe we need it in cloud-e2e-set as well, i'll check

edit: nope, not sure how we can force the failure
edit2: @nchaulet the -@set -o pipefail ignores the status code of a command, however we want this as we want to clean a test cluster if there is a failure

[michel-laterman]

We need to hold off on merging this; it succeeds in my local env, but still has an issue in buildkite:

cd testing/cloude2e; FLEET_SERVER_URL=make[1]: Entering directory '/opt/buildkite-agent/builds/bk-agent-prod-gcp-1700682594679915170/elastic/fleet-server/dev-tools/cloud' "" make[1]: Leaving directory '/opt/buildkite-agent/builds/bk-agent-prod-gcp-1700682594679915170/elastic/fleet-server/dev-tools/cloud' go test -v -tags=cloude2e -count=1 -race -p 1 ./...
--
  | bash: Entering: command not found
  | make[1]: *** [Makefile:396: test-cloude2e-set] Error 127
  | make[1]: Leaving directory '/opt/buildkite-agent/builds/bk-agent-prod-gcp-1700682594679915170/elastic/fleet-server'
  | make: [Makefile:389: test-cloude2e] Error 2 (ignored)

[michel-laterman]

Another strange thing, when buildkite runs the cloude2e job part of the terrafrom output is fleet_url = "", but if make test-cloude2e is ran locally, it has value.
@mrodm, Do you have any idea what's different in buildkite that can be leading to these issues?

[michel-laterman]

Progress; the tests are running, and failing now:

fleet_url = ""
...
make[1]: Leaving directory '/opt/buildkite-agent/builds/bk-agent-prod-gcp-1700705081745419311/elastic/fleet-server/dev-tools/cloud'
make[1]: Entering directory '/opt/buildkite-agent/builds/bk-agent-prod-gcp-1700705081745419311/elastic/fleet-server'
cd testing/cloude2e; \
FLEET_SERVER_URL="" \
go test -v -tags=cloude2e -count=1 -race -p 1 ./...
...
=== RUN   TestBaseE2ETestSuite
=== RUN   TestBaseE2ETestSuite/TestFleetServerStatusOK
status_test.go:58:
 Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-gcp-1700705081745419311/elastic/fleet-server/testing/cloude2e/status_test.go:58
 Error:      	Received unexpected error:
 Get "/api/status": unsupported protocol scheme ""

The terraform state in buildkite does not properly record the fleet-server URL.
I'm not able to recreate locally

[mrodm]

Another strange thing, when buildkite runs the cloude2e job part of the terrafrom output is fleet_url = "", but if make test-cloude2e is ran locally, it has value. @mrodm, Do you have any idea what's different in buildkite that can be leading to these issues?

I just tried locally to run make -C ./dev-tools/cloud/ cloud-deploy , and I got the same outputs as the latest build in Buildkite:

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

elasticsearch_password = <sensitive>
elasticsearch_url = "https://xxx.us-west2.gcp.elastic-cloud.com:443"
elasticsearch_username = <sensitive>
fleet_url = ""
kibana_url = "https://xxxx.us-west2.gcp.elastic-cloud.com:9243"
stack_version = "8.12.0-SNAPSHOT"
make: Leaving directory '/home/mariorodriguez/Coding/work/fleet-server/dev-tools/cloud'

 $ make -C ./dev-tools/cloud/ cloud-get-fleet-url 
make: Entering directory '/home/mariorodriguez/Coding/work/fleet-server/dev-tools/cloud'
""
make: Leaving directory '/home/mariorodriguez/Coding/work/fleet-server/dev-tools/cloud'

Checked manually also the tfstate file, and those variables are empty strings.

[mergify]

This pull request is now in conflicts. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix-cloude2e upstream/fix-cloude2e
git merge upstream/main
git push upstream fix-cloude2e

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[jlind23]

@michel-laterman @kpollich How can we move forward with this one? I really would like us to rely on some working E2E fairly soon.

[mergify]

This pull request is now in conflicts. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix-cloude2e upstream/fix-cloude2e
git merge upstream/main
git push upstream fix-cloude2e

[michel-laterman]

terraform provider has been updated in another PR.
The issue with the endpoints not being populated can't be resolved through fixes to the test setup.

I'm closing the PR for now, if we want to port the status retry i added to the test we can do so at a later date