Failing test: Fleet Serverless Observability API Integration Tests.x-pack/test_serverless/api_integration/test_suites/observability/fleet/fleet·ts - fleet rejects request to create a new fleet server hosts if host url is different from default

[kibanamachine]

A test failed on a tracked branch

JestAssertionError: expect(received).toEqual(expected) // deep equality

- Expected  - 1
+ Received  + 1

  Object {
    "error": "Forbidden",
-   "message": "Fleet server host must have default URL in serverless: https://localhost:8220",
+   "message": "Fleet server host must have default URL in serverless",
    "statusCode": 403,
  }
    at Context.<anonymous> (fleet.ts:26:20)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Object.apply (wrap_function.js:73:16) {
  matcherResult: {
    actual: {
      statusCode: 403,
      error: 'Forbidden',
      message: 'Fleet server host must have default URL in serverless'
    },
    expected: {
      statusCode: 403,
      error: 'Forbidden',
      message: 'Fleet server host must have default URL in serverless: https://localhost:8220'
    },
    message: '\x1B[2mexpect(\x1B[22m\x1B[31mreceived\x1B[39m\x1B[2m).\x1B[22mtoEqual\x1B[2m(\x1B[22m\x1B[32mexpected\x1B[39m\x1B[2m) // deep equality\x1B[22m\n' +
      '\n' +
      '\x1B[32m- Expected  - 1\x1B[39m\n' +
      '\x1B[31m+ Received  + 1\x1B[39m\n' +
      '\n' +
      '\x1B[2m  Object {\x1B[22m\n' +
      '\x1B[2m    "error": "Forbidden",\x1B[22m\n' +
      '\x1B[32m-   "message": "Fleet server host must have default URL in serverless: https://localhost:8220",\x1B[39m\n' +
      '\x1B[31m+   "message": "Fleet server host must have default URL in serverless",\x1B[39m\n' +
      '\x1B[2m    "statusCode": 403,\x1B[22m\n' +
      '\x1B[2m  }\x1B[22m',
    name: 'toEqual',
    pass: false
  }
}

First failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jbudz]

/skip

[kibanamachine]

Skipped

main: 1b04fae

[juliaElastic]

@jillguyonnet hey, any ideas why this test is failing?

[jillguyonnet]

The error is coming from here, essentially getDefaultFleetServerHost(soClient) did not retrieve a default Fleet server. I put the conditional in the error message to avoid exceptions in case this happened. Looking into why there wasn't a default Fleet server 👀

[jillguyonnet]

It seems these tests are flaky, see e.g. this build. The test configs (oblt, security) should correctly set up a default Fleet server. Assuming that works, it could be that this request fails or times out. Perhaps we could add some logging to try to narrow it down further, @elastic/fleet any opinions?
Edit: these tests never failed locally.

[kibanamachine]

New failure: CI Build - main

[jen-huang]

@jillguyonnet Looks like we are still having a failure here :(