Fleet/flaky serverless api tests

[jillguyonnet]

Summary

Closes #176754
Closes #176858

Fleet serverless API integration tests tests have already experienced flakiness due to slowness as they expect a preconfigured Fleet Server host and Elasticsearch output to exist. #176808 added retry logic in the tests, which improved flakiness a lot.

However, it seems we are still experiencing failure due to Saved Object not found errors. This indicates that the test setup with retrying is not fully working as intended: if the default fleet server host or output was not found during setup, the failure should be one of:

Expected default Fleet Server id default-fleet-server to exist
Expected default Elasticsearch output id es-default-output to exist

This PR increases the timeout of the setup retry and renames the error message thrown by Fleet backend when the SO is not found.

Flaky Test Runner builds

• 🟢 x-pack/test_serverless/api_integration/test_suites/security/fleet/config.ts x 100
• 🟢 x-pack/test_serverless/api_integration/test_suites/observability/fleet/config.ts x 200
• 🟢 x-pack/test_serverless/api_integration/test_suites/security/fleet/config.ts x 200
• 🟢 x-pack/test_serverless/api_integration/test_suites/observability/fleet/config.ts x 200

[apmmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• /oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[jillguyonnet]

Just realised that #176808 was tagged for 8.14.0 and never backported... @kpollich Do you think this could be the issue? Should I add the backport: prev-minor to #176808?

[kpollich]

@jillguyonnet - Sure let's try to backport that PR.

[jillguyonnet]

The fix was backported to 8.13 in #178508. I've run 700 successful tests with the flaky test runner (cf. PR description) for good measure, so closing this for now.