add support for kibana policy overrides in system tests

[leehinman]

Adds support for a new environment variable ELASTIC_PACKAGE_KIBANA_POLICY_OVERRIDES

If this variable is set and is a path to a yaml file, then the contents of the yaml file will be used for the kibana policy overrides in the system tests.

This allows you to test how integrations will behave with experimental settings.

Example

ELASTIC_PACKAGE_KIBANA_POLICY_OVERRIDES=/tmp/overrides.yml elastic-package test system

and the /tmp/overrides.yml file has the following contents:

agent:
  monitoring:
    _runtime_experimental: otel

Will result in the system test running with the agent monitoring using the otel runtime.

Closes #2975

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 60d2241

[jsoriano]

Given that this intended for system tests, I think it would be better to include this in the test config files.

vars: ~
data_stream:
  vars: ~
overrides: |
  agent:
    monitoring:
      _runtime_experimental: otel

Though maybe the question is how this setting is intended to be used. An environment variable would be better if we expect to have CI pipelines testing everything with different configurations, while placing it in the test config files would allow to selectively enable it per test.

[leehinman]

We certainly could do them in the test config file.

One reason I went for the env var was because overrides are "unusual". I don't think we want to rely on them for CI. From a customer standpoint you would only use an override if something went wrong and you needed a work around until Elastic issues a fix.

The reason we want to have access to overrides for system tests is so we can test a configuration option without editing the integration in any way. And as you mentioned it gives us the option of doing a CI run with every integration using a particular override.

A concrete example is that we want to run all the metric system tests with a different runtime to make sure we get the same fields. We wouldn't want to edit all the test config files to do that.

[jsoriano]

And as you mentioned it gives us the option of doing a CI run with every integration using a particular override.

So being more specific about CI, the idea would be to be able to open a PR with an override file and the environment variable. And this PR would be used only for testing, without merging it. Is that right? For this I agree that an environment variable would make sense.