Skip to content

add support for kibana policy overrides in system tests #3094

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 26, 2025
Merged

add support for kibana policy overrides in system tests #3094

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions docs/howto/system_testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -977,6 +977,23 @@ In this case, `elastic-package test system` will fail with an error and print a
skip_ignored_fields:
- field.to.ignore
```
### Kibana policy overrides

If you need to test a system test with a Kibana policy override you can do that by setting the environment variable `ELASTIC_PACKAGE_KIBANA_POLICY_OVERRIDES` to be a path to a yaml file that contains the policy override. For example:

```shell
ELASTIC_PACKAGE_KIBANA_POLICY_OVERRIDES=/tmp/overrides.yml elastic-package test system
```

and the `/tmp/overrides.yml` file has the following contents:

```yaml
agent:
monitoring:
_runtime_experimental: otel
```
Comment on lines +984 to +994
Copy link
Member

@jsoriano jsoriano Nov 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that this intended for system tests, I think it would be better to include this in the test config files.

vars: ~
data_stream:
  vars: ~
overrides: |
  agent:
    monitoring:
      _runtime_experimental: otel

Though maybe the question is how this setting is intended to be used. An environment variable would be better if we expect to have CI pipelines testing everything with different configurations, while placing it in the test config files would allow to selectively enable it per test.

Copy link
Contributor Author

@leehinman leehinman Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We certainly could do them in the test config file.

One reason I went for the env var was because overrides are "unusual". I don't think we want to rely on them for CI. From a customer standpoint you would only use an override if something went wrong and you needed a work around until Elastic issues a fix.

The reason we want to have access to overrides for system tests is so we can test a configuration option without editing the integration in any way. And as you mentioned it gives us the option of doing a CI run with every integration using a particular override.

A concrete example is that we want to run all the metric system tests with a different runtime to make sure we get the same fields. We wouldn't want to edit all the test config files to do that.

Copy link
Member

@jsoriano jsoriano Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And as you mentioned it gives us the option of doing a CI run with every integration using a particular override.

So being more specific about CI, the idea would be to be able to open a PR with an override file and the environment variable. And this PR would be used only for testing, without merging it. Is that right? For this I agree that an environment variable would make sense.


Will result in the system test running with the agent monitoring using the `otel` runtime.

## Continuous Integration

Expand Down
19 changes: 10 additions & 9 deletions internal/kibana/policies.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,16 @@ import (

// Policy represents an Agent Policy in Fleet.
type Policy struct {
ID string `json:"id,omitempty"`
Name string `json:"name"`
Description string `json:"description"`
Namespace string `json:"namespace"`
Revision int `json:"revision,omitempty"`
MonitoringEnabled []string `json:"monitoring_enabled,omitempty"`
MonitoringOutputID string `json:"monitoring_output_id,omitempty"`
DataOutputID string `json:"data_output_id,omitempty"`
IsDefaultFleetServer bool `json:"is_default_fleet_server,omitempty"`
ID string `json:"id,omitempty"`
Name string `json:"name"`
Description string `json:"description"`
Namespace string `json:"namespace"`
Revision int `json:"revision,omitempty"`
MonitoringEnabled []string `json:"monitoring_enabled,omitempty"`
MonitoringOutputID string `json:"monitoring_output_id,omitempty"`
DataOutputID string `json:"data_output_id,omitempty"`
IsDefaultFleetServer bool `json:"is_default_fleet_server,omitempty"`
Overrides map[string]any `json:"overrides,omitempty"`
}

// DownloadedPolicy represents a policy as returned by the download policy API.
Expand Down
18 changes: 18 additions & 0 deletions internal/testrunner/runners/system/tester.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,7 @@ var (
dumpScenarioDocsEnv = environment.WithElasticPackagePrefix("TEST_DUMP_SCENARIO_DOCS")
fieldValidationTestMethodEnv = environment.WithElasticPackagePrefix("FIELD_VALIDATION_TEST_METHOD")
prefixServiceTestRunIDEnv = environment.WithElasticPackagePrefix("PREFIX_SERVICE_TEST_RUN_ID")
kibanaPolicyOverridesEnv = environment.WithElasticPackagePrefix("KIBANA_POLICY_OVERRIDES")
)

type fieldValidationMethod int
Expand Down Expand Up @@ -230,6 +231,8 @@ type tester struct {

globalTestConfig testrunner.GlobalRunnerTestConfig

kibanaPolicyOverrides map[string]any

// Execution order of following handlers is defined in runner.TearDown() method.
removeAgentHandler func(context.Context) error
deleteTestPolicyHandler func(context.Context) error
Expand Down Expand Up @@ -348,6 +351,20 @@ func NewSystemTester(options SystemTesterOptions) (*tester, error) {
r.fieldValidationMethod = method
}

// kibana policy overrides
v, ok = os.LookupEnv(kibanaPolicyOverridesEnv)
if ok {
data, err := os.ReadFile(v)
if err != nil {
return nil, fmt.Errorf("error reading file %q specified in %s: %w", v, kibanaPolicyOverridesEnv, err)
}
overrides := make(map[string]any)
if err = yaml.Unmarshal(data, &overrides); err != nil {
return nil, fmt.Errorf("error reading yaml from %s: %w", v, err)
}
r.kibanaPolicyOverrides = overrides
}

return &r, nil
}

Expand Down Expand Up @@ -1374,6 +1391,7 @@ func (r *tester) createOrGetKibanaPolicies(ctx context.Context, serviceStateData
Name: fmt.Sprintf("ep-test-system-%s-%s-%s-%s-%s", r.testFolder.Package, r.testFolder.DataStream, r.serviceVariant, r.configFileName, testTime),
Description: fmt.Sprintf("test policy created by elastic-package test system for data stream %s/%s", r.testFolder.Package, r.testFolder.DataStream),
Namespace: common.CreateTestRunID(),
Overrides: r.kibanaPolicyOverrides,
}
// Assign the data_output_id to the agent policy to configure the output to logstash. The value is inferred from stack/_static/kibana.yml.tmpl
// TODO: Migrate from stack.logstash_enabled to the stack config.
Expand Down