Support system tests otel input #2885

mrodm · 2025-09-10T11:07:38Z

Relates #2835
Depends on elastic/kibana#233090
Depends on elastic/kibana#235226
Depends on elastic/kibana#236333
Depends on elastic/fleet-server#5469

This PR enables to run system tests also in input packages using otelcol input.

In order to achieve that the validation process for fields and mappings is currently disabled because we need to determine the specific requirements for validation when the OTEL collector input is configured.

How to test this PR locally

Testing this requires:

running Elastic stack 9.2.0-SNAPSHOT.
enabling Kibana feature flag enableOtelIntegrations (already managed by elastic-package).

Steps:

elastic-package stack up -v -d --version 9.2.0-SNAPSHOT

elastic-package -C test/packages/parallel/httpcheck test system -v

elastic-package stack down -v

This reverts commit 77de02ed6bc3c765f8e88954fa3bee5e780ad37a.

mrodm · 2025-09-10T15:31:49Z

@jsoriano it looks like there is no "Index Template" created when these input packages using otelcol input are installed:

As a reference, for instance sql_input input package creates this Index Template:

This makes the system test fail with this error:

[0] failed to load mappings from index template preview (metrics-httpcheck.check.otel): error getting mapping: [400 Bad Request] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"unable to simulate template [metrics-httpcheck.check.otel] that does not exist"}],"type":"illegal_argument_exception","reason":"unable to simulate template [metrics-httpcheck.check.otel] that does not exist"},"status":400}

This is used to get the mappings that were installed to compare them with those that are available after being ingested the documents.

mrodm · 2025-09-10T15:38:03Z

Ok, it seems that it is pending as mentioned in elastic/kibana#232628

In addition they should contain @package and @custom templates included in current packages, so package developers and users can still customize mappings or pipelines.

Do you think there is going to be an Index Template installed for that as in the other packages?

internal/testrunner/runners/system/tester.go

mrodm · 2025-09-12T11:38:59Z

internal/testrunner/runners/system/tester.go

 		if err := r.kibanaClient.AddPackageDataStreamToPolicy(ctx, ds); err != nil {
 			return nil, fmt.Errorf("could not add data stream config to policy: %w", err)
 		}
 	}


It looks like that it does not need to update the datasets the Integration Policies created by elastic-package via:

ds := createPackageDatastream(*policyToTest, *r.pkgManifest, policyTemplate, *r.dataStreamManifest, *config, policyToTest.Namespace)

mrodm · 2025-09-12T14:33:00Z

internal/testrunner/runners/system/tester.go

+			if scenario.policyTemplateInput == otelCollectorInputName {
+				// Input packages whose input is `otelcol` must add the `.otel` suffix
+				// Example: httpcheck.metrics.otel
+				expectedDataset += "." + otelSuffixDataset
+			}


Documents ingested contain the dataset with suffix otel:

"data_stream": { "dataset": "httpcheck.check.otel", "namespace": "83653", "type": "metrics" },

Packages using OTEL collector input will perform the validation based on the mappings found before and after ingesting documents.

mrodm · 2025-09-23T08:31:10Z

/test

mrodm · 2025-09-23T09:01:45Z

/test

mrodm · 2025-09-24T07:44:03Z

/test

mrodm · 2025-09-30T08:23:40Z

internal/testrunner/runners/system/tester.go

+			// When using the OTEL collector input, just a subset of validations are performed (e.g. check expected datasets)
+			fields.WithOTELValidation(r.isTestUsingOTELCollectorInput(policyTemplateInput)),


Could input packages contain transforms ?
I think they cannot, so probably this validation based on fields for transforms could be kept as it is without disabling anything. WDYT ?

They cannot contain transforms. So yeah, I guess that transform tests will be a noop even if not explicitly disabled.

Ok, for input package there would not be any issue.
I was thinking if it should be kept that fields.WithOTELValidation option, if at some point integrations packages start using OTEL collector inputs.

WDYT ? Would it be better to keep it or remove it ? @jsoriano

I was thinking if it should be kept that fields.WithOTELValidation option, if at some point integrations packages start using OTEL collector inputs.

Yes, please, keep it. When we support integrations with OTel inputs there is no reason to think that they won't support transforms.

elasticmachine · 2025-09-30T09:14:52Z

💚 Build Succeeded

Buildkite Build
Commit: 4d9935a

History

💔 Build #6254 failed 9d8778d
💔 Build #6253 failed 93bd856
💔 Build #6248 failed 40fb639
💔 Build #6241 failed 93e382b
💔 Build #6239 failed 4d347ff

cc @mrodm

jsoriano · 2025-09-30T11:40:15Z

internal/testrunner/runners/system/tester.go

+			// When using the OTEL collector input, just a subset of validations are performed (e.g. check expected datasets)
+			fields.WithOTELValidation(r.isTestUsingOTELCollectorInput(policyTemplateInput)),


They cannot contain transforms. So yeah, I guess that transform tests will be a noop even if not explicitly disabled.

jsoriano · 2025-09-30T11:40:28Z

test/packages/parallel/httpcheck/_dev/test/system/test-local-config.yml

@@ -1,8 +1,7 @@
-skip:
-  reason: Not supported system tests with input type otelcol.
-  link: https://github.com/elastic/elastic-package/issues/2835


mrodm added 8 commits September 10, 2025 12:10

Update index template name

b161cb5

Remove outdated log message

6469ae4

Extract function to create kibana policies

fdb23e9

Revert "Extract function to create kibana policies"

2147b14

This reverts commit 77de02ed6bc3c765f8e88954fa3bee5e780ad37a.

Update package datastream - integration policy

8ae0043

Add otel as suffic for dataset

eb1450b

Keep package datastream / integration policy with the previous values

6fe40eb

Enable system test for httpcheck package

2bfebb8

mrodm self-assigned this Sep 10, 2025

mrodm mentioned this pull request Sep 10, 2025

Ensure input packages with otelcol input type (e.g. "OTel integrations") can be built, tested, and published with elastic-package #2835

Closed

mrodm added 3 commits September 10, 2025 17:41

httpcheck - Added field definitions

82f082f

httpcheck - replace string by keyword type

0a5db1b

Merge upstream/main into support-system-tests-otel-input

81df3a5

mrodm commented Sep 12, 2025

View reviewed changes

internal/testrunner/runners/system/tester.go Outdated Show resolved Hide resolved

mrodm commented Sep 12, 2025

View reviewed changes

Update comments

68d15da

mrodm commented Sep 12, 2025

View reviewed changes

mrodm added 2 commits September 18, 2025 16:08

Skip some OTEL fields - to be validated

fcc9f71

Update index template and data stream names for system tests

c70169d

mrodm mentioned this pull request Sep 19, 2025

[System tests] Support getting docs when it is set otelcol as policy template input #2933

Merged

mrodm added 3 commits September 22, 2025 12:54

Skip validation per fields in OTEL collector input

947cb6a

Remove skip fields for OTEL

cc93829

Packages using OTEL collector input will perform the validation based on the mappings found before and after ingesting documents.

Merge upstream/main into support-system-tests-otel-input

e9b5e34

mrodm mentioned this pull request Sep 23, 2025

[System tests] Refactor mapping validation #2939

Merged

mrodm mentioned this pull request Sep 25, 2025

[System tests] Skip dynamic templates that do not contain match conditions #2956

Merged

mrodm added 12 commits September 25, 2025 13:06

Skip dynamic templates without match condition

ff0213c

Replace stringInArray

928852e

Validate dataset fields also in OTEL

d812670

Remove duplicated assignment

d0664a8

Remove unnecessary else block

4d347ff

Merge upstream/main into support-system-tests-otel-input

93e382b

Add warning message

40fb639

Update test package

93bd856

Add OTEL option when validating fields in transform

5bd82b4

Remove all field definitions

9d8778d

Restore base-fields field definitions

dbc6b32

Remove skip test

4d9935a

mrodm commented Sep 30, 2025

View reviewed changes

mrodm marked this pull request as ready for review September 30, 2025 09:47

mrodm requested a review from a team September 30, 2025 09:48

jsoriano approved these changes Sep 30, 2025

View reviewed changes

mrodm merged commit 3c02bcd into elastic:main Oct 1, 2025
3 checks passed

mrodm deleted the support-system-tests-otel-input branch October 1, 2025 10:26

		// When using the OTEL collector input, just a subset of validations are performed (e.g. check expected datasets)
		fields.WithOTELValidation(r.isTestUsingOTELCollectorInput(policyTemplateInput)),

Support system tests otel input #2885

Support system tests otel input #2885

Uh oh!

Conversation

mrodm commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to test this PR locally

Uh oh!

mrodm commented Sep 10, 2025

Uh oh!

mrodm commented Sep 10, 2025

Uh oh!

Uh oh!

mrodm Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrodm Sep 12, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm commented Sep 23, 2025

Uh oh!

mrodm commented Sep 23, 2025

Uh oh!

mrodm commented Sep 24, 2025

Uh oh!

mrodm Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

jsoriano Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

jsoriano Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Sep 30, 2025

💚 Build Succeeded

History

Uh oh!

jsoriano Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

jsoriano Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mrodm commented Sep 10, 2025 •

edited

Loading

mrodm Sep 12, 2025 •

edited

Loading