Use fields if synthetic mode is enabled #1283

mrodm · 2023-05-30T15:32:38Z

This PR adds the support to check fields key from the documents retrieved from the Elasticsearch instead of _source.
For that, for each system test it is checked whether or not the synthetic mode is enabled:

This value is retrieved querying to the Component templates API

In case that data stream (component template) has synthetic enabled, there are some changes required:

validation in system tests:
- disabled Normalization
  - This normalization was introduced in Validate expected fields normalization #765
  - If synthetic is used, all elements in fields are arrays and this validation cannot be ensured.
- update validation about dataset
  - all the elements in fields are arrays, and that's need to be taken into account when checking for exact values
update how sample event files are generated (so static tests keep working):
- when synthetics source mode is enabled, this file is generated using the fields key from the documents with some modifications
  - if normalization is not set and there is just one element in the field, change the field to be just an scalar value.
  - if normalization is set, keep the the field as array

A test package has been included to test the synthetic mode:

sample_event.json is written using the fields key from the document
until 8.7.1 is not used as default version, synthetics is not going to be used

As part of this PR:

remove uses of errors.Wrap in the internal/fields/validate.go.
add a simple test for GetValue method from common.MapStr.

Considerations

As synthetic source mode is enabled (as part of the time series), _source field gets built from the documents and the fields on the document could be differnt.

For that, in case synthetic source mode is enabled, it is used the fields key from the documents retrieved from Elasticsearch instead of the _source field.

This makes that sample_event.json files must be re-generated for those data streams.

Packages (data streams) that do not use this mode are tested using the same _source field. No changes for them.

How to test

For this it can be used the apache_tomcat from the integrations repository

Build elastic-package from this branch
Go to the apache_tomcat package folder
Update the manifest of the cache datastream to set the time_series mode. Add at the end of the manifest this definition:
```
elasticsearch:
  index_mode: time_series
```

Run the following commands:

elastic-package stack down
elastic-package build -v
elastic-package stack up -v -d --version 8.7.1
# regenerate sample_event.json files for the desired data streams
elastic-package test system -v -g --data-streams cache,requests
elastic-package test -v

internal/testrunner/runners/system/runner.go

mrodm · 2023-05-30T18:57:54Z

internal/testrunner/runners/system/runner.go

+	if len(results.ComponentTemplates) == 0 {
+		logger.Debugf("no component template found for data stream %s", dataStream)
+		return false, nil
+	}


In current elastic stack version, there is no component template for sql_input package (input) for the Elastic stack version started (8.6.2).

In that scenario, I assume that synthetic is not enabled.

mrodm · 2023-05-31T15:59:48Z

/test

mrodm · 2023-06-01T10:03:37Z

The test package fails with these errors:

synthetic_mode/synthetic Verify sample_event.json:
[0] field "event.dataset" should have value "synthetic_mode.synthetic", it has "[synthetic_mode.synthetic]"
[1] field "data_stream.dataset" should have value "synthetic_mode.synthetic", it has "[synthetic_mode.synthetic]"

Probably it needs to wait until elastic-package runs by default 8.7.1 version so the synthetic source mode is detected.

jlind23 · 2023-06-01T10:14:10Z

Probably it needs to wait until elastic-package runs by default 8.7.1 version so the synthetic source mode is detected.

Shall we update the stack version being used then?

mrodm · 2023-06-01T10:24:46Z

Probably it needs to wait until elastic-package runs by default 8.7.1 version so the synthetic source mode is detected.

Shall we update the stack version being used then?

It is being done in this PR #1274
We could wait for this.

jlind23 · 2023-06-01T12:54:41Z

It is being done in this PR #1274
We could wait for this.

I guess we are good now :)

mrodm · 2023-06-01T14:56:49Z

It is being done in this PR #1274
We could wait for this.

I guess we are good now :)

There are still errors to be fixed in field validations when running static tests:

If the sample_event.json is written using the fields key:

Run static tests for the package
--- Test results for package: synthetic_mode - START ---
FAILURE DETAILS:
synthetic_mode/synthetic Verify sample_event.json:
[0] field "event.dataset" should have value "synthetic_mode.synthetic", it has "[synthetic_mode.synthetic]"
[1] field "data_stream.dataset" should have value "synthetic_mode.synthetic", it has "[synthetic_mode.synthetic]"


╭────────────────┬─────────────┬───────────┬──────────────────────────┬────────────────────────────────────────────┬──────────────╮
│ PACKAGE        │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT                                     │ TIME ELAPSED │
├────────────────┼─────────────┼───────────┼──────────────────────────┼────────────────────────────────────────────┼──────────────┤
│ synthetic_mode │ synthetic   │ static    │ Verify sample_event.json │ FAIL: one or more errors found in document │  53.778195ms │
╰────────────────┴─────────────┴───────────┴──────────────────────────┴────────────────────────────────────────────┴──────────────╯

If the sample_event.json is written using the _source key:

Run static tests for the package
--- Test results for package: synthetic_mode - START ---
FAILURE DETAILS:
synthetic_mode/synthetic Verify sample_event.json:
[0] field "event.type" is not normalized as expected: expected array, found "info" (string)
[1] field "event.category" is not normalized as expected: expected array, found "web" (string)


╭────────────────┬─────────────┬───────────┬──────────────────────────┬────────────────────────────────────────────┬──────────────╮
│ PACKAGE        │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT                                     │ TIME ELAPSED │
├────────────────┼─────────────┼───────────┼──────────────────────────┼────────────────────────────────────────────┼──────────────┤
│ synthetic_mode │ synthetic   │ static    │ Verify sample_event.json │ FAIL: one or more errors found in document │  53.692446ms │
╰────────────────┴─────────────┴───────────┴──────────────────────────┴────────────────────────────────────────────┴──────────────╯
--- Test results for package: synthetic_mode - END   ---

Trying to find a way to solve that yet.

jlind23 · 2023-06-05T05:40:22Z

@mrodm can I also mark this as closing elastic/integrations#6099 ?

mrodm · 2023-06-05T08:19:49Z

@mrodm can I also mark this as closing elastic/integrations#6099 ?

@jlind23 Not sure if it could be considered closed. If the current approach is approved, some changes in that integration would be needed so its tests are successful:

at least, it would be needed to update sample_event.json file for the data streams that require synthetic mode.

jsoriano

Good job here, there are quite some tricky cases.

I wonder if we can make this code less dependant on synthetic mode being enabled or not. For example we could try to always merge the fields in the source and the fields in fields. And assume that we have to disable normalization validation for all fields coming from fields. Then maybe we wouldn't even need to check if synthetic is enabled. But we can revisit this later when supporting runtime fields.

Something I think we have to confirm in any case before merging this is the format of the sample event, objects used to be expanded, but they are not here for the synthetic mode.

test/packages/other/synthetic_mode/data_stream/synthetic/sample_event.json

internal/fields/validate.go

internal/testrunner/runners/system/runner.go

jsoriano · 2023-06-05T21:12:20Z

internal/testrunner/runners/system/runner.go

+func (h hits) getDocs(syntheticsEnabled bool) []common.MapStr {
+	if syntheticsEnabled {
+		return h.Fields
+	}
+	return h.Source
+}


This is probably fine for synthetics, but I wonder if we should actually merge fields and source, to support also the runtime fields cases. But we can think about this when supporting tests with runtime fields.

mrodm · 2023-06-06T09:34:46Z

/test

Expand objects when docs retrieved from Elasticsearch and synthetic mode is enabled. That would keep the same format of the sample_event.json when that file is generated.

internal/fields/validate.go

jsoriano

Nice, the sample document looks better now, added a couple of suggestions more about it.

internal/fields/validate.go

test/packages/other/synthetic_mode/data_stream/synthetic/sample_event.json

internal/fields/validate.go

elasticmachine · 2023-06-07T12:29:59Z

💚 Build Succeeded

Buildkite Build
Commit: c3f1a30

History

💚 Build #919 succeeded 5a1cc24
💔 Build #918 failed 4855ce2
💚 Build #916 succeeded b1d4921
💔 Build #915 failed 935beb5
💚 Build #914 succeeded e3003d2
💔 Build #913 failed e3003d2

cc @mrodm

jsoriano

Nice 👍

mrodm added 11 commits May 18, 2023 15:31

Add fields into the search response

f5d9620

Test with fields

9d5a6c5

Request whether or not synthetics mode is enabled

fd4074b

Add option to disable normalization if synthetics enabled

2c41de3

add struct for hits source and fields

59b1252

Add some tests for mapstr

abfd266

Add helper to get string slice

6e5bd94

Remove debug

d3d7177

check dataset value if disabled normalization

f52a36d

Remove usages of errors.Wrap

5cbf156

Fix condition

abb0666

mrodm self-assigned this May 30, 2023

mrodm added 3 commits May 30, 2023 17:36

Add license header

3a4faca

Add testdata folder

5cec1e6

Remove unused test data file

fd482c0

mrodm commented May 30, 2023

View reviewed changes

internal/testrunner/runners/system/runner.go Show resolved Hide resolved

mrodm added 3 commits May 30, 2023 18:20

Add condition for no component template

6396fd7

fix log message

63dc94b

Show status

bfdb279

mrodm commented May 30, 2023

View reviewed changes

mrodm added 2 commits May 31, 2023 22:02

Add test package for synthetic

d981f9c

Add sample_event

3bd4cf1

mrodm requested a review from a team June 1, 2023 09:41

Merge remote-tracking branch 'upstream/main' into use_fields_synthetics

4468413

mrodm added 5 commits June 2, 2023 17:29

update settings test package

1a319cc

Sanitize docs when synthetic mode is enabled

e3c0a6d

comment debug messages

6afdeb2

Solve redundant break statement

bf55234

Fix lint

14179f6

mrodm added 2 commits June 5, 2023 13:03

Rename function

597d17c

Merge remote-tracking branch 'upstream/main' into use_fields_synthetics

8aacea2

mrodm marked this pull request as ready for review June 5, 2023 12:52

jsoriano reviewed Jun 5, 2023

View reviewed changes

mrodm added 4 commits June 6, 2023 10:48

Extract function

538c1c2

Extract function

abe7a0f

Rename

e669293

Move to a constant all fields body

e3003d2

mrodm added 2 commits June 6, 2023 13:01

Add context to error

935beb5

Add more context to some erros

b1d4921

jlind23 requested a review from jsoriano June 7, 2023 06:47

Expand objects in the docs retrieved from Elasticsearch

4855ce2

Expand objects when docs retrieved from Elasticsearch and synthetic mode is enabled. That would keep the same format of the sample_event.json when that file is generated.

mrodm commented Jun 7, 2023

View reviewed changes

internal/fields/validate.go Show resolved Hide resolved

Fix lint

5a1cc24

jsoriano reviewed Jun 7, 2023

View reviewed changes

internal/fields/validate.go Show resolved Hide resolved

test/packages/other/synthetic_mode/data_stream/synthetic/sample_event.json Outdated Show resolved Hide resolved

internal/fields/validate.go Outdated Show resolved Hide resolved

Remove array for those fields without definition

c3f1a30

mrodm requested a review from jsoriano June 7, 2023 13:20

jsoriano approved these changes Jun 8, 2023

View reviewed changes

mrodm merged commit a28f696 into elastic:main Jun 8, 2023

mrodm deleted the use_fields_synthetics branch June 8, 2023 14:03

mrodm mentioned this pull request Jul 11, 2024

Test elastic-package#1944 - DO NOT MERGE elastic/integrations#10353

Closed

Use fields if synthetic mode is enabled #1283

Use fields if synthetic mode is enabled #1283

Uh oh!

Conversation

mrodm commented May 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Considerations

How to test

Uh oh!

Uh oh!

mrodm May 30, 2023

Choose a reason for hiding this comment

Uh oh!

mrodm commented May 31, 2023

Uh oh!

mrodm commented Jun 1, 2023

Uh oh!

jlind23 commented Jun 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrodm commented Jun 1, 2023

Uh oh!

jlind23 commented Jun 1, 2023

Uh oh!

mrodm commented Jun 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jlind23 commented Jun 5, 2023

Uh oh!

mrodm commented Jun 5, 2023

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsoriano Jun 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrodm commented Jun 6, 2023

Uh oh!

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Jun 7, 2023

💚 Build Succeeded

History

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mrodm commented May 30, 2023 •

edited

Loading

jlind23 commented Jun 1, 2023 •

edited

Loading

mrodm commented Jun 1, 2023 •

edited

Loading

jsoriano Jun 5, 2023 •

edited

Loading