Check logs from elastic-agent in system tests #1256

mrodm · 2023-05-09T13:52:52Z

This PR adds into system test a new check to look for errors in container logs. These errors could be specified by means of patterns in the code.

A new parameter --profile (or -p) has been added into elastic-package test so it can be retrieved the profile information, required to dump the logs of the services.

This adds a new test result whose Test Name is (<service> logs) under the test type system.

Example of output setting fake patterns:

 $ elastic-package test system -v
...
FAILURE DETAILS:
elastic_package_registry/metrics (elastic-agent logs):
[0] found error "New State ID is AbF6RZN5"
[1] found error "New State ID is kTKFMOWJ"
elastic_package_registry/metrics (package-registry logs):
[0] found error "GET / HTTP/2.0"
[1] found error "GET /package/elastic_package_registry/0.0.6 HTTP/1.1"
[2] found error "GET /package/elastic_package_registry/0.0.6/ HTTP/1.1"
[3] found error "GET /search HTTP/1.1"


╭──────────────────────────┬─────────────┬───────────┬─────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────┬───────────────╮
│ PACKAGE                  │ DATA STREAM │ TEST TYPE │ TEST NAME               │ RESULT                                                                                │  TIME ELAPSED │
├──────────────────────────┼─────────────┼───────────┼─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┼───────────────┤
│ elastic_package_registry │ metrics     │ system    │ default                 │ PASS                                                                                  │ 22.331858707s │
│ elastic_package_registry │ metrics     │ system    │ (elastic-agent logs)    │ FAIL: test case failed: one or more errors found while examining elastic-agent.log    │    4.287049ms │
│ elastic_package_registry │ metrics     │ system    │ (package-registry logs) │ FAIL: test case failed: one or more errors found while examining package-registry.log │    85.89657ms │
╰──────────────────────────┴─────────────┴───────────┴─────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────┴───────────────╯
--- Test results for package: elastic_package_registry - END   ---
Done

How to test this locally

In a given package, run the following commands:

 $ elastic-package build -v
 # start the stack with the required version (e.g. 8.0.0)
 $ elastic-package stack up -v -d --version <version>
 $ elastic-package test system -v
 $ elastic-package stack down

cmd/profiles.go

jsoriano

Approach looks good in general, added some observations and nitpicking.

cmd/profiles.go

internal/stack/parse_logs.go

internal/testrunner/runners/system/runner.go

Do not store all the messages of docker-compose logs in memory, they are now processed with a function passed as a parameter. Removed unnecessary variables.

mrodm · 2023-05-11T08:20:13Z

cmd/testrunner.go

 	cmd.PersistentFlags().BoolP(cobraext.TestCoverageFlagName, "", false, cobraext.TestCoverageFlagDescription)
 	cmd.PersistentFlags().DurationP(cobraext.DeferCleanupFlagName, "", 0, cobraext.DeferCleanupFlagDescription)
 	cmd.PersistentFlags().String(cobraext.VariantFlagName, "", cobraext.VariantFlagDescription)
+	cmd.PersistentFlags().StringP(cobraext.ProfileFlagName, "p", "", fmt.Sprintf(cobraext.ProfileFlagDescription, install.ProfileNameEnvVar))


Added profile parameter into elastic-package test command

mrodm · 2023-05-11T09:50:35Z

@mrodm I don't think it is a problem, it means that this packages might have to be fixed. Do you know what the problem is? Could it tests that on purpose logs errors?

I don't know the reason of these errors nor the root cause of them

jlind23 · 2023-05-11T09:58:57Z

As soon as we merge we should ask the teams to look at these. You can already ping them btw

mrodm · 2023-05-15T16:03:02Z

internal/testrunner/runners/system/runner.go

+			// logsRegexp{
+			// 	includes: regexp.MustCompile("New State ID"),
+			// 	excludes: []*regexp.Regexp{
+			// 		regexp.MustCompile("is unahorized API key id"),
+			// 	},
+			// },
+			// logsRegexp{
+			// 	includes: regexp.MustCompile("->HEALTHY"),
+			// 	excludes: []*regexp.Regexp{
+			// 		regexp.MustCompile(`Healthy$`),
+			// 	},
+			// },


to be deleted

jsoriano

This is looking good, could we add some tests that cover the pattern matching logic?

cmd/stack.go

internal/stack/parselogs.go

internal/testrunner/runners/system/runner.go

…nt_in_tests

elasticmachine · 2023-05-29T09:22:05Z

💚 Build Succeeded

Buildkite Build
Commit: 57b2764

History

💚 Build #756 succeeded f2dca7a
💚 Build #755 succeeded 1167bcd
💚 Build #753 succeeded 5e91558
💚 Build #751 succeeded 4e8b885
💚 Build #738 succeeded d5f338b
💚 Build #737 succeeded 1ffcafa

cc @mrodm

jlind23 · 2023-06-01T07:24:26Z

@mrodm @jsoriano are we good with merging this or is there still some work to do?

mrodm · 2023-06-01T08:05:34Z

@mrodm @jsoriano are we good with merging this or is there still some work to do?

@jlind23 there is still a final review pending, but I hope it shouldn't be pending too many changes.
But, considering that this PR would be approved, I have some concerns on merging this.

There are still three issues pending to be fixed (one was already fixed):

If this PR is merged, then any attempt to update elastic-package on integrations repository would be blocked (there would be failures) and even future elastic-package releases.

If that following PR in integrations is merged (those failures are ignored), then all those packages affected would be failing and they could not release any new version of the package.

What should we do here? @jsoriano @jlind23

jlind23 · 2023-06-01T08:24:40Z

@mrodm in order to move forward can't we mark these 3 failing packages as an exception until it's fixed? If we wait more some new failures can be introduced in the mean time

mrodm · 2023-06-01T10:26:26Z

@mrodm in order to move forward can't we mark these 3 failing packages as an exception until it's fixed? If we wait more some new failures can be introduced in the mean time

Not sure how to filter that, packages could fail for other reasons too.

An option (not the best one), they could ask us to merge those PRs...

jsoriano · 2023-06-01T11:36:34Z

@mrodm in order to move forward can't we mark these 3 failing packages as an exception until it's fixed? If we wait more some new failures can be introduced in the mean time

Not sure how to filter that, packages could fail for other reasons too.

An option (not the best one), they could ask us to merge those PRs...

It is possible to skip system tests (see docs). We could add something like this to their test definition when updating elastic-package in integrations:

skip:
  link: https://github.com/elastic/integrations/issues/6178
  reason: "This integration produces events that cannot be indexed"

Each one with its link, and include a comment on these issues about the test being skipped.

jsoriano

👍

mrodm · 2023-06-01T13:33:46Z

It is possible to skip system tests (see docs). We could add something like this to their test definition when updating elastic-package in integrations:
skip:
  link: https://github.com/elastic/integrations/issues/6178
  reason: "This integration produces events that cannot be indexed"
Each one with its link, and include a comment on these issues about the test being skipped.

Thanks @jsoriano !
Then, it is good to be merged now, and we can add those skips in the test system configuration in the required packages in the integrations repository 👍

mrodm added 6 commits May 9, 2023 10:50

Use profile in tests

f491083

Add method to check errors in logs

74d0851

Do no run check errors if there are no patterns

7a565a6

Update README

a392d52

Add service name into reason

3434fa1

Add stack parse logs

889473c

mrodm commented May 9, 2023

View reviewed changes

cmd/profiles.go Outdated Show resolved Hide resolved

mrodm changed the title ~~Check logs from elastic-agent in system tests~~ WIP - Check logs from elastic-agent in system tests May 9, 2023

Add comments

b1c8d3f

mrodm force-pushed the check_logs_from_agent_in_tests branch from e4ec44a to b1c8d3f Compare May 9, 2023 14:51

Use regex

7e44426

mrodm mentioned this pull request May 9, 2023

update elastic-package elastic/integrations#6133

Closed

jsoriano reviewed May 9, 2023

View reviewed changes

Comments from code review

9e6d45e

Do not store all the messages of docker-compose logs in memory, they are now processed with a function passed as a parameter. Removed unnecessary variables.

mrodm force-pushed the check_logs_from_agent_in_tests branch from 73e2434 to 9e6d45e Compare May 10, 2023 09:51

mrodm added 7 commits May 10, 2023 11:52

Move helpers to internal/cobraext

49069bd

Add missing license header

2c8e538

Remove debug messages

8d20ebf

Fix errors related to helpers

a580527

Make public the helpers

64513fb

Use logs file path instead of service name

65784c7

Comment panic/runtime errors

039d672

mrodm mentioned this pull request May 10, 2023

[Testing] Fail test if there are errors in Agent component logs #1188

Closed

mrodm self-assigned this May 11, 2023

mrodm commented May 11, 2023

View reviewed changes

mrodm added 2 commits May 11, 2023 10:24

Support check logs from several services

58fb1bd

Use just errorPatterns to check services

4d48fd9

mrodm changed the title ~~WIP - Check logs from elastic-agent in system tests~~ Check logs from elastic-agent in system tests May 11, 2023

mrodm marked this pull request as ready for review May 11, 2023 08:43

Remove debug patterns

1ffcafa

mrodm requested a review from jsoriano May 11, 2023 09:58

Add a new regex for elastic-agent

d5f338b

mrodm mentioned this pull request May 11, 2023

add steps: check-ci, make-local, unit-test, int-test, annotate elastic/fleet-server#2543

Merged

8 tasks

Add includes and excludes regex

4e8b885

mrodm commented May 15, 2023

View reviewed changes

jsoriano reviewed May 15, 2023

View reviewed changes

cmd/stack.go Outdated Show resolved Hide resolved

internal/stack/parselogs.go Show resolved Hide resolved

internal/testrunner/runners/system/runner.go Outdated Show resolved Hide resolved

mrodm added 5 commits May 16, 2023 11:47

Rename function

5e91558

Add test for checkAgentLogs

acfe286

Change struct to ensure same order for logs found

e483b4f

Remove debug regexes

1167bcd

Add test using excludes

f2dca7a

mrodm requested a review from jsoriano May 17, 2023 07:49

Merge remote-tracking branch 'upstream/main' into check_logs_from_age…

57b2764

…nt_in_tests

jsoriano approved these changes Jun 1, 2023

View reviewed changes

mrodm merged commit f1106b9 into elastic:main Jun 1, 2023

mrodm deleted the check_logs_from_agent_in_tests branch June 1, 2023 13:39

mrodm mentioned this pull request Jun 5, 2023

Bump github.com/elastic/elastic-package from 0.80.0 to 0.81.0 elastic/integrations#6472

Merged

efd6 mentioned this pull request Jun 7, 2023

f1106b9b causes build failures due to valid log lines #1296

Closed

This was referenced Apr 10, 2024

feat: introduce --fail-on-ingest-warnings to detect, and treat as errors, warnings from ingest pipeline #1755

Merged

Add Github workflow to execute Go tests elastic/go-resource#4

Merged

mrodm mentioned this pull request Jun 19, 2025

Log messages like "Cannot index events" are not detected anymore in system tests #2669

Open

Check logs from elastic-agent in system tests #1256

Check logs from elastic-agent in system tests #1256

Uh oh!

Conversation

mrodm commented May 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to test this locally

Uh oh!

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mrodm May 11, 2023

Choose a reason for hiding this comment

Uh oh!

mrodm commented May 11, 2023

Uh oh!

jlind23 commented May 11, 2023

Uh oh!

mrodm May 15, 2023

Choose a reason for hiding this comment

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented May 29, 2023

💚 Build Succeeded

History

Uh oh!

jlind23 commented Jun 1, 2023

Uh oh!

mrodm commented Jun 1, 2023

Uh oh!

jlind23 commented Jun 1, 2023

Uh oh!

mrodm commented Jun 1, 2023

Uh oh!

jsoriano commented Jun 1, 2023

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

mrodm commented Jun 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mrodm commented May 9, 2023 •

edited

Loading

mrodm commented Jun 1, 2023 •

edited

Loading