[Testing] Fail test if there are errors in Agent component logs

[andrewkroh]

As a developer I want to be able to detect if there are errors occurring during system tests. These are some scenarios I would like to detected as early as possible:

• An event is failing to index (elastic-agent-20230313-80.ndjson-{"log.level":"warn","@timestamp":"2023-03-13T15:43:25.350Z","message":"Cannot index event publisher.Event{Content:beat.Event{...). The system test will pass as long as other events are indexed.
• An input is logging warnings, errors, or deprecation warnings.
• Any panics occurring in the Agent during the system test.

I'd love discuss some options for implementing these types of checks for system tests without making a brittle system.

[cmacknz]

I like this, it would have caught several problems that eventually escalated into more serious issues. I think the problem is it will frustrate package developers because there are often errors that have nothing to do with the package under development. It can also take some time to fix the underlying root cause.

Perhaps we can add a sort of error white list where an error can be ignored in the logs as long as it is linked to an open GitHub issue. This would ensure they all get reported, but it will still lead to errors potentially being ignored for a while.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[jlind23]

@jen-huang added this to your backlog.

[mrodm]

Currently, system tests do not check any logs for any container. That should be implemented/added to the system tests as a final check. There is some logic already implemented to dump logs (elastic-package stack dump), maybe this can be leveraged to get the logs of the elastic-agent container.

What logs should be checked ? The output of the docker-compose logs command for the elastic-agent container or the internal logs written in the path inside the container at /usr/share/elastic-agent/state/data/logs/. Do those two have the same contents?

It could be started this by checking just a subset of the errors pointed out as a PoC. I was thinking to check in those logs for messages that contain Cannot index event publisher.Event as well as if there is any panic.

[jlind23]

@michalpristas can you help us answer Mario's question above?

[mrodm]

Just checked that there is an issue when retrieving logs from container using docker-compose v2 (docker-compose logs command used by elastic-package). No logs are written in the elastic-stack-dump folder.

As CI already runs in docker-compose v2 (#1116) , I created a new issue to solve this for docker-compose v2 #1251

[mrodm]

Working in a PoC for this issue in #1256, elastic-package is able to check the elastic-agent logs (the ones from docker-compose logs, not the internal ones).

Currently, that elastic-package looks for elastic-agent logs that contain this string:

• Cannot index event publisher.Event

It has been set just one type of messages, to ensure that it works in this PoC and to try to get errors related to the packages itself.

Having that branch, I created a PR in integrations where this new custom elastic-package is used, so it can be tested with the current integrations.

In this stage, there are currently two packages that have those kind of errors (systems and elasticsearch):

• Jenkins link: https://fleet-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations/detail/PR-6133/4/tests
• update elastic-package integrations#6133 (comment)

Elastic stack versions used for those packages:

• system package is using Elastic stack 8.7.1 (jenkins stage link)
• elasticsearch package is using Elastic stack 8.8.0-SNAPSHOT (jenkins stage link)