Cherry-pick #20915 to 7.x: [Autodiscover] Handle input-not-finished errors in config reload #20928

ChrsMark · 2020-09-02T14:32:00Z

Cherry-pick of PR #20915 to 7.x branch. Original message:

What does this PR do?

This PR adds special error handling in config reload flow to handle state errors specially by not logging in ERROR level to avoid confusion in logging.

Current state:

2020-08-26T07:05:05.386Z        ERROR   [autodiscover]  cfgfile/list.go:95      Error creating runner from config: Can only start an input when all related states are finished: {Id: ea745ab688be85a9-native::1308836-2049, Finished: false, Fileinfo: &{628a5ca3a5e5037056ebabc9a016b13d4a6ebb16c44a6216beff387b65ff5c4b-json.log 99 416 {29223485 63734022237 0x608b880} {2049 1308836 1 33184 0 0 0 0 99 4096 8 {1598425436 865214956} {1598425437 29223485} {1598425437 29223485} [0 0 0]}}, Source: /var/lib/docker/containers/628a5ca3a5e5037056ebabc9a016b13d4a6ebb16c44a6216beff387b65ff5c4b/628a5ca3a5e5037056ebabc9a016b13d4a6ebb16c44a6216beff387b65ff5c4b-json.log, Offset: 42864, Timestamp: 2020-08-26 07:05:03.816329242 +0000 UTC m=+187.623544858, TTL: -1ns, Type: container, Meta: map[stream:stdout], FileStateOS: 1308836-2049}

Patch:

2020-09-02T08:19:12.330Z	DEBUG	[autodiscover]	cfgfile/list.go:97	Error creating runner from config: Can only start an input when all related states are finished: {Id: native::5784858-64769, Finished: false, Fileinfo: &{mytarget2_default_test-63b37f8e982045d4bdac75d302e687e788e979ffeb857b5c1e754f019275d6e5.log 15108 416 {330272804 63734631529 0x36610e0} {64769 5784858 1 33184 0 0 0 0 15108 4096 32 {1599034103 795344632} {1599034729 330272804} {1599034729 330272804} [0 0 0]}}, Source: /var/log/containers/mytarget2_default_test-63b37f8e982045d4bdac75d302e687e788e979ffeb857b5c1e754f019275d6e5.log, Offset: 15465, Timestamp: 2020-09-02 08:19:05.456366051 +0000 UTC m=+12.261964777, TTL: -1ns, Type: container, Meta: map[], FileStateOS: 5784858-64769}

Why is it important?

To avoid confusion in logging because of a non actual ERROR is logged as an ERROR. We need to log with lower severity this kind of errors.

How to test this PR locally

Using testing steps from #20305.

Related issues

…stic#20915) (cherry picked from commit 35e6b60)

elasticmachine · 2020-09-02T14:32:04Z

Pinging @elastic/integrations-platforms (Team:Platforms)

elasticmachine · 2020-09-02T16:26:29Z

💔 Tests Failed

Expand to view the summary

Build stats

Build Cause: [Pull request #20928 updated]
Start Time: 2020-09-03T10:28:16.372+0000
Duration: 71 min 52 sec

Test stats 🧪

Test	Results
Failed	1
Passed	19442
Skipped	1854
Total	21297

Test errors

Expand to view the tests failures

Name: Build and Test / Metricbeat OSS Python Integration tests / test_remote_write – metricbeat.module.prometheus.test_prometheus.TestRemoteWrite
- Age: 1
- Duration: 63.267
- Error Details: beat.beat.TimeoutError: Timeout waiting for 'cond' to be true. Waited 60 seconds.

Steps errors

Expand to view the steps failures

Name: Mage pythonIntegTest
- Description: mage pythonIntegTest
- Duration: 35 min 50 sec
- Start Time: 2020-09-03T10:51:49.966+0000
- log
Name: Mage build unitTest
- Description: mage build unitTest
- Duration: 6 min 30 sec
- Start Time: 2020-09-03T10:59:43.132+0000
- log

Log output

Expand to view the last 100 lines of log output

[2020-09-03T11:38:40.910Z] + rm source.tgz
[2020-09-03T11:38:40.922Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats
[2020-09-03T11:38:40.949Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Lint
[2020-09-03T11:38:41.048Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-09-03T11:38:41.148Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Winlogbeat-oss
[2020-09-03T11:38:41.245Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-09-03T11:38:41.342Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/dockerlogbeat
[2020-09-03T11:38:41.432Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Journalbeat
[2020-09-03T11:38:41.514Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-09-03T11:38:41.607Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-09-03T11:38:41.692Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Packetbeat-Linux
[2020-09-03T11:38:41.779Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-09-03T11:38:41.870Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-09-03T11:38:41.954Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-09-03T11:38:42.033Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Heartbeat-oss
[2020-09-03T11:38:42.121Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Heartbeat-Windows
[2020-09-03T11:38:42.210Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-09-03T11:38:42.296Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-09-03T11:38:42.381Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-09-03T11:38:42.463Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-09-03T11:38:42.559Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-09-03T11:38:42.643Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Filebeat-Windows
[2020-09-03T11:38:42.724Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Packetbeat-Windows
[2020-09-03T11:38:42.812Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-09-03T11:38:42.913Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-09-03T11:38:42.998Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-09-03T11:38:43.080Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-09-03T11:38:43.164Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Libbeat-x-pack
[2020-09-03T11:38:43.246Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Functionbeat-Windows
[2020-09-03T11:38:43.334Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-09-03T11:38:43.429Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-09-03T11:38:43.549Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-09-03T11:38:43.630Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-09-03T11:38:43.711Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Metricbeat-Windows
[2020-09-03T11:38:43.790Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-09-03T11:38:43.872Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-09-03T11:38:43.954Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Packetbeat-Mac-OS-X
[2020-09-03T11:38:44.100Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-09-03T11:38:44.179Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-09-03T11:38:44.263Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-09-03T11:38:44.355Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-09-03T11:38:44.435Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Filebeat-x-pack
[2020-09-03T11:38:44.514Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Filebeat-oss
[2020-09-03T11:38:44.594Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-09-03T11:38:44.679Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests
[2020-09-03T11:38:44.759Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Metricbeat-OSS-Python-Integration-tests
[2020-09-03T11:38:44.843Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Libbeat-oss
[2020-09-03T11:38:44.924Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Libbeat-crosscompile
[2020-09-03T11:38:45.005Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Libbeat-stress-tests
[2020-09-03T11:38:45.093Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-09-03T11:38:45.496Z] + cat
[2020-09-03T11:38:45.496Z] + /usr/local/bin/runbld ./runbld-script --job-name elastic+beats+pull-request
[2020-09-03T11:38:45.496Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-09-03T11:38:52.071Z] runbld>>> runbld started
[2020-09-03T11:38:52.071Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-09-03T11:38:53.007Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-09-03T11:38:53.007Z] runbld>>> Matches in the system config:
[2020-09-03T11:38:53.007Z] runbld>>> - Matched ^elastic\+beats
[2020-09-03T11:38:53.007Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-09-03T11:38:54.383Z] runbld>>> Debug logging enabled.
[2020-09-03T11:38:54.383Z] runbld>>> Storing result
[2020-09-03T11:38:54.383Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-09-03T11:38:54.383Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200903113854-F53C62A7
[2020-09-03T11:38:54.642Z] runbld>>> Adding system facts.
[2020-09-03T11:38:55.580Z] runbld>>> Adding vcs info for the latest commit:  db06ab7a80b167f8a1a57cd46ce8e550f2bc6f52
[2020-09-03T11:38:55.581Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-09-03T11:38:55.581Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-09-03T11:38:55.581Z] + echo 'Processing JUnit reports with runbld...'
[2020-09-03T11:38:55.581Z] Processing JUnit reports with runbld...
[2020-09-03T11:38:55.840Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-09-03T11:38:55.840Z] runbld>>> DURATION: 21ms
[2020-09-03T11:38:55.840Z] runbld>>> STDOUT: 40 bytes
[2020-09-03T11:38:55.840Z] runbld>>> STDERR: 49 bytes
[2020-09-03T11:38:55.840Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-09-03T11:38:55.840Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-20928
[2020-09-03T11:38:56.777Z] runbld>>> Storing build metadata: 
[2020-09-03T11:38:56.777Z] runbld>>> Adding test report.
[2020-09-03T11:38:56.777Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats
[2020-09-03T11:38:57.344Z] runbld>>> Found 139 test output files
[2020-09-03T11:38:59.877Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-09-03T11:38:59.877Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-09-03T11:38:59.877Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-09-03T11:38:59.877Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-09-03T11:38:59.877Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-09-03T11:39:00.446Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-09-03T11:39:00.446Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20928/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-09-03T11:39:01.016Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 21146 Skipped: 1573
[2020-09-03T11:39:01.275Z] runbld>>> Storing result
[2020-09-03T11:39:01.275Z] runbld>>> FAILURES: 1
[2020-09-03T11:39:01.534Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-09-03T11:39:01.534Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200903113854-F53C62A7
[2020-09-03T11:39:01.793Z] runbld>>> Email notification disabled by environment variable.
[2020-09-03T11:39:01.793Z] runbld>>> Slack notification disabled by environment variable.
[2020-09-03T11:39:07.489Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-20928
[2020-09-03T11:39:07.737Z] [INFO] getVaultSecret: Getting secrets
[2020-09-03T11:39:07.810Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-09-03T11:39:08.612Z] + chmod 755 generate-build-data.sh
[2020-09-03T11:39:08.612Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20928/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20928/runs/3 FAILURE 4251979
[2020-09-03T11:39:08.612Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20928/runs/3/steps/?limit=10000 -o steps-info.json
[2020-09-03T11:39:11.095Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20928/runs/3/tests/?status=FAILED -o tests-errors.json

jsoriano · 2020-09-03T10:18:42Z

CHANGELOG.next.asciidoc

+- [Autodiscover] Handle input-not-finished errors in config reload. {pull}20915[20915]
+- Explicitly detect missing variables in autodiscover configuration, log them at the debug level. {issue}20568[20568] {pull}20898[20898]


This line is not related to this change.

😬 thanks!

CHANGELOG.next.asciidoc

[Autodiscover] Handle input-not-finished errors in config reload (ela…

e191fa4

…stic#20915) (cherry picked from commit 35e6b60)

ChrsMark added [zube]: In Review backport Team:Platforms Label for the Integrations - Platforms team labels Sep 2, 2020

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 2, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 2, 2020

ChrsMark requested a review from jsoriano September 3, 2020 07:16

jsoriano reviewed Sep 3, 2020

View reviewed changes

Update CHANGELOG.next.asciidoc

2a0fe5e

jsoriano reviewed Sep 3, 2020

View reviewed changes

CHANGELOG.next.asciidoc Show resolved Hide resolved

Merge remote-tracking branch 'upstream/7.x' into backport_20915_7.x

057780a

ChrsMark force-pushed the backport_20915_7.x branch from c83f552 to 057780a Compare September 3, 2020 10:27

jsoriano approved these changes Sep 3, 2020

View reviewed changes

ChrsMark merged commit 7e74d0e into elastic:7.x Sep 3, 2020

zube bot added [zube]: Done and removed [zube]: In Review labels Sep 3, 2020

zube bot removed the [zube]: Done label Dec 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cherry-pick #20915 to 7.x: [Autodiscover] Handle input-not-finished errors in config reload #20928

Cherry-pick #20915 to 7.x: [Autodiscover] Handle input-not-finished errors in config reload #20928

ChrsMark commented Sep 2, 2020 •

edited by zube bot

Loading

elasticmachine commented Sep 2, 2020

elasticmachine commented Sep 2, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

jsoriano Sep 3, 2020

ChrsMark Sep 3, 2020

		- [Autodiscover] Handle input-not-finished errors in config reload. {pull}20915[20915]
		- Explicitly detect missing variables in autodiscover configuration, log them at the debug level. {issue}20568[20568] {pull}20898[20898]

Cherry-pick #20915 to 7.x: [Autodiscover] Handle input-not-finished errors in config reload #20928

Cherry-pick #20915 to 7.x: [Autodiscover] Handle input-not-finished errors in config reload #20928

Conversation

ChrsMark commented Sep 2, 2020 • edited by zube bot Loading

What does this PR do?

Why is it important?

How to test this PR locally

Related issues

elasticmachine commented Sep 2, 2020

elasticmachine commented Sep 2, 2020 • edited by jenkins-beats-ci bot Loading

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Steps errors

Log output

jsoriano Sep 3, 2020

Choose a reason for hiding this comment

ChrsMark Sep 3, 2020

Choose a reason for hiding this comment

ChrsMark commented Sep 2, 2020 •

edited by zube bot

Loading

elasticmachine commented Sep 2, 2020 •

edited by jenkins-beats-ci bot

Loading