[Metricbeat] Add static mapping for metricsets in aws module #17650

kaiyan-sheng · 2020-04-09T18:22:59Z

What does this PR do?

This PR adds static mappings for light weight metricsets in aws module: billing, ebs, elb, lambda, natgateway, sns, transitgateway, usage, vpn.

Why is it important?

To avoid having breaking non TSVB visualisations when there no data in ES to create the respective fields with the dynamic mapping.

This PR is an extension for #17614.

elasticmachine · 2020-04-09T18:23:23Z

Pinging @elastic/integrations-platforms (Team:Platforms)

x-pack/metricbeat/module/aws/vpn/_meta/fields.yml

ChrsMark

Overall looks good. Just left some notes. Didn't test them, but maybe we need to so as to verify these fields' types, since some of them, like sums, could be tricky, wdyt?

elasticmachine · 2020-04-28T20:01:59Z

💔 Build Failed

Expand to view the summary

Build stats

Build Cause: [Pull request [Metricbeat] Add static mapping for metricsets in aws module #17650 updated]
Start Time: 2020-04-29T02:27:13.923+0000
Duration: 57 min 58 sec (3417726)
Commit: 6723a31

Test stats 🧪

Test	Results
Failed	1
Passed	2852
Skipped	757
Total	3610

Test errors

Expand to view the tests failures

Name: Build and Test / Metricbeat x-pack / test_server – test_statsd.Test
- Status: FAILED
- Age: 1
- Duration: 5.586
- Error Details: 'ascii' codec can't decode byte 0xe2 in position 701907: ordinal not in range(128)

Steps errors

Expand to view the steps failures

Name: Mage build test
- Description: mage build test
- Result: FAILURE
- Duration: 8 min 24 sec<
- Start Time: 2020-04-29T02:50:50.647+0000

Log output

Expand to view the last 100 lines of log output

[2020-04-29T03:24:04.106Z] [success] 0.23% test_haproxy.HaproxyTest_3.test_stat_http: 0.5367s
[2020-04-29T03:24:04.106Z] [success] 0.23% test_system.Test.test_cpu_ticks_option: 0.5348s
[2020-04-29T03:24:04.106Z] [success] 0.23% test_processors.Test.test_drop_fields: 0.5337s
[2020-04-29T03:24:04.106Z] [success] 0.23% test_haproxy.HaproxyTest_1.test_stat_http: 0.5332s
[2020-04-29T03:24:04.106Z] [success] 0.23% test_haproxy.HaproxyTest_1.test_stat_https: 0.5330s
[2020-04-29T03:24:04.106Z] [success] 0.23% test_autodiscover_jolokia.Test.test_jolokia_jmx: 0.5325s
[2020-04-29T03:24:04.106Z] [success] 0.23% test_haproxy.HaproxyTest_4.test_stat_https: 0.5315s
[2020-04-29T03:24:04.106Z] [success] 0.23% test_processors.Test.test_dropfields_with_condition: 0.5293s
[2020-04-29T03:24:04.106Z] [success] 0.23% test_haproxy.HaproxyTest_2.test_stat_http: 0.5284s
[2020-04-29T03:24:04.106Z] [success] 0.23% test_system.Test.test_process_metricbeat: 0.5254s
[2020-04-29T03:24:04.106Z] [success] 0.23% test_haproxy.HaproxyTest_3.test_stat_https: 0.5250s
[2020-04-29T03:24:04.106Z] [success] 0.23% test_processors.Test.test_contradictory_multiple_actions: 0.5239s
[2020-04-29T03:24:04.106Z] [success] 0.23% test_processors.Test.test_rename_field: 0.5230s
[2020-04-29T03:24:04.106Z] [success] 0.23% test_processors.Test.test_dropevent_with_complex_condition: 0.5224s
[2020-04-29T03:24:04.106Z] [success] 0.23% test_processors.Test.test_multiple_actions: 0.5220s
[2020-04-29T03:24:04.106Z] [success] 0.23% test_processors.Test.test_include_fields: 0.5214s
[2020-04-29T03:24:04.106Z] [success] 0.22% test_prometheus.Test.test_stats: 0.4983s
[2020-04-29T03:24:04.106Z] [success] 0.20% test_docker.Test.test_image_fields: 0.4720s
[2020-04-29T03:24:04.106Z] [success] 0.20% test_base.Test.test_start_stop: 0.4687s
[2020-04-29T03:24:04.106Z] [success] 0.17% test_cmd.TestCommands.test_modules_test_error: 0.4003s
[2020-04-29T03:24:04.106Z] [success] 0.17% test_cmd.TestCommands.test_modules_test: 0.3883s
[2020-04-29T03:24:04.106Z] [success] 0.16% test_cmd.TestCommands.test_modules_test_filter_no_result: 0.3770s
[2020-04-29T03:24:04.106Z] [success] 0.16% test_cmd.TestCommands.test_modules_test_filter: 0.3735s
[2020-04-29T03:24:04.107Z] [success] 0.06% test_reload.Test.test_wrong_module_no_reload: 0.1302s
[2020-04-29T03:24:04.107Z] ----------------------------------------------------------------------
[2020-04-29T03:24:04.107Z] Ran 221 tests in 1708.881s
[2020-04-29T03:24:04.107Z] 
[2020-04-29T03:24:04.107Z] OK (SKIP=19)
[2020-04-29T03:24:04.107Z] >> python test: Integration Testing Complete
[2020-04-29T03:24:04.368Z] >> Stopping Docker test environment...
[2020-04-29T03:24:05.365Z] Recording test results
[2020-04-29T03:24:08.138Z] Archiving artifacts
[2020-04-29T03:24:08.722Z] java.lang.InterruptedException: no matches found within 10000
[2020-04-29T03:24:08.722Z] 	at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2826)
[2020-04-29T03:24:08.722Z] 	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2705)
[2020-04-29T03:24:08.722Z] 	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2686)
[2020-04-29T03:24:08.723Z] 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3073)
[2020-04-29T03:24:08.723Z] Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from beats-ci-immutable-ubuntu-1604-1588128508095566002.c.elastic-ci-prod.internal/10.224.0.158:44156
[2020-04-29T03:24:08.723Z] 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1788)
[2020-04-29T03:24:08.723Z] 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
[2020-04-29T03:24:08.723Z] 		at hudson.remoting.Channel.call(Channel.java:998)
[2020-04-29T03:24:08.723Z] 		at hudson.FilePath.act(FilePath.java:1069)
[2020-04-29T03:24:08.723Z] 		at hudson.FilePath.act(FilePath.java:1058)
[2020-04-29T03:24:08.723Z] 		at hudson.FilePath.validateAntFileMask(FilePath.java:2684)
[2020-04-29T03:24:08.723Z] 		at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:265)
[2020-04-29T03:24:08.723Z] 		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
[2020-04-29T03:24:08.723Z] 		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
[2020-04-29T03:24:08.723Z] 		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-04-29T03:24:08.723Z] 		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-04-29T03:24:08.723Z] 		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-04-29T03:24:08.723Z] 		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-04-29T03:24:08.723Z] 		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-04-29T03:24:08.723Z] Caused: hudson.FilePath$TunneledInterruptedException
[2020-04-29T03:24:08.723Z] 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3075)
[2020-04-29T03:24:08.723Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
[2020-04-29T03:24:08.723Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
[2020-04-29T03:24:08.723Z] 	at hudson.remoting.Request$2.run(Request.java:369)
[2020-04-29T03:24:08.723Z] 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
[2020-04-29T03:24:08.723Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-04-29T03:24:08.723Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-04-29T03:24:08.723Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-04-29T03:24:08.723Z] 	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
[2020-04-29T03:24:08.723Z] Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
[2020-04-29T03:24:08.723Z] 	at hudson.FilePath.act(FilePath.java:1071)
[2020-04-29T03:24:08.723Z] 	at hudson.FilePath.act(FilePath.java:1058)
[2020-04-29T03:24:08.723Z] 	at hudson.FilePath.validateAntFileMask(FilePath.java:2684)
[2020-04-29T03:24:08.723Z] 	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:265)
[2020-04-29T03:24:08.723Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
[2020-04-29T03:24:08.723Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
[2020-04-29T03:24:08.723Z] 	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-04-29T03:24:08.723Z] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-04-29T03:24:08.723Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-04-29T03:24:08.723Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-04-29T03:24:08.723Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-04-29T03:24:08.723Z] 	at java.lang.Thread.run(Thread.java:748)
[2020-04-29T03:24:08.723Z] No artifacts found that match the file pattern "**/build/TEST*.out". Configuration error?
[2020-04-29T03:24:09.077Z] + curl -sSLo codecov https://codecov.io/bash
[2020-04-29T03:24:09.337Z] + FILE=auditbeat/build/coverage/full.cov
[2020-04-29T03:24:09.337Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-04-29T03:24:09.337Z] + FILE=filebeat/build/coverage/full.cov
[2020-04-29T03:24:09.337Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-04-29T03:24:09.337Z] + FILE=heartbeat/build/coverage/full.cov
[2020-04-29T03:24:09.337Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-04-29T03:24:09.337Z] + FILE=libbeat/build/coverage/full.cov
[2020-04-29T03:24:09.337Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-04-29T03:24:09.337Z] + FILE=metricbeat/build/coverage/full.cov
[2020-04-29T03:24:09.337Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-04-29T03:24:09.337Z] + FILE=packetbeat/build/coverage/full.cov
[2020-04-29T03:24:09.337Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-04-29T03:24:09.337Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-04-29T03:24:09.337Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-04-29T03:24:09.337Z] + FILE=journalbeat/build/coverage/full.cov
[2020-04-29T03:24:09.337Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-04-29T03:24:10.980Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17650
[2020-04-29T03:24:11.199Z] [INFO] getVaultSecret: Getting secrets
[2020-04-29T03:24:11.250Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-04-29T03:24:11.912Z] + chmod 755 generate-build-data.sh
[2020-04-29T03:24:11.912Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-17650/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-17650/runs/17 FAILURE 3417726
[2020-04-29T03:24:12.824Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-17650/runs/17/steps/?limit=10000 -o steps-info.json
[2020-04-29T03:24:13.074Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-17650/runs/17/tests/?status=FAILED -o tests-errors.json

kaiyan-sheng · 2020-04-29T17:24:51Z

CI failure is in statsd module:

Traceback (most recent call last):
  File "/usr/lib/python3.5/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/usr/lib/python3.5/unittest/case.py", line 600, in run
    testMethod()
  File "/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17650/src/github.com/elastic/beats/x-pack/metricbeat/module/statsd/test_statsd.py", line 46, in test_server
    self.assert_fields_are_documented(evt)
  File "/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17650/src/github.com/elastic/beats/libbeat/tests/system/beat/beat.py", line 688, in assert_fields_are_documented
    expected_fields, dict_fields, aliases = self.load_fields()
  File "/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17650/src/github.com/elastic/beats/libbeat/tests/system/beat/beat.py", line 596, in load_fields
    content += f.read()
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 701907: ordinal not in range(128)

…sets in aws module (#18101) * [Metricbeat] Add static mapping for metricsets in aws module (#17650) * Add static mapping for metricsets in aws module (cherry picked from commit 1fbe167) * add changelog

Add static mapping for metricsets in aws module

dafb664

kaiyan-sheng self-assigned this Apr 9, 2020

kaiyan-sheng added Metricbeat Metricbeat Team:Platforms Label for the Integrations - Platforms team labels Apr 9, 2020

kaiyan-sheng added the needs_backport PR is waiting to be backported to other branches. label Apr 9, 2020

kaiyan-sheng requested a review from ChrsMark April 9, 2020 20:01

andresrc added [zube]: Inbox [zube]: In Review and removed [zube]: Inbox labels Apr 13, 2020

zube bot added [zube]: Inbox [zube]: In Review and removed [zube]: In Review [zube]: Inbox labels Apr 13, 2020

rerun mage update

a571415

ChrsMark reviewed Apr 15, 2020

View reviewed changes

x-pack/metricbeat/module/aws/vpn/_meta/fields.yml Outdated Show resolved Hide resolved

ChrsMark reviewed Apr 15, 2020

View reviewed changes

x-pack/metricbeat/module/aws/vpn/_meta/fields.yml Show resolved Hide resolved

ChrsMark reviewed Apr 15, 2020

View reviewed changes

kaiyan-sheng added 5 commits April 15, 2020 20:59

Change TunnelState.avg from integer to double

ab8d097

Merge remote-tracking branch 'upstream/master' into static_mapping

4e99fe4

Merge remote-tracking branch 'upstream/master' into static_mapping

3771cdd

Merge remote-tracking branch 'upstream/master' into static_mapping

7962014

Merge remote-tracking branch 'upstream/master' into static_mapping

ee3567e

ChrsMark approved these changes Apr 22, 2020

View reviewed changes

kaiyan-sheng added 5 commits April 22, 2020 14:49

add missed statistic suffix

40eb4d4

Merge remote-tracking branch 'upstream/master' into static_mapping

ceb2e3a

Merge remote-tracking branch 'upstream/master' into static_mapping

fc62d23

add missing field into usage/_meta/fields.yml

bc4de29

Merge remote-tracking branch 'upstream/master' into static_mapping

aef7f62

kaiyan-sheng added 2 commits April 28, 2020 20:12

Merge remote-tracking branch 'upstream/master' into static_mapping

753866c

run mage fmt update

6723a31

kaiyan-sheng merged commit 1fbe167 into elastic:master Apr 29, 2020

kaiyan-sheng deleted the static_mapping branch April 29, 2020 17:25

zube bot added [zube]: Done and removed [zube]: In Review labels Apr 29, 2020

kaiyan-sheng mentioned this pull request Apr 29, 2020

Cherry-pick #17650 to 7.x: [Metricbeat] Add static mapping for metricsets in aws module #18101

Merged

kaiyan-sheng added v7.8.0 and removed needs_backport PR is waiting to be backported to other branches. labels Apr 29, 2020

kaiyan-sheng added a commit that referenced this pull request Apr 29, 2020

add changelog for PR #17650 and PR #17614 (#18102)

1d85b30

jsoriano mentioned this pull request Apr 30, 2020

Fix fields validation in non utf-8 environments #18130

Merged

andresrc removed the [zube]: Done label May 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metricbeat] Add static mapping for metricsets in aws module #17650

[Metricbeat] Add static mapping for metricsets in aws module #17650

kaiyan-sheng commented Apr 9, 2020

elasticmachine commented Apr 9, 2020

ChrsMark left a comment

elasticmachine commented Apr 28, 2020 •

edited

Build stats

Test stats 🧪

kaiyan-sheng commented Apr 29, 2020

[Metricbeat] Add static mapping for metricsets in aws module #17650

[Metricbeat] Add static mapping for metricsets in aws module #17650

Conversation

kaiyan-sheng commented Apr 9, 2020

What does this PR do?

Why is it important?

elasticmachine commented Apr 9, 2020

ChrsMark left a comment

Choose a reason for hiding this comment

elasticmachine commented Apr 28, 2020 • edited

💔 Build Failed

Build stats

Test stats 🧪

Test errors

Steps errors

Log output

kaiyan-sheng commented Apr 29, 2020

elasticmachine commented Apr 28, 2020 •

edited