Filebeat: Update aws/cloudtrail dataset to ECS 1.8 #23911

adriansr · 2021-02-08T19:03:42Z

Updates aws/cloudtrail to map multiuser events to ECS 1.8.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related #23118

Removes usage of script processor.

elasticmachine · 2021-02-08T19:04:26Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

botelastic · 2021-02-08T19:04:40Z

This pull request doesn't have a Team:<team> label.

elasticmachine · 2021-02-08T19:52:19Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #23911 opened
Start Time: 2021-02-08T19:04:22.380+0000
Duration: 48 min 47 sec
Commit: eef321a

Test stats 🧪

Test	Results
Failed	0
Passed	6823
Skipped	1065
Total	7888

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	6823
Skipped	1065
Total	7888

elasticmachine · 2021-02-10T12:28:37Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #23911 updated
Start Time: 2021-02-10T12:25:37.672+0000
Duration: 3 min 38 sec
Commit: cba6ab3

Trends 🧪

Steps errors

Expand to view the steps failures

`Git fetch`

Took 0 min 1 sec . View more details on here
Description: git fetch https://${GIT_USERNAME}:${GIT_PASSWORD}@github.com/elastic/beats.git +refs/pull/*/head:refs/remotes/origin/pr/* > fetch.log 2>&1

`Archive the artifacts`

Took 0 min 0 sec . View more details on here
Description: fetch.log

Log output

Expand to view the last 100 lines of log output

[2021-02-10T12:26:07.606Z] Fetching upstream changes from origin
[2021-02-10T12:26:07.606Z]  > git --version # timeout=10
[2021-02-10T12:26:07.610Z]  > git --version # 'git version 2.17.1'
[2021-02-10T12:26:07.611Z]  > git config --get remote.origin.url # timeout=10
[2021-02-10T12:26:07.615Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-02-10T12:26:07.621Z]  > git fetch --tags --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10
[2021-02-10T12:26:08.191Z]  > git rev-parse current^{commit} # timeout=10
[2021-02-10T12:26:08.197Z]  > git branch -a -v --no-abbrev --contains dcbd74e89167d9e860b4232a3a4a39f943cf659e # timeout=10
[2021-02-10T12:26:08.220Z] Selected match: feature/reproduce-error-with-recurrency revision dcbd74e89167d9e860b4232a3a4a39f943cf659e
[2021-02-10T12:26:08.221Z] The recommended git tool is: git
[2021-02-10T12:26:08.221Z] using credential f6c7695a-671e-4f4f-a331-acdce44ff9ba
[2021-02-10T12:26:08.270Z]  > git rev-parse --is-inside-work-tree # timeout=10
[2021-02-10T12:26:08.276Z] Fetching changes from the remote Git repository
[2021-02-10T12:26:08.276Z]  > git config remote.origin.url git@github.com:elastic/apm-pipeline-library.git # timeout=10
[2021-02-10T12:26:08.281Z] Fetching without tags
[2021-02-10T12:26:08.281Z] Fetching upstream changes from git@github.com:elastic/apm-pipeline-library.git
[2021-02-10T12:26:08.281Z]  > git --version # timeout=10
[2021-02-10T12:26:08.286Z]  > git --version # 'git version 2.17.1'
[2021-02-10T12:26:08.286Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-02-10T12:26:08.291Z]  > git fetch --no-tags --progress -- git@github.com:elastic/apm-pipeline-library.git +refs/heads/*:refs/remotes/origin/* # timeout=10
[2021-02-10T12:26:08.932Z] Checking out Revision dcbd74e89167d9e860b4232a3a4a39f943cf659e (feature/reproduce-error-with-recurrency)
[2021-02-10T12:26:08.932Z]  > git config core.sparsecheckout # timeout=10
[2021-02-10T12:26:08.937Z]  > git checkout -f dcbd74e89167d9e860b4232a3a4a39f943cf659e # timeout=10
[2021-02-10T12:26:09.134Z] Commit message: "docs: update CHANGELOG.md"
[2021-02-10T12:26:09.756Z] Excluding src/test/ from checkout of git git@github.com:elastic/apm-pipeline-library.git so that shared library test code cannot be accessed by Pipelines.
[2021-02-10T12:26:09.756Z] To remove this log message, move the test code outside of src/. To restore the previous behavior that allowed access to files in src/test/, pass -Dorg.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.INCLUDE_SRC_TEST_IN_LIBRARIES=true to the java command used to start Jenkins.
[2021-02-10T12:26:25.902Z] Still waiting to schedule task
[2021-02-10T12:26:25.902Z] All nodes of label ‘ubuntu-18&&immutable’ are offline
[2021-02-10T12:27:19.752Z] Running on beats-ci-immutable-ubuntu-1804-1612959984534861606 in /var/lib/jenkins/workspace/Beats_beats_PR-23911
[2021-02-10T12:27:19.854Z] �[39;49m[INFO] Override default checkout�[0m
[2021-02-10T12:27:19.897Z] Sleeping for 10 sec
[2021-02-10T12:27:29.991Z] The recommended git tool is: git
[2021-02-10T12:27:32.292Z] using credential f6c7695a-671e-4f4f-a331-acdce44ff9ba
[2021-02-10T12:27:32.339Z] Wiping out workspace first.
[2021-02-10T12:27:32.368Z] Cloning the remote Git repository
[2021-02-10T12:27:32.368Z] Using shallow clone with depth 10
[2021-02-10T12:27:32.368Z] Avoid fetching tags
[2021-02-10T12:27:32.391Z] Cloning repository git@github.com:elastic/beats.git
[2021-02-10T12:27:32.435Z]  > git init /var/lib/jenkins/workspace/Beats_beats_PR-23911 # timeout=10
[2021-02-10T12:27:32.495Z] Fetching upstream changes from git@github.com:elastic/beats.git
[2021-02-10T12:27:32.495Z]  > git --version # timeout=10
[2021-02-10T12:27:32.501Z]  > git --version # 'git version 2.17.1'
[2021-02-10T12:27:32.502Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-02-10T12:27:32.527Z]  > git fetch --no-tags --progress -- git@github.com:elastic/beats.git +refs/heads/*:refs/remotes/origin/* # timeout=15
[2021-02-10T12:27:53.813Z] Cleaning workspace
[2021-02-10T12:27:53.830Z] Using shallow fetch with depth 10
[2021-02-10T12:27:53.830Z] Pruning obsolete local branches
[2021-02-10T12:27:53.786Z]  > git config remote.origin.url git@github.com:elastic/beats.git # timeout=10
[2021-02-10T12:27:53.795Z]  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2021-02-10T12:27:53.805Z]  > git config remote.origin.url git@github.com:elastic/beats.git # timeout=10
[2021-02-10T12:27:53.815Z]  > git rev-parse --verify HEAD # timeout=10
[2021-02-10T12:27:53.821Z] No valid HEAD. Skipping the resetting
[2021-02-10T12:27:53.821Z]  > git clean -fdx # timeout=10
[2021-02-10T12:27:53.834Z] Fetching upstream changes from git@github.com:elastic/beats.git
[2021-02-10T12:27:53.835Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-02-10T12:27:53.839Z]  > git fetch --no-tags --progress --prune -- git@github.com:elastic/beats.git +refs/pull/23911/head:refs/remotes/origin/PR-23911 +refs/heads/feature-ecs-1.8:refs/remotes/origin/feature-ecs-1.8 # timeout=15
[2021-02-10T12:27:54.853Z] Merging remotes/origin/feature-ecs-1.8 commit 0b27310e17b4e3c733712aae45fbc45920c42dc9 into PR head commit cba6ab3ae720fa9128bf29ac477d01eeac0150eb
[2021-02-10T12:27:54.860Z]  > git config core.sparsecheckout # timeout=10
[2021-02-10T12:27:54.865Z]  > git checkout -f cba6ab3ae720fa9128bf29ac477d01eeac0150eb # timeout=15
[2021-02-10T12:27:57.481Z] Merge succeeded, producing a2da292189cfb0ae8e0214d543c74f0cf6721d7b
[2021-02-10T12:27:57.481Z] Checking out Revision a2da292189cfb0ae8e0214d543c74f0cf6721d7b (PR-23911)
[2021-02-10T12:27:56.606Z]  > git remote # timeout=10
[2021-02-10T12:27:56.611Z]  > git config --get remote.origin.url # timeout=10
[2021-02-10T12:27:56.616Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-02-10T12:27:56.622Z]  > git merge 0b27310e17b4e3c733712aae45fbc45920c42dc9 # timeout=10
[2021-02-10T12:27:57.474Z]  > git rev-parse HEAD^{commit} # timeout=10
[2021-02-10T12:27:57.483Z]  > git config core.sparsecheckout # timeout=10
[2021-02-10T12:27:57.487Z]  > git checkout -f a2da292189cfb0ae8e0214d543c74f0cf6721d7b # timeout=15
[2021-02-10T12:28:01.103Z] Commit message: "Merge commit '0b27310e17b4e3c733712aae45fbc45920c42dc9' into HEAD"
[2021-02-10T12:28:01.106Z]  > git rev-list --no-walk eef321ae4089c8f745eeddaae18152898fdf4a6a # timeout=10
[2021-02-10T12:28:01.140Z] Cleaning workspace
[2021-02-10T12:28:01.386Z] Timeout set to expire in 3 hr 0 min
[2021-02-10T12:28:01.396Z] The timestamps step is unnecessary when timestamps are enabled for all Pipeline builds.
[2021-02-10T12:28:01.530Z] [INFO] Number of builds to be searched 10
[2021-02-10T12:28:02.157Z] [INFO] 'shallow' is forced to be disabled when running on PullRequests
[2021-02-10T12:28:02.167Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-23911/src/github.com/elastic/beats
[2021-02-10T12:28:02.179Z] [INFO] gitCheckout: Checkout SCM PR-23911 with default customisation from the Item.
[2021-02-10T12:28:02.198Z] [INFO] Override default checkout
[2021-02-10T12:28:02.225Z] Sleeping for 10 sec
[2021-02-10T12:28:01.141Z]  > git rev-parse --verify HEAD # timeout=10
[2021-02-10T12:28:01.145Z] Resetting working tree
[2021-02-10T12:28:01.145Z]  > git reset --hard # timeout=10
[2021-02-10T12:28:01.237Z]  > git clean -fdx # timeout=10
[2021-02-10T12:28:12.409Z] Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD
[2021-02-10T12:28:13.078Z] + git fetch https://****:****@github.com/elastic/beats.git +refs/pull/*/head:refs/remotes/origin/pr/*
[2021-02-10T12:28:13.130Z] [WARN] gitCmd failed, further details in the archived file 'fetch.log'
[2021-02-10T12:28:13.194Z] Archiving artifacts
[2021-02-10T12:28:13.517Z] Stage "Lint" skipped due to earlier failure(s)
[2021-02-10T12:28:13.540Z] Stage "Build&Test" skipped due to earlier failure(s)
[2021-02-10T12:28:13.562Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-02-10T12:28:13.599Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-23911/src/github.com/elastic/beats
[2021-02-10T12:28:14.634Z] Running on worker-395930 in /var/lib/jenkins/workspace/Beats_beats_PR-23911
[2021-02-10T12:28:14.702Z] [INFO] getVaultSecret: Getting secrets
[2021-02-10T12:28:14.783Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-02-10T12:28:16.800Z] + chmod 755 generate-build-data.sh
[2021-02-10T12:28:16.800Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23911/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23911/runs/2 FAILURE 157723
[2021-02-10T12:28:16.800Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23911/runs/2/steps/?limit=10000 -o steps-info.json
[2021-02-10T12:28:17.502Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23911/runs/2/tests/?status=FAILED -o tests-errors.json
[2021-02-10T12:28:18.203Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2021-02-10T12:28:19.657Z] Retry 2/3 exited 22, retrying in 2 seconds...

❕ Flaky test report

No test was executed to be analysed.

Incorporates ECS 1.8 changes from the following PRs: Support host.type field in add_host_metadata processor and Auditbeat's system/host #23513 Winlogbeat #23563 Auditbeat auditd #23594 Journalbeat #23737 Packetbeat #23783 Filebeat: auditd #23723 cisco #23819 cef #23832 crowdstrike falcon #23875 fortinet firewall #23902 microsoft #23897 elasticsearch/audit #24000 Gsuite/Workspace #23709 o365 #23896 zoom #23904 okta #23929 aws/cloudtrail #23911 aws/s3access #23920 azure #23927 juniper/srx #23936 panw #23931 sophos/xg #23967 system/auth #23961 mysqlenterprise #23978 zeek #23847 Make all Beats and modules report ECS 1.8.0 #23992 Closes #23118 Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

Incorporates ECS 1.8 changes from the following PRs: Support host.type field in add_host_metadata processor and Auditbeat's system/host elastic#23513 Winlogbeat elastic#23563 Auditbeat auditd elastic#23594 Journalbeat elastic#23737 Packetbeat elastic#23783 Filebeat: auditd elastic#23723 cisco elastic#23819 cef elastic#23832 crowdstrike falcon elastic#23875 fortinet firewall elastic#23902 microsoft elastic#23897 elasticsearch/audit elastic#24000 Gsuite/Workspace elastic#23709 o365 elastic#23896 zoom elastic#23904 okta elastic#23929 aws/cloudtrail elastic#23911 aws/s3access elastic#23920 azure elastic#23927 juniper/srx elastic#23936 panw elastic#23931 sophos/xg elastic#23967 system/auth elastic#23961 mysqlenterprise elastic#23978 zeek elastic#23847 Make all Beats and modules report ECS 1.8.0 elastic#23992 Closes elastic#23118 Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> (cherry picked from commit 048c3cc)

Incorporates ECS 1.8 changes from the following PRs: Support host.type field in add_host_metadata processor and Auditbeat's system/host #23513 Winlogbeat #23563 Auditbeat auditd #23594 Journalbeat #23737 Packetbeat #23783 Filebeat: auditd #23723 cisco #23819 cef #23832 crowdstrike falcon #23875 fortinet firewall #23902 microsoft #23897 elasticsearch/audit #24000 Gsuite/Workspace #23709 o365 #23896 zoom #23904 okta #23929 aws/cloudtrail #23911 aws/s3access #23920 azure #23927 juniper/srx #23936 panw #23931 sophos/xg #23967 system/auth #23961 mysqlenterprise #23978 zeek #23847 Make all Beats and modules report ECS 1.8.0 #23992 Closes #23118 Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> (cherry picked from commit 048c3cc)

Updates aws/cloudtrail to map multiuser events to ECS 1.8.

adriansr added 18 commits February 8, 2021 16:38

Use append processor for related.user

57cf476

Removes usage of script processor.

Enrich user fields for AddUserToGroup

9f8dd43

CreateAccessKey + CreateGroup

c7e0853

CreateUser

edc5997

DeleteAccessKey

4508f2d

DeleteGroup

f3e7385

DeleteSSHPublicKey / DeleteUser

4559ba9

EnableMFADevice

23c3300

RemoveUserFromGroup

7d29881

Update AccessKey/Group/Login/SSH Public Key

0628eb6

UpdateUser / UploadSSHPublicKey

66b04d2

Merge group.name processors

18febdb

Refactor user.id

39ae0db

Bigger refactor

ea1c63c

Append main user to related.user

345d714

Add user performing the action to related.user

5307253

Update ECS version field

c0d8372

Changelog entry

eef321a

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 8, 2021

adriansr changed the title ~~Fb aws ct ecs 1.8~~ Filebeat: Update aws/cloudtrail dataset to ECS 1.8 Feb 8, 2021

adriansr added the Team:Security-External Integrations label Feb 8, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 8, 2021

adriansr added ecs enhancement needs_team Indicates that the issue/PR needs a Team:* label review labels Feb 8, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 8, 2021

adriansr requested a review from leehinman February 8, 2021 19:04

adriansr requested a review from marc-gr February 8, 2021 19:04

adriansr mentioned this pull request Feb 8, 2021

[ECS] Upgrade modules to 1.8 #23118

Closed

89 tasks

marc-gr approved these changes Feb 9, 2021

View reviewed changes

Merge branch 'feature-ecs-1.8' into fb-aws-ct-ecs-1.8

cba6ab3

adriansr merged commit 0b27310 into elastic:feature-ecs-1.8 Feb 10, 2021

adriansr mentioned this pull request Feb 12, 2021

Update Beats to ECS 1.8.0 #23465

Merged

28 tasks

adriansr mentioned this pull request Feb 16, 2021

Cherry-pick #23465 to 7.x: Update Beats to ECS 1.8.0 #24072

Merged

28 tasks

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

Filebeat: Update aws/cloudtrail dataset to ECS 1.8 (elastic#23911)

eb15834

Updates aws/cloudtrail to map multiuser events to ECS 1.8.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Filebeat: Update aws/cloudtrail dataset to ECS 1.8 #23911

Filebeat: Update aws/cloudtrail dataset to ECS 1.8 #23911

adriansr commented Feb 8, 2021 •

edited

elasticmachine commented Feb 8, 2021

botelastic bot commented Feb 8, 2021

elasticmachine commented Feb 8, 2021

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

elasticmachine commented Feb 10, 2021

Build stats

Trends 🧪

`Git fetch`

`Archive the artifacts`

Filebeat: Update aws/cloudtrail dataset to ECS 1.8 #23911

Filebeat: Update aws/cloudtrail dataset to ECS 1.8 #23911

Conversation

adriansr commented Feb 8, 2021 • edited

Checklist

elasticmachine commented Feb 8, 2021

botelastic bot commented Feb 8, 2021

elasticmachine commented Feb 8, 2021

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented Feb 10, 2021

💔 Build Failed

Build stats

Trends 🧪

Steps errors

Git fetch

Archive the artifacts

Log output

❕ Flaky test report

adriansr commented Feb 8, 2021 •

edited

`Git fetch`

`Archive the artifacts`