Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #19376 to 7.x: [Filebeat] Improve ECS categorization field mappings for azure module #19737

Merged
merged 1 commit into from
Jul 8, 2020

Conversation

leehinman
Copy link
Contributor

@leehinman leehinman commented Jul 8, 2020

Cherry-pick of PR #19376 to 7.x branch. Original message:

What does this PR do?

Improve ECS categorization field mappings for azure module.
Specifically:

  • activitylogs
    • add azure.activitylogs.result_type
    • set default_field: false
    • populate event.outcome with allowed values
    • set event.action
    • populate event.category with allowed values
    • set event.kind
    • set event.type
    • add support tickets example
    • add geoip for source.ip
    • add AS info for source.ip
    • add user.name
    • add user.full_name
    • add user.domain
  • auditlogs
    • set default_field: false
    • add azure.auditlogs.category
    • populate event.outcome with allowed values
    • set event.action
    • set event.kind
  • signinlogs
    • set default_field: false
    • set event.action
    • populate event.category with allowed values
    • set event.type
    • populate event.outcome with allowed values
    • add azure.signinlogs.category
    • add azure.signinlogs.result_type
    • set user.name
    • set user.domain
    • set user.full_name
    • set user.id
    • add geoip for source.ip
    • add AS info for source.ip

Why is it important?

ECS categorization fields allow cross correlation between filesets.

Checklist

  • My code follows the style guidelines of this project
    - [ ] I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

TESTING_FILEBEAT_MODULES=azure mage -v pythonIntegTest

Related issues

  • Closes 16155

…elastic#19376)

* Improve ECS categorization field mappings in azure module

- activitylogs
  + convert pipeline to yml
  + add azure.activitylogs.result_type
  + set default_field: false
  + populate event.outcome with allowed values
  + set event.action
  + populate event.category with allowed values
  + set event.kind
  + set event.type
  + add support tickets example
  + add geoip for source.ip
  + add AS info for source.ip
  + add user.name
  + add user.full_name
  + add user.domain
  + update dashboards
- auditlogs
  + convert pipeline to yml
  + set default_field: false
  + add azure.auditlogs.category
  + populate event.outcome with allowed values
  + set event.action
  + set event.kind
  + update dashboards
- signinlogs
  + convert pipeline to yml
  + set default_field: false
  + set event.action
  + populate event.category with allowed values
  + set event.type
  + populate event.outcome with allowed values
  + add azure.signinlogs.category
  + add azure.signinlogs.result_type
  + set user.name
  + set user.domain
  + set user.full_name
  + set user.id
  + add geoip for source.ip
  + add AS info for source.ip
  + update dashboards

Closes elastic#16155

(cherry picked from commit 00a274e)
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 8, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 8, 2020
@elasticmachine
Copy link
Collaborator

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19737 opened]

  • Start Time: 2020-07-08T13:29:48.304+0000

  • Duration: 124 min 7 sec

Test stats 🧪

Test Results
Failed 1
Passed 3537
Skipped 677
Total 4215

Test errors

Expand to view the tests failures

  • Name: Build and Test / Filebeat Mac OS X / test_tail_files – test_crawler.Test

    • Age: 1
    • Duration: 6.276
    • Error Details: Timeout waiting for 'cond' to be true. Waited 5 seconds.

Steps errors

Expand to view the steps failures

  • Name: Mage build unitTest

    • Description: mage build unitTest

    • Duration: 8 min 45 sec

    • Start Time: 2020-07-08T13:54:24.040+0000

    • log

  • Name: Recursively delete the current directory from the workspace

    • Description: script returned exit code 1

    • Duration: 0 min 11 sec

    • Start Time: 2020-07-08T14:04:48.831+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-07-08T15:33:27.159Z] 		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
[2020-07-08T15:33:27.159Z] 		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
[2020-07-08T15:33:27.159Z] 		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-07-08T15:33:27.159Z] 		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-07-08T15:33:27.159Z] 		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-07-08T15:33:27.159Z] 		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-07-08T15:33:27.159Z] 		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-07-08T15:33:27.159Z] Caused: hudson.FilePath$TunneledInterruptedException
[2020-07-08T15:33:27.159Z] 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3075)
[2020-07-08T15:33:27.159Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
[2020-07-08T15:33:27.159Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
[2020-07-08T15:33:27.159Z] 	at hudson.remoting.Request$2.run(Request.java:369)
[2020-07-08T15:33:27.159Z] 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
[2020-07-08T15:33:27.159Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-07-08T15:33:27.159Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-07-08T15:33:27.159Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-07-08T15:33:27.159Z] 	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
[2020-07-08T15:33:27.159Z] Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
[2020-07-08T15:33:27.159Z] 	at hudson.FilePath.act(FilePath.java:1071)
[2020-07-08T15:33:27.159Z] 	at hudson.FilePath.act(FilePath.java:1058)
[2020-07-08T15:33:27.159Z] 	at hudson.FilePath.validateAntFileMask(FilePath.java:2684)
[2020-07-08T15:33:27.159Z] 	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:265)
[2020-07-08T15:33:27.159Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
[2020-07-08T15:33:27.159Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
[2020-07-08T15:33:27.159Z] 	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-07-08T15:33:27.159Z] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-07-08T15:33:27.159Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-07-08T15:33:27.159Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-07-08T15:33:27.159Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-07-08T15:33:27.159Z] 	at java.lang.Thread.run(Thread.java:748)
[2020-07-08T15:33:27.159Z] No artifacts found that match the file pattern "**/build/TEST*.out". Configuration error?
[2020-07-08T15:33:27.670Z] + curl -sSLo codecov https://codecov.io/bash
[2020-07-08T15:33:27.697Z] Failed in branch Filebeat x-pack
[2020-07-08T15:33:27.932Z] + FILE=auditbeat/build/coverage/full.cov
[2020-07-08T15:33:27.932Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-07-08T15:33:27.932Z] + FILE=filebeat/build/coverage/full.cov
[2020-07-08T15:33:27.932Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-07-08T15:33:27.932Z] + FILE=heartbeat/build/coverage/full.cov
[2020-07-08T15:33:27.932Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-07-08T15:33:27.932Z] + FILE=libbeat/build/coverage/full.cov
[2020-07-08T15:33:27.932Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-07-08T15:33:27.932Z] + FILE=metricbeat/build/coverage/full.cov
[2020-07-08T15:33:27.932Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-07-08T15:33:27.932Z] + FILE=packetbeat/build/coverage/full.cov
[2020-07-08T15:33:27.932Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-07-08T15:33:27.932Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-07-08T15:33:27.932Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-07-08T15:33:27.932Z] + FILE=journalbeat/build/coverage/full.cov
[2020-07-08T15:33:27.932Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-07-08T15:33:29.407Z] Failed in branch Filebeat oss
[2020-07-08T15:33:31.165Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19737/src/github.com/elastic/beats
[2020-07-08T15:33:31.514Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-08T15:33:31.540Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19737/src/github.com/elastic/beats/Lint
[2020-07-08T15:33:31.708Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19737/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-07-08T15:33:31.867Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19737/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-07-08T15:33:32.028Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19737/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-07-08T15:33:32.195Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19737/src/github.com/elastic/beats/Filebeat-Windows
[2020-07-08T15:33:32.360Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19737/src/github.com/elastic/beats/Filebeat-x-pack
[2020-07-08T15:33:32.528Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19737/src/github.com/elastic/beats/Filebeat-oss
[2020-07-08T15:33:32.994Z] + cat
[2020-07-08T15:33:32.994Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-08T15:33:32.994Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-08T15:33:39.598Z] runbld>>> runbld started
[2020-07-08T15:33:39.598Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-08T15:33:41.521Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-19737' in order of occurrence in the config (last value wins).
[2020-07-08T15:33:42.913Z] runbld>>> Debug logging enabled.
[2020-07-08T15:33:42.913Z] runbld>>> Storing result
[2020-07-08T15:33:43.180Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-08T15:33:43.180Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200708153342-78325299
[2020-07-08T15:33:43.180Z] runbld>>> Adding system facts.
[2020-07-08T15:33:44.574Z] runbld>>> Adding vcs info for the latest commit:  877cb9e0779d4a21a73acd4c6a9ed6263b933335
[2020-07-08T15:33:44.574Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-08T15:33:44.574Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-08T15:33:44.574Z] Processing JUnit reports with runbld...
[2020-07-08T15:33:44.574Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-08T15:33:44.840Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-08T15:33:44.840Z] runbld>>> DURATION: 19ms
[2020-07-08T15:33:44.840Z] runbld>>> STDOUT: 40 bytes
[2020-07-08T15:33:44.840Z] runbld>>> STDERR: 49 bytes
[2020-07-08T15:33:44.840Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-08T15:33:44.840Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19737/src/github.com/elastic/beats
[2020-07-08T15:33:45.800Z] runbld>>> Storing build metadata: 
[2020-07-08T15:33:45.800Z] runbld>>> Adding test report.
[2020-07-08T15:33:45.800Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19737/src/github.com/elastic/beats
[2020-07-08T15:33:46.376Z] runbld>>> Found 11 test output files
[2020-07-08T15:33:47.825Z] runbld>>> Test output logs contained: Errors: 1 Failures: 0 Tests: 4215 Skipped: 653
[2020-07-08T15:33:47.825Z] runbld>>> Storing result
[2020-07-08T15:33:47.825Z] runbld>>> FAILURES: 1
[2020-07-08T15:33:48.090Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-08T15:33:48.090Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200708153342-78325299
[2020-07-08T15:33:48.351Z] runbld>>> Email notification disabled by environment variable.
[2020-07-08T15:33:48.351Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-08T15:33:54.450Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19737
[2020-07-08T15:33:54.754Z] [INFO] getVaultSecret: Getting secrets
[2020-07-08T15:33:54.847Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-08T15:33:55.933Z] + chmod 755 generate-build-data.sh
[2020-07-08T15:33:55.933Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19737/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19737/runs/1 FAILURE 7447355
[2020-07-08T15:33:55.933Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19737/runs/1/steps/?limit=10000 -o steps-info.json
[2020-07-08T15:33:56.844Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19737/runs/1/tests/?status=FAILED -o tests-errors.json
[2020-07-08T15:33:57.095Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19737/runs/1/log/ -o pipeline-log.txt

@leehinman leehinman merged commit 29ef3bd into elastic:7.x Jul 8, 2020
@leehinman leehinman deleted the backport_19376_7.x branch October 5, 2020 19:06
@zube zube bot removed the [zube]: Done label Oct 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants