Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[box] New Integration #3677

Merged
merged 141 commits into from
Sep 14, 2022
Merged

[box] New Integration #3677

merged 141 commits into from
Sep 14, 2022

Conversation

djptek
Copy link
Contributor

@djptek djptek commented Jul 11, 2022

  • Enhancement

What does this PR do?

Adds integration for BOX events.

Checklist

  • [Y] I have reviewed tips for building integrations and this pull request is aligned with them.
  • [Y] I have verified that all data streams collect metrics or logs.
  • [Y] I have added an entry to my package's changelog.yml file.
  • [Y] I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

- [] Complete system tests

Add Shield Alerts

  • Suspicious locations

  • Suspicious sessions

  • Anomalous downloads

  • Malicious content

  • Update Dashboards

  • Update Screenshots

  • Update docs template _dev/build/docs/README.md

How to test this PR locally

Use elastic-package test to run the tests.

The pipeline tests for the events were created by ingesting data directly from the Box API using Filebeat HTTPJSON with no processors applied, then cut-and-paste of the message field for each resultant Elasticsearch Document to the relevant inputs.

An additional test was added using fake events, one corresponding to each of all documented box event types. This test is intended to exercise the mapping of the box event_type field value to the corresponding values of ECS fields event.category and event.type.

The simulated shield alerts were created by:

  • composing the alerts in accordance with the Box Shield Alerts documentation and publishing these via a web proxy
  • ingest of all alerts using Filebeat running 4 distinct configurations, each based on the relevant httpjson.yml.hbs for the relevant data stream, to ensure that the distinct response.split.split and drop_event pipeline processors were applied to denormalise alerts to constituent documents and select these for routing to the relevant data-stream pipeline

Box Shield Alerts should be considered as Beta until such time as access to a suitable target system with alerts relating to suspected malign activity is available.

To run an end-to-end test with live data, you will need an account on Box, a developer account is fine, don't need enterprise.

Add some test data to that and copy/delete/upload/download/view the data to create some events.

Have your box credentials available

  • client.id
  • client.secret
  • subject_id

(do not store these in Github)

see the documentation of the box_events Integration for details.

Start the stack using elastic-package stack up -v -d --version 8.3.0

Add and configure your box_events Integration using your Box credentials.

image

Ensure to scroll down to

image

and assign the policy to Existing hosts-> Agent Policy -> Elastic-Agent

Generate some events in Box by Upload/Download/Delete/Visualisation of files then go to the [Logs Box Events Integration] Events dashboard to verify that events were ingested and are visible.

To add Simulated Shield Events, you can cut and paste the contents of this _bulk request into Dev Tools and send the request to the stack. This will ingest a selection of alerts to validate the [Logs Box Events Integration] Box Shield Alerts dashboard, you will need to set the time picker to July 2022.

The Simulated Shield Alerts were created by manually appending raw agent etc. fields created during the end-to-end tests for each alert's pipeline test document(s), and then rectifying the data-stream for each.

Related issues

Screenshots

For Kibana UI options see test procedure, above.

Dashboards

Showing Elastic Stack 7.17.0 compatible dashboards

Box Events

image

Box Shield Alerts

image

image

image

Test Results

% elastic-package test                                                                 (box)integrations
Run test suite for the package
Run asset tests for the package
--- Test results for package: box_events - START ---
╭────────────┬─────────────────────────────┬───────────┬─────────────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE    │ DATA STREAM                 │ TEST TYPE │ TEST NAME                                                                   │ RESULT │ TIME ELAPSED │
├────────────┼─────────────────────────────┼───────────┼─────────────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ box_events │                             │ asset     │ dashboard box_events-ce6fbf50-2df9-11ed-8003-6d5721603181 is loaded         │ PASS   │      4.062µs │
│ box_events │                             │ asset     │ dashboard box_events-ff3d9940-2e03-11ed-8003-6d5721603181 is loaded         │ PASS   │        110ns │
│ box_events │ anomalous_download_alerts   │ asset     │ index_template logs-box_events.anomalous_download_alerts is loaded          │ PASS   │        276ns │
│ box_events │ anomalous_download_alerts   │ asset     │ ingest_pipeline logs-box_events.anomalous_download_alerts-0.1.0 is loaded   │ PASS   │         66ns │
│ box_events │ events                      │ asset     │ index_template logs-box_events.events is loaded                             │ PASS   │        205ns │
│ box_events │ events                      │ asset     │ ingest_pipeline logs-box_events.events-0.1.0 is loaded                      │ PASS   │        169ns │
│ box_events │ malicious_content_alerts    │ asset     │ index_template logs-box_events.malicious_content_alerts is loaded           │ PASS   │        196ns │
│ box_events │ malicious_content_alerts    │ asset     │ ingest_pipeline logs-box_events.malicious_content_alerts-0.1.0 is loaded    │ PASS   │        119ns │
│ box_events │ suspicious_locations_alerts │ asset     │ index_template logs-box_events.suspicious_locations_alerts is loaded        │ PASS   │        141ns │
│ box_events │ suspicious_locations_alerts │ asset     │ ingest_pipeline logs-box_events.suspicious_locations_alerts-0.1.0 is loaded │ PASS   │        185ns │
│ box_events │ suspicious_sessions_alerts  │ asset     │ index_template logs-box_events.suspicious_sessions_alerts is loaded         │ PASS   │        157ns │
│ box_events │ suspicious_sessions_alerts  │ asset     │ ingest_pipeline logs-box_events.suspicious_sessions_alerts-0.1.0 is loaded  │ PASS   │        108ns │
╰────────────┴─────────────────────────────┴───────────┴─────────────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: box_events - END   ---
Done
Run pipeline tests for the package
--- Test results for package: box_events - START ---
╭────────────┬─────────────────────────────┬───────────┬───────────────────────────────┬────────┬──────────────╮
│ PACKAGE    │ DATA STREAM                 │ TEST TYPE │ TEST NAME                     │ RESULT │ TIME ELAPSED │
├────────────┼─────────────────────────────┼───────────┼───────────────────────────────┼────────┼──────────────┤
│ box_events │ anomalous_download_alerts   │ pipeline  │ test-anomalous-download.log   │ PASS   │   6.899616ms │
│ box_events │ events                      │ pipeline  │ test-copy.log                 │ PASS   │     2.5308ms │
│ box_events │ events                      │ pipeline  │ test-create.log               │ PASS   │    3.65869ms │
│ box_events │ events                      │ pipeline  │ test-download.log             │ PASS   │   3.548946ms │
│ box_events │ events                      │ pipeline  │ test-event-types.log          │ PASS   │  65.519602ms │
│ box_events │ events                      │ pipeline  │ test-preview.log              │ PASS   │   2.882847ms │
│ box_events │ events                      │ pipeline  │ test-rename.log               │ PASS   │   2.556427ms │
│ box_events │ events                      │ pipeline  │ test-trash.log                │ PASS   │   3.062823ms │
│ box_events │ events                      │ pipeline  │ test-upload.log               │ PASS   │   2.883961ms │
│ box_events │ malicious_content_alerts    │ pipeline  │ test-malicious-content.log    │ PASS   │   3.108011ms │
│ box_events │ suspicious_locations_alerts │ pipeline  │ test-suspicious-locations.log │ PASS   │   2.560171ms │
│ box_events │ suspicious_sessions_alerts  │ pipeline  │ test-suspicious-sessions.log  │ PASS   │    3.06026ms │
╰────────────┴─────────────────────────────┴───────────┴───────────────────────────────┴────────┴──────────────╯
--- Test results for package: box_events - END   ---
Done
Run static tests for the package
--- Test results for package: box_events - START ---
╭────────────┬─────────────────────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE    │ DATA STREAM                 │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├────────────┼─────────────────────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ box_events │ anomalous_download_alerts   │ static    │ Verify sample_event.json │ PASS   │  44.056967ms │
│ box_events │ events                      │ static    │ Verify sample_event.json │ PASS   │  41.476353ms │
│ box_events │ malicious_content_alerts    │ static    │ Verify sample_event.json │ PASS   │  43.201784ms │
│ box_events │ suspicious_locations_alerts │ static    │ Verify sample_event.json │ PASS   │  40.801869ms │
│ box_events │ suspicious_sessions_alerts  │ static    │ Verify sample_event.json │ PASS   │  54.500493ms │
╰────────────┴─────────────────────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: box_events - END   ---
Done
Run system tests for the package
--- Test results for package: box_events - START ---
No test results
--- Test results for package: box_events - END   ---
Done

@djptek djptek marked this pull request as draft July 11, 2022 12:00
@elasticmachine
Copy link

elasticmachine commented Jul 11, 2022

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-09-14T09:40:05.028+0000

  • Duration: 12 min 18 sec

Test stats 🧪

Test Results
Failed 0
Passed 29
Skipped 0
Total 29

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Jul 11, 2022

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (5/5) 💚
Files 100.0% (5/5) 💚 2.717
Classes 100.0% (5/5) 💚 2.717
Methods 91.525% (54/59) 👍 1.833
Lines 94.498% (1460/1545) 👍 3.137
Conditionals 100.0% (0/0) 💚

@djptek djptek marked this pull request as ready for review September 12, 2022 19:09
@djptek
Copy link
Contributor Author

djptek commented Sep 12, 2022

Huge thanks for the extensive reviewing @efd6

I've applied all your recommendations

processors:
- set:
field: ecs.version
value: "8.3.0"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has this been changed or is it being intentionally held at 8.3.0?

@djptek
Copy link
Contributor Author

djptek commented Sep 13, 2022

@efd6 re:

      field: ecs.version
      value: "8.3.0"

Has this been changed or is it being intentionally held at 8.3.0?

Nope, I've been testing against 8.3 - makes more sense to move to 8.4 so I've updated across the board

@djptek djptek requested a review from efd6 September 13, 2022 08:43
Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pipeline test expectations need to be updated

It looks like the sample events were obtained from somewhere else (there are no system tests that would have generated them and the fields are not in lexical order as would be expected from generated samples). Is this the case

After the tests are fixed, this LGTM, but it would be good to get input on the event type mappings to ECS from someone else as well.

@djptek
Copy link
Contributor Author

djptek commented Sep 14, 2022

@efd6

It looks like the sample events were obtained from somewhere else (there are no system tests that would have generated them and the fields are not in lexical order as would be expected from generated samples). Is this the case

This is partially correct.

  • The event datastream is actually using real events extracted from the API during an earlier iteration of a system test, which required full setup of the API and wasn't practical to incorporate into the integration test.
  • The pipeline tests for Shield events came from the original API documentation. I'll add an explanation, I think they better stay in Beta until there is a relevant license available

@djptek
Copy link
Contributor Author

djptek commented Sep 14, 2022

Root comment updated with detail on origins of pipeline test events

@djptek
Copy link
Contributor Author

djptek commented Sep 14, 2022

@kgeller @ebeahan @jamiehynds @epixa

I'd be grateful if one of you might provide input on the event type mappings for the box integration, which are summarised in the following 5 sample documents, listed below with links to source documentation and target ingest document

Box Events

Box Shield Alerts

There is also an implicit mapping between the Box event_type field and ECS event.category[] and event.type[] arrays. The 3rd party documentation is brief, so I've made my best estimate for the 130 entries. If you want to take a look, the most concise representation is encapsulated in this map

@kgeller
Copy link
Contributor

kgeller commented Sep 14, 2022

@djptek LGTM! It aligns with my understanding, and what I've done for my two integrations so far

@djptek
Copy link
Contributor Author

djptek commented Sep 14, 2022

Thanks @kgeller

@djptek djptek merged commit ce52809 into elastic:main Sep 14, 2022
@djptek djptek deleted the box branch September 14, 2022 13:59
@djptek djptek restored the box branch September 24, 2022 08:03
@djptek djptek mentioned this pull request Oct 26, 2022
27 tasks
djptek pushed a commit to djptek-legacy/integrations that referenced this pull request Nov 4, 2022
# This is the 1st commit message:

squash commits from PR elastic#3677

# This is the commit message #2:

bump ecs version to 8.4

# This is the commit message #3:

bump version to 8.4

# This is the commit message #4:

update sample events and simplify event type script

# This is the commit message #5:

revert kibana version else can't run tests
djptek pushed a commit to djptek-legacy/integrations that referenced this pull request Nov 7, 2022
Squash Commits picked from Original Box PR elastic#3677
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants