-
Notifications
You must be signed in to change notification settings - Fork 444
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[box] New Integration #3677
[box] New Integration #3677
Conversation
🌐 Coverage report
|
Huge thanks for the extensive reviewing @efd6 I've applied all your recommendations |
packages/box_events/data_stream/events/_dev/test/pipeline/test-event-types.log
Show resolved
Hide resolved
...tream/anomalous_download_alerts/_dev/test/pipeline/test-anomalous-download.log-expected.json
Show resolved
Hide resolved
packages/box_events/data_stream/events/elasticsearch/ingest_pipeline/default.yml
Outdated
Show resolved
Hide resolved
processors: | ||
- set: | ||
field: ecs.version | ||
value: "8.3.0" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Has this been changed or is it being intentionally held at 8.3.0?
@efd6 re:
Nope, I've been testing against |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The pipeline test expectations need to be updated
It looks like the sample events were obtained from somewhere else (there are no system tests that would have generated them and the fields are not in lexical order as would be expected from generated samples). Is this the case
After the tests are fixed, this LGTM, but it would be good to get input on the event type mappings to ECS from someone else as well.
This is partially correct.
|
Root comment updated with detail on origins of pipeline test events |
@kgeller @ebeahan @jamiehynds @epixa I'd be grateful if one of you might provide input on the event type mappings for the box integration, which are summarised in the following 5 sample documents, listed below with links to source documentation and target ingest document Box EventsBox Shield Alerts
There is also an implicit mapping between the Box |
@djptek LGTM! It aligns with my understanding, and what I've done for my two integrations so far |
Thanks @kgeller |
# This is the 1st commit message: squash commits from PR elastic#3677 # This is the commit message #2: bump ecs version to 8.4 # This is the commit message #3: bump version to 8.4 # This is the commit message #4: update sample events and simplify event type script # This is the commit message #5: revert kibana version else can't run tests
Squash Commits picked from Original Box PR elastic#3677
What does this PR do?
Adds integration for BOX events.
Checklist
changelog.yml
file.Author's Checklist
- [] Complete system testsAdd Shield Alerts
Suspicious locations
Suspicious sessions
Anomalous downloads
Malicious content
Update Dashboards
Update Screenshots
Update docs template
_dev/build/docs/README.md
How to test this PR locally
Use
elastic-package test
to run the tests.The pipeline tests for the events were created by ingesting data directly from the Box API using Filebeat HTTPJSON with no processors applied, then cut-and-paste of the message field for each resultant Elasticsearch Document to the relevant inputs.
An additional test was added using fake events, one corresponding to each of all documented box event types. This test is intended to exercise the mapping of the box
event_type
field value to the corresponding values of ECS fieldsevent.category
andevent.type
.The simulated shield alerts were created by:
httpjson.yml.hbs
for the relevant data stream, to ensure that the distinctresponse.split.split
anddrop_event
pipeline processors were applied to denormalise alerts to constituent documents and select these for routing to the relevant data-stream pipelineBox Shield Alerts should be considered as Beta until such time as access to a suitable target system with alerts relating to suspected malign activity is available.
To run an end-to-end test with live data, you will need an account on Box, a developer account is fine, don't need enterprise.
Add some test data to that and copy/delete/upload/download/view the data to create some events.
Have your
box
credentials available(do not store these in Github)
see the documentation of the
box_events
Integration for details.Start the stack using
elastic-package stack up -v -d --version 8.3.0
Add and configure your
box_events
Integration using your Box credentials.Ensure to scroll down to
and assign the policy to
Existing hosts
->Agent Policy
->Elastic-Agent
Generate some events in Box by Upload/Download/Delete/Visualisation of files then go to the
[Logs Box Events Integration] Events
dashboard to verify that events were ingested and are visible.To add Simulated Shield Events, you can cut and paste the contents of this
_bulk
request intoDev Tools
and send the request to the stack. This will ingest a selection of alerts to validate the[Logs Box Events Integration] Box Shield Alerts
dashboard, you will need to set the time picker to July 2022.The Simulated Shield Alerts were created by manually appending raw
agent
etc. fields created during the end-to-end tests for each alert's pipeline test document(s), and then rectifying the data-stream for each.Related issues
Screenshots
For Kibana UI options see test procedure, above.
Dashboards
Showing Elastic Stack 7.17.0 compatible dashboards
Box Events
Box Shield Alerts
Test Results