[7.x] [Security Solution][Detection Engine] Bubbles up errors when it cannot create signal documents (#77687) #77977
+183
−30
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t create signal documents (elastic#77687) ## Summary Fixes: elastic#77255, elastic#63712 This bubbles up errors when we cannot correctly create signal documents. Before this PR, we sometimes would mark documents as being in the error state for the end user but ask them to look in their Kibana logs for the specific error. In some cases we did not bubble up any error states and the signal would look like it had 0 signals when that wasn't true. It had valid signals but could not write the signals to its index because the source index had incompatibilities with ECS and cannot write the document to the signals index. This fixes those issues to correctly bubble up the errors. If you're interested in manual testing there are two ways. The first way is to take advantage of an existing "threshold bug" by making a "threshold rule" which has a CIDR in it like so below: <img width="1046" alt="Screen Shot 2020-09-10 at 4 08 18 PM" src="https://user-images.githubusercontent.com/1151048/93532721-cba21480-f8fe-11ea-90e7-27c39fdb870b.png"> On output you should see that the threshold is in an error state for the rule and also additional details: <img width="1852" alt="Screen Shot 2020-09-16 at 1 26 37 PM" src="https://user-images.githubusercontent.com/1151048/93532789-eaa0a680-f8fe-11ea-9327-81cf128a344e.png"> The second way to trigger this is to create a mock invalid ECS index with an invalid mapping in dev tools: ```ts # This is invalid because it has an odd "original" inside of it PUT mock-bad-ecs-index { "mappings": { "properties": { "@timestamp": { "type": "date" }, "message": { "properties": { "original": { "type": "text", "index": false, "doc_values": false } } } } } } # You might have to change your timestamp to be 5 minutes from now to catch it as a signal or use a really long look back time PUT mock-bad-ecs-index/_doc/1 { "@timestamp": "2020-09-17T21:50:54.240Z", "message": { "original": "invalid subobject" } } ``` Then create a rule against this index: <img width="1045" alt="Screen Shot 2020-09-17 at 3 52 40 PM" src="https://user-images.githubusercontent.com/1151048/93532922-30f60580-f8ff-11ea-8131-172aab7b7c68.png"> And you should see an error banner and error state where before it would not show the error message: <img width="1866" alt="Screen Shot 2020-09-17 at 3 53 20 PM" src="https://user-images.githubusercontent.com/1151048/93532972-4408d580-f8ff-11ea-8105-44b0f767cc70.png"> ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
💚 Build SucceededBuild metrics
To update your PR or re-run it, just comment with: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backports the following commits to 7.x: