-
Notifications
You must be signed in to change notification settings - Fork 8.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Security Solution] Migrates siem-detection-engine-rule-actions ruleAlertId and actions to saved object references array #113577
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍🏽 Just doing a quick first code pass. Wanted to manually test after #113205 goes in.
|
||
return legacyGetRuleActionsFromSavedObject(ruleActionsSavedObject); | ||
}: LegacyCreateRuleActionsSavedObject): Promise<void> => { | ||
const referenceWithAlertId: SavedObjectReference[] = [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: This may just be me but I'm starting to get confused between references to "alerts" and "rules". This is the reference to the rule, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That is correct. It is a reference to the rule.
...solution/server/lib/detection_engine/rule_actions/legacy_create_rule_actions_saved_object.ts
Outdated
Show resolved
Hide resolved
...lution/server/lib/detection_engine/rule_actions/legacy_get_bulk_rule_actions_saved_object.ts
Show resolved
Hide resolved
acc[savedObject.attributes.ruleAlertId] = legacyGetRuleActionsFromSavedObject(savedObject); | ||
const ruleAlertId = savedObject.references.find((reference) => { | ||
// Find the first rule alert and assume that is the one we want since we should only ever have 1. | ||
return reference.type === 'alert'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we add a check to log if more than one is found? Doesn't seem to have been an issue ever up till now, but maybe with the migrations and changes, could be worth checking these edge cases we weren't too worried about before. 🤷🏽♀️
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I can. You are correct that we can't guarantee that there aren't more than 1. I figured if there was more than 1 I would just pull 1. If there's 2 or more, I think we have to just pull the first 1. Best I can do would be to check if there is more than 1 and then write out an error message in case we begin encountering it we will see it fairly quickly in the log files.
group: { | ||
type: 'keyword', | ||
}, | ||
id: { | ||
// "actions.id" is no longer used since the saved object references and "actionRef" was introduced. It is still here for legacy reasons such as migrations. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the one we've decided not to delete yet in case of any migration issues right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Correct:
Actually I (think?) I have to keep it because older data will need to be migrated and we cannot remove the mapping as far as I know. I don't know if the mappings are changed before the migration but I assume it would have to be since I already am adding the actionRef
above. So if the mappings change before the migration (which I'm pretty sure it does and would need to), and I remove this here I don't know what the effect is going to be. I might still be able to access it without it but I figured it was best to leave it alone for now?
Overall I felt like I should just leave it alone for now.
* Post 7.16 this is how it is serialized from the saved object from disk since we are using saved object references. | ||
* @deprecated Remove this once the legacy notification/side car is gone | ||
*/ | ||
export interface LegacyRuleAlertAction { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These are where you hit the issues that come from typing migrations, right? You'd talked me through this change here on zoom (thank you!) and I'm wondering if it's worth creating an issue that we can socialize with core (I'm happy to take that on). Nothing to solve asap, but def important to try and get some consensus on how to go about this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should talk more in person. I don't think it's an easy job to make types per schema version. It's not that bad at this moment fwiw.
...erver/lib/detection_engine/rule_actions/legacy_update_or_create_rule_actions_saved_object.ts
Show resolved
Hide resolved
...solution/server/lib/detection_engine/rule_actions/legacy_update_rule_actions_saved_object.ts
Show resolved
Hide resolved
...solution/server/lib/detection_engine/rule_actions/legacy_update_rule_actions_saved_object.ts
Outdated
Show resolved
Hide resolved
x-pack/plugins/security_solution/server/lib/detection_engine/rule_actions/legacy_utils.ts
Show resolved
Hide resolved
…f the legacy pr PR review
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the added tests and addressing all my nits 😬
LGTM!
@elasticmachine merge upstream |
@elasticmachine merge upstream |
@elasticmachine merge upstream |
@elasticmachine merge upstream |
💚 Build Succeeded
Metrics [docs]Saved Objects .kibana field count
History
To update your PR or re-run it, just comment with: |
…lertId and actions to saved object references array (elastic#113577) ## Summary Fixes elastic#113278 * Migrates the legacy `siem-detection-engine-rule-actions` `ruleAlertId` and `actions` to saved object references arrays * Adds an e2e test for `siem-detection-engine-rule-actions` * Updates the types to work with the migrations and the new and old data structures. * Decouples and removes reliance on alerting within the types since we do not want development of alerting to get in the way of legacy things and have migration changes by accident. * Updates the REST interface and code to produce post migration data structures. Removes some types and code where w can since those parts are no longer needed/used. * Adds `actionRef` to the mapping Before migration you should see data structures like this if you query: ```json GET .kibana/_search { "query": { "term": { "type": { "value": "siem-detection-engine-rule-actions" } } } } ``` ```json { "siem-detection-engine-rule-actions": { "ruleAlertId": "fb1046a0-0452-11ec-9b15-d13d79d162f3", <-- ruleAlertId which we want in the references array and removed "actions": [ { "action_type_id": ".slack", "id": "f6e64c00-0452-11ec-9b15-d13d79d162f3", <-- id which we want in the references array and removed "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "group": "default" } ], "ruleThrottle": "7d", "alertThrottle": "7d" }, "type": "siem-detection-engine-rule-actions", "references": [], <-- Array is empty which instead needs the id's of alerts and actions "migrationVersion": { "siem-detection-engine-rule-actions": "7.11.2" }, "coreMigrationVersion": "7.14.0", "updated_at": "2021-09-15T22:18:48.369Z" } ``` After migration you should see data structures like this: ```json { "siem-detection-engine-rule-actions": { "actions": [ { "action_type_id": ".slack", "actionRef" : "action_0", <-- We use the name and "actionRef" to be consistent with kibana alerting "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "group": "default" } ], "ruleThrottle": "7d", "alertThrottle": "7d" }, "type": "siem-detection-engine-rule-actions", "references" : [ { "name" : "alert_0", <-- Name is "alert_0" "id" : "fb1046a0-0452-11ec-9b15-d13d79d162f3", <-- Alert id is now here "type" : "alert" <-- Type should be "alert" }, { "name" : "action_0", <-- Name is "action_0" and should be the same as kibana alerting names theirs for consistencty "id" : "f6e64c00-0452-11ec-9b15-d13d79d162f3", <-- Id of the action is now here. "type" : "action" <-- Type should be "action" } ], "migrationVersion": { "siem-detection-engine-rule-actions": "7.16.0" }, "coreMigrationVersion": "8.0.0", "updated_at": "2021-09-15T22:18:48.369Z" } ``` Manual testing --- There are e2e tests but for any manual testing or verification you can do the following: If you have a 7.14.0 system and can migrate it forward that is the most straight forward way to ensure this does migrate correctly and forward. You should see that the legacy notification system still operates as expected. If you are a developer off of master and want to test different scenarios then this section is for below as it is more involved and harder to do but goes into more depth: * Create a rule and activate it normally within security_solution * Do not add actions to the rule at this point as we are exercising the older legacy system. However, you want at least one action configured such as a slack notification. * Within dev tools do a query for all your actions and grab one of the `_id` of them without their prefix: ```json # See all your actions GET .kibana/_search { "query": { "term": { "type": "action" } } } ``` Mine was `"_id" : "action:879e8ff0-1be1-11ec-a722-83da1c22a481"`, so I will be copying the ID of `879e8ff0-1be1-11ec-a722-83da1c22a481` Go to the file `detection_engine/scripts/legacy_notifications/one_action.json` and add this id to the file. Something like this: ```json { "name": "Legacy notification with one action", "interval": "1m", <--- You can use whatever you want. Real values are "1h", "1d", "1w". I use "1m" for testing purposes. "actions": [ { "id": "879e8ff0-1be1-11ec-a722-83da1c22a481", <--- My action id "group": "default", "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "actionTypeId": ".slack" <--- I am a slack action id type. } ] } ``` Query for an alert you want to add manually add back a legacy notification to it. Such as: ```json # See all your siem.signals alert types and choose one GET .kibana/_search { "query": { "term": { "alert.alertTypeId": "siem.signals" } } } ``` Grab the `_id` without the alert prefix. For mine this was `933ca720-1be1-11ec-a722-83da1c22a481` Within the directory of detection_engine/scripts execute the script: ```json ./post_legacy_notification.sh 933ca720-1be1-11ec-a722-83da1c22a481 { "ok": "acknowledged" } ``` which is going to do a few things. See the file `detection_engine/routes/rules/legacy_create_legacy_notification.ts` for the definition of the route and what it does in full, but we should notice that we have now: Created a legacy side car action object of type `siem-detection-engine-rule-actions` you can see in dev tools: ```json # See the actions "side car" which are part of the legacy notification system. GET .kibana/_search { "query": { "term": { "type": { "value": "siem-detection-engine-rule-actions" } } } } ``` Take note that this actually creates the rule migrated since this PR updated the code to produce new side cars. So we have to use some scripting to change the actions to utilize the old format. However, before continuing you should verify that this does fire correctly and that the new format is working as expected. After that replace the structure with the older structure like so below and downgrade the migration version so that we can restart Kibana and ensure that this does migrate correctly forward: ```json # Get your id of your rules side car above and then use this script to downgrade the data structure POST .kibana/_update/siem-detection-engine-rule-actions:210f4c90-2233-11ec-98c6-ed2574588902 { "script" : { "source": """ ctx._source.migrationVersion['siem-detection-engine-rule-actions'] = "7.15.0"; ctx._source['siem-detection-engine-rule-actions'].actions[0].id = ctx._source.references[1].id; ctx._source['siem-detection-engine-rule-actions'].actions[0].remove('actionRef'); ctx._source['siem-detection-engine-rule-actions'].ruleAlertId = ctx._source.references[0].id; ctx._source.references.remove(0); ctx._source.references.remove(0); """, "lang": "painless" } } ``` Restart Kibana and now it should be migrated correctly and the system should fire the notifications as expected. You shouldn't see any errors in your console. In the scripts folder execute the `find_rules.sh` and expect to see actions like so in the rule with the `id` still in the REST interface and we shouldn't see `actionRef` within the actions: ```json "actions": [{ "id": "42534430-2092-11ec-99a6-05d79563c01a", "group": "default", "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "action_type_id": ".slack" }], ``` Take the rule id and query that as well using `./get_rule_by_id.sh` and verify that the action also looks the same and is present within the rule. You can also verify all of this within the UI's as well for rules to ensure the action is still present and as we expect it to be and work. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
💚 Backport successful
This backport PR will be merged automatically after passing CI. |
…lertId and actions to saved object references array (#113577) (#114201) ## Summary Fixes #113278 * Migrates the legacy `siem-detection-engine-rule-actions` `ruleAlertId` and `actions` to saved object references arrays * Adds an e2e test for `siem-detection-engine-rule-actions` * Updates the types to work with the migrations and the new and old data structures. * Decouples and removes reliance on alerting within the types since we do not want development of alerting to get in the way of legacy things and have migration changes by accident. * Updates the REST interface and code to produce post migration data structures. Removes some types and code where w can since those parts are no longer needed/used. * Adds `actionRef` to the mapping Before migration you should see data structures like this if you query: ```json GET .kibana/_search { "query": { "term": { "type": { "value": "siem-detection-engine-rule-actions" } } } } ``` ```json { "siem-detection-engine-rule-actions": { "ruleAlertId": "fb1046a0-0452-11ec-9b15-d13d79d162f3", <-- ruleAlertId which we want in the references array and removed "actions": [ { "action_type_id": ".slack", "id": "f6e64c00-0452-11ec-9b15-d13d79d162f3", <-- id which we want in the references array and removed "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "group": "default" } ], "ruleThrottle": "7d", "alertThrottle": "7d" }, "type": "siem-detection-engine-rule-actions", "references": [], <-- Array is empty which instead needs the id's of alerts and actions "migrationVersion": { "siem-detection-engine-rule-actions": "7.11.2" }, "coreMigrationVersion": "7.14.0", "updated_at": "2021-09-15T22:18:48.369Z" } ``` After migration you should see data structures like this: ```json { "siem-detection-engine-rule-actions": { "actions": [ { "action_type_id": ".slack", "actionRef" : "action_0", <-- We use the name and "actionRef" to be consistent with kibana alerting "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "group": "default" } ], "ruleThrottle": "7d", "alertThrottle": "7d" }, "type": "siem-detection-engine-rule-actions", "references" : [ { "name" : "alert_0", <-- Name is "alert_0" "id" : "fb1046a0-0452-11ec-9b15-d13d79d162f3", <-- Alert id is now here "type" : "alert" <-- Type should be "alert" }, { "name" : "action_0", <-- Name is "action_0" and should be the same as kibana alerting names theirs for consistencty "id" : "f6e64c00-0452-11ec-9b15-d13d79d162f3", <-- Id of the action is now here. "type" : "action" <-- Type should be "action" } ], "migrationVersion": { "siem-detection-engine-rule-actions": "7.16.0" }, "coreMigrationVersion": "8.0.0", "updated_at": "2021-09-15T22:18:48.369Z" } ``` Manual testing --- There are e2e tests but for any manual testing or verification you can do the following: If you have a 7.14.0 system and can migrate it forward that is the most straight forward way to ensure this does migrate correctly and forward. You should see that the legacy notification system still operates as expected. If you are a developer off of master and want to test different scenarios then this section is for below as it is more involved and harder to do but goes into more depth: * Create a rule and activate it normally within security_solution * Do not add actions to the rule at this point as we are exercising the older legacy system. However, you want at least one action configured such as a slack notification. * Within dev tools do a query for all your actions and grab one of the `_id` of them without their prefix: ```json # See all your actions GET .kibana/_search { "query": { "term": { "type": "action" } } } ``` Mine was `"_id" : "action:879e8ff0-1be1-11ec-a722-83da1c22a481"`, so I will be copying the ID of `879e8ff0-1be1-11ec-a722-83da1c22a481` Go to the file `detection_engine/scripts/legacy_notifications/one_action.json` and add this id to the file. Something like this: ```json { "name": "Legacy notification with one action", "interval": "1m", <--- You can use whatever you want. Real values are "1h", "1d", "1w". I use "1m" for testing purposes. "actions": [ { "id": "879e8ff0-1be1-11ec-a722-83da1c22a481", <--- My action id "group": "default", "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "actionTypeId": ".slack" <--- I am a slack action id type. } ] } ``` Query for an alert you want to add manually add back a legacy notification to it. Such as: ```json # See all your siem.signals alert types and choose one GET .kibana/_search { "query": { "term": { "alert.alertTypeId": "siem.signals" } } } ``` Grab the `_id` without the alert prefix. For mine this was `933ca720-1be1-11ec-a722-83da1c22a481` Within the directory of detection_engine/scripts execute the script: ```json ./post_legacy_notification.sh 933ca720-1be1-11ec-a722-83da1c22a481 { "ok": "acknowledged" } ``` which is going to do a few things. See the file `detection_engine/routes/rules/legacy_create_legacy_notification.ts` for the definition of the route and what it does in full, but we should notice that we have now: Created a legacy side car action object of type `siem-detection-engine-rule-actions` you can see in dev tools: ```json # See the actions "side car" which are part of the legacy notification system. GET .kibana/_search { "query": { "term": { "type": { "value": "siem-detection-engine-rule-actions" } } } } ``` Take note that this actually creates the rule migrated since this PR updated the code to produce new side cars. So we have to use some scripting to change the actions to utilize the old format. However, before continuing you should verify that this does fire correctly and that the new format is working as expected. After that replace the structure with the older structure like so below and downgrade the migration version so that we can restart Kibana and ensure that this does migrate correctly forward: ```json # Get your id of your rules side car above and then use this script to downgrade the data structure POST .kibana/_update/siem-detection-engine-rule-actions:210f4c90-2233-11ec-98c6-ed2574588902 { "script" : { "source": """ ctx._source.migrationVersion['siem-detection-engine-rule-actions'] = "7.15.0"; ctx._source['siem-detection-engine-rule-actions'].actions[0].id = ctx._source.references[1].id; ctx._source['siem-detection-engine-rule-actions'].actions[0].remove('actionRef'); ctx._source['siem-detection-engine-rule-actions'].ruleAlertId = ctx._source.references[0].id; ctx._source.references.remove(0); ctx._source.references.remove(0); """, "lang": "painless" } } ``` Restart Kibana and now it should be migrated correctly and the system should fire the notifications as expected. You shouldn't see any errors in your console. In the scripts folder execute the `find_rules.sh` and expect to see actions like so in the rule with the `id` still in the REST interface and we shouldn't see `actionRef` within the actions: ```json "actions": [{ "id": "42534430-2092-11ec-99a6-05d79563c01a", "group": "default", "params": { "message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts" }, "action_type_id": ".slack" }], ``` Take the rule id and query that as well using `./get_rule_by_id.sh` and verify that the action also looks the same and is present within the rule. You can also verify all of this within the UI's as well for rules to ensure the action is still present and as we expect it to be and work. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Frank Hassanabad <frank.hassanabad@elastic.co>
## Summary During the work here: #113577 I accidentally have introduced a bug where on migration I was deleting the attributes of `ruleThrottle` and `alertThrottle` because I was not using splat correctly. Added unit and e2e tests to fix this. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
## Summary During the work here: elastic#113577 I accidentally have introduced a bug where on migration I was deleting the attributes of `ruleThrottle` and `alertThrottle` because I was not using splat correctly. Added unit and e2e tests to fix this. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
## Summary During the work here: #113577 I accidentally have introduced a bug where on migration I was deleting the attributes of `ruleThrottle` and `alertThrottle` because I was not using splat correctly. Added unit and e2e tests to fix this. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Frank Hassanabad <frank.hassanabad@elastic.co>
…d to saved object references array (#114585) ## Summary Resolves (a portion of) #107068 for the `siem-detection-engine-rule-status` type by migrating the `alertId` to be within the `SO references[]`. Based on: #113577 * Migrates the legacy `siem-detection-engine-rule-status` `alertId` to saved object references array * Adds an e2e test for `siem-detection-engine-rule-status` * Breaks out `siem-detection-engine-rule-status` & `security-rule` SO's to their own dedicated files/directories, and cleaned up typings/imports Before migration you can observe the existing data structure of `siem-detection-engine-rule-status` via Dev tools as follows: ``` GET .kibana/_search { "size": 10000, "query": { "term": { "type": { "value": "siem-detection-engine-rule-status" } } } } ``` ``` JSON { "_index" : ".kibana-spong_8.0.0_001", "_id" : "siem-detection-engine-rule-status:d580f1a0-2afe-11ec-8621-8d6bfcdfd75e", "_score" : 2.150102, "_source" : { "siem-detection-engine-rule-status" : { "alertId" : "d62d2980-27c4-11ec-92b0-f7b47106bb35", <-- alertId which we want in the references array and removed "statusDate" : "2021-10-12T01:50:52.898Z", "status" : "failed", "lastFailureAt" : "2021-10-12T01:50:52.898Z", "lastSuccessAt" : "2021-10-12T01:18:29.195Z", "lastFailureMessage" : "6 minutes (385585ms) were not queried between this rule execution and the last execution, so signals may have been missed. Consider increasing your look behind time or adding more Kibana instances. name: \"I am the Host who Names!\" id: \"d62d2980-27c4-11ec-92b0-f7b47106bb35\" rule id: \"214ccef6-e98e-493a-98c5-5bcc2d497b79\" signals index: \".siem-signals-spong-default\"", "lastSuccessMessage" : "succeeded", "gap" : "6 minutes", "lastLookBackDate" : "2021-10-07T23:43:27.961Z" }, "type" : "siem-detection-engine-rule-status", "references" : [ ], "coreMigrationVersion" : "7.14.0", "updated_at" : "2021-10-12T01:50:53.404Z" } } ``` Post migration the data structure should be updated as follows: ``` JSON { "_index": ".kibana-spong_8.0.0_001", "_id": "siem-detection-engine-rule-status:d580f1a0-2afe-11ec-8621-8d6bfcdfd75e", "_score": 2.1865466, "_source": { "siem-detection-engine-rule-status": { "statusDate": "2021-10-12T01:50:52.898Z", <-- alertId is no more! "status": "failed", "lastFailureAt": "2021-10-12T01:50:52.898Z", "lastSuccessAt": "2021-10-12T01:18:29.195Z", "lastFailureMessage": "6 minutes (385585ms) were not queried between this rule execution and the last execution, so signals may have been missed. Consider increasing your look behind time or adding more Kibana instances. name: \"I am the Host who Names!\" id: \"d62d2980-27c4-11ec-92b0-f7b47106bb35\" rule id: \"214ccef6-e98e-493a-98c5-5bcc2d497b79\" signals index: \".siem-signals-spong-default\"", "lastSuccessMessage": "succeeded", "gap": "6 minutes", "lastLookBackDate": "2021-10-07T23:43:27.961Z" }, "type": "siem-detection-engine-rule-status", "references": [ { "id": "d62d2980-27c4-11ec-92b0-f7b47106bb35", <-- previous alertId has been converted to references[] "type": "alert", "name": "alert_0" } ], "migrationVersion": { "siem-detection-engine-rule-status": "7.16.0" }, "coreMigrationVersion": "8.0.0", "updated_at": "2021-10-12T01:50:53.406Z" } }, ``` #### Manual testing --- There are e2e tests but for any manual testing or verification you can do the following: ##### Manual upgrade test If you have a 7.15.0 system and can migrate it forward that is the most straight forward way to ensure this does migrate correctly. You should see that the `Rule Monitoring` table and Rule Details `Failure History` table continue to function without error. ##### Downgrade via script and test migration on kibana reboot If you have a migrated `Rule Status SO` and want to test the migration, you can run the below script to downgrade the status SO then restart Kibana and observe the migration on startup. Note: Since this PR removes the mapping, you would need to [update the SO mapping](https://github.com/elastic/kibana/pull/114585/files#r729386126) to include `alertId` again else you will receive a strict/dynamic mapping error. ```json # Replace id w/ correct Rule Status SO id of existing migrated object POST .kibana/_update/siem-detection-engine-rule-status:d580ca91-2afe-11ec-8621-8d6bfcdfd75e { "script" : { "source": """ ctx._source.migrationVersion['siem-detection-engine-rule-status'] = "7.15.0"; ctx._source['siem-detection-engine-rule-status'].alertId = ctx._source.references[0].id; ctx._source.references.remove(0); """, "lang": "painless" } } ``` Restart Kibana and now it should be migrated correctly and you shouldn't see any errors in your console. You should also see that the `Rule Monitoring` table and Rule Details `Failure History` table continue to function without error. ### Checklist Delete any items that are not applicable to this PR. - [ ] ~[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials~ - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### For maintainers - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
…d to saved object references array (elastic#114585) ## Summary Resolves (a portion of) elastic#107068 for the `siem-detection-engine-rule-status` type by migrating the `alertId` to be within the `SO references[]`. Based on: elastic#113577 * Migrates the legacy `siem-detection-engine-rule-status` `alertId` to saved object references array * Adds an e2e test for `siem-detection-engine-rule-status` * Breaks out `siem-detection-engine-rule-status` & `security-rule` SO's to their own dedicated files/directories, and cleaned up typings/imports Before migration you can observe the existing data structure of `siem-detection-engine-rule-status` via Dev tools as follows: ``` GET .kibana/_search { "size": 10000, "query": { "term": { "type": { "value": "siem-detection-engine-rule-status" } } } } ``` ``` JSON { "_index" : ".kibana-spong_8.0.0_001", "_id" : "siem-detection-engine-rule-status:d580f1a0-2afe-11ec-8621-8d6bfcdfd75e", "_score" : 2.150102, "_source" : { "siem-detection-engine-rule-status" : { "alertId" : "d62d2980-27c4-11ec-92b0-f7b47106bb35", <-- alertId which we want in the references array and removed "statusDate" : "2021-10-12T01:50:52.898Z", "status" : "failed", "lastFailureAt" : "2021-10-12T01:50:52.898Z", "lastSuccessAt" : "2021-10-12T01:18:29.195Z", "lastFailureMessage" : "6 minutes (385585ms) were not queried between this rule execution and the last execution, so signals may have been missed. Consider increasing your look behind time or adding more Kibana instances. name: \"I am the Host who Names!\" id: \"d62d2980-27c4-11ec-92b0-f7b47106bb35\" rule id: \"214ccef6-e98e-493a-98c5-5bcc2d497b79\" signals index: \".siem-signals-spong-default\"", "lastSuccessMessage" : "succeeded", "gap" : "6 minutes", "lastLookBackDate" : "2021-10-07T23:43:27.961Z" }, "type" : "siem-detection-engine-rule-status", "references" : [ ], "coreMigrationVersion" : "7.14.0", "updated_at" : "2021-10-12T01:50:53.404Z" } } ``` Post migration the data structure should be updated as follows: ``` JSON { "_index": ".kibana-spong_8.0.0_001", "_id": "siem-detection-engine-rule-status:d580f1a0-2afe-11ec-8621-8d6bfcdfd75e", "_score": 2.1865466, "_source": { "siem-detection-engine-rule-status": { "statusDate": "2021-10-12T01:50:52.898Z", <-- alertId is no more! "status": "failed", "lastFailureAt": "2021-10-12T01:50:52.898Z", "lastSuccessAt": "2021-10-12T01:18:29.195Z", "lastFailureMessage": "6 minutes (385585ms) were not queried between this rule execution and the last execution, so signals may have been missed. Consider increasing your look behind time or adding more Kibana instances. name: \"I am the Host who Names!\" id: \"d62d2980-27c4-11ec-92b0-f7b47106bb35\" rule id: \"214ccef6-e98e-493a-98c5-5bcc2d497b79\" signals index: \".siem-signals-spong-default\"", "lastSuccessMessage": "succeeded", "gap": "6 minutes", "lastLookBackDate": "2021-10-07T23:43:27.961Z" }, "type": "siem-detection-engine-rule-status", "references": [ { "id": "d62d2980-27c4-11ec-92b0-f7b47106bb35", <-- previous alertId has been converted to references[] "type": "alert", "name": "alert_0" } ], "migrationVersion": { "siem-detection-engine-rule-status": "7.16.0" }, "coreMigrationVersion": "8.0.0", "updated_at": "2021-10-12T01:50:53.406Z" } }, ``` #### Manual testing --- There are e2e tests but for any manual testing or verification you can do the following: ##### Manual upgrade test If you have a 7.15.0 system and can migrate it forward that is the most straight forward way to ensure this does migrate correctly. You should see that the `Rule Monitoring` table and Rule Details `Failure History` table continue to function without error. ##### Downgrade via script and test migration on kibana reboot If you have a migrated `Rule Status SO` and want to test the migration, you can run the below script to downgrade the status SO then restart Kibana and observe the migration on startup. Note: Since this PR removes the mapping, you would need to [update the SO mapping](https://github.com/elastic/kibana/pull/114585/files#r729386126) to include `alertId` again else you will receive a strict/dynamic mapping error. ```json # Replace id w/ correct Rule Status SO id of existing migrated object POST .kibana/_update/siem-detection-engine-rule-status:d580ca91-2afe-11ec-8621-8d6bfcdfd75e { "script" : { "source": """ ctx._source.migrationVersion['siem-detection-engine-rule-status'] = "7.15.0"; ctx._source['siem-detection-engine-rule-status'].alertId = ctx._source.references[0].id; ctx._source.references.remove(0); """, "lang": "painless" } } ``` Restart Kibana and now it should be migrated correctly and you shouldn't see any errors in your console. You should also see that the `Rule Monitoring` table and Rule Details `Failure History` table continue to function without error. ### Checklist Delete any items that are not applicable to this PR. - [ ] ~[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials~ - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### For maintainers - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
…d to saved object references array (#114585) (#115359) ## Summary Resolves (a portion of) #107068 for the `siem-detection-engine-rule-status` type by migrating the `alertId` to be within the `SO references[]`. Based on: #113577 * Migrates the legacy `siem-detection-engine-rule-status` `alertId` to saved object references array * Adds an e2e test for `siem-detection-engine-rule-status` * Breaks out `siem-detection-engine-rule-status` & `security-rule` SO's to their own dedicated files/directories, and cleaned up typings/imports Before migration you can observe the existing data structure of `siem-detection-engine-rule-status` via Dev tools as follows: ``` GET .kibana/_search { "size": 10000, "query": { "term": { "type": { "value": "siem-detection-engine-rule-status" } } } } ``` ``` JSON { "_index" : ".kibana-spong_8.0.0_001", "_id" : "siem-detection-engine-rule-status:d580f1a0-2afe-11ec-8621-8d6bfcdfd75e", "_score" : 2.150102, "_source" : { "siem-detection-engine-rule-status" : { "alertId" : "d62d2980-27c4-11ec-92b0-f7b47106bb35", <-- alertId which we want in the references array and removed "statusDate" : "2021-10-12T01:50:52.898Z", "status" : "failed", "lastFailureAt" : "2021-10-12T01:50:52.898Z", "lastSuccessAt" : "2021-10-12T01:18:29.195Z", "lastFailureMessage" : "6 minutes (385585ms) were not queried between this rule execution and the last execution, so signals may have been missed. Consider increasing your look behind time or adding more Kibana instances. name: \"I am the Host who Names!\" id: \"d62d2980-27c4-11ec-92b0-f7b47106bb35\" rule id: \"214ccef6-e98e-493a-98c5-5bcc2d497b79\" signals index: \".siem-signals-spong-default\"", "lastSuccessMessage" : "succeeded", "gap" : "6 minutes", "lastLookBackDate" : "2021-10-07T23:43:27.961Z" }, "type" : "siem-detection-engine-rule-status", "references" : [ ], "coreMigrationVersion" : "7.14.0", "updated_at" : "2021-10-12T01:50:53.404Z" } } ``` Post migration the data structure should be updated as follows: ``` JSON { "_index": ".kibana-spong_8.0.0_001", "_id": "siem-detection-engine-rule-status:d580f1a0-2afe-11ec-8621-8d6bfcdfd75e", "_score": 2.1865466, "_source": { "siem-detection-engine-rule-status": { "statusDate": "2021-10-12T01:50:52.898Z", <-- alertId is no more! "status": "failed", "lastFailureAt": "2021-10-12T01:50:52.898Z", "lastSuccessAt": "2021-10-12T01:18:29.195Z", "lastFailureMessage": "6 minutes (385585ms) were not queried between this rule execution and the last execution, so signals may have been missed. Consider increasing your look behind time or adding more Kibana instances. name: \"I am the Host who Names!\" id: \"d62d2980-27c4-11ec-92b0-f7b47106bb35\" rule id: \"214ccef6-e98e-493a-98c5-5bcc2d497b79\" signals index: \".siem-signals-spong-default\"", "lastSuccessMessage": "succeeded", "gap": "6 minutes", "lastLookBackDate": "2021-10-07T23:43:27.961Z" }, "type": "siem-detection-engine-rule-status", "references": [ { "id": "d62d2980-27c4-11ec-92b0-f7b47106bb35", <-- previous alertId has been converted to references[] "type": "alert", "name": "alert_0" } ], "migrationVersion": { "siem-detection-engine-rule-status": "7.16.0" }, "coreMigrationVersion": "8.0.0", "updated_at": "2021-10-12T01:50:53.406Z" } }, ``` #### Manual testing --- There are e2e tests but for any manual testing or verification you can do the following: ##### Manual upgrade test If you have a 7.15.0 system and can migrate it forward that is the most straight forward way to ensure this does migrate correctly. You should see that the `Rule Monitoring` table and Rule Details `Failure History` table continue to function without error. ##### Downgrade via script and test migration on kibana reboot If you have a migrated `Rule Status SO` and want to test the migration, you can run the below script to downgrade the status SO then restart Kibana and observe the migration on startup. Note: Since this PR removes the mapping, you would need to [update the SO mapping](https://github.com/elastic/kibana/pull/114585/files#r729386126) to include `alertId` again else you will receive a strict/dynamic mapping error. ```json # Replace id w/ correct Rule Status SO id of existing migrated object POST .kibana/_update/siem-detection-engine-rule-status:d580ca91-2afe-11ec-8621-8d6bfcdfd75e { "script" : { "source": """ ctx._source.migrationVersion['siem-detection-engine-rule-status'] = "7.15.0"; ctx._source['siem-detection-engine-rule-status'].alertId = ctx._source.references[0].id; ctx._source.references.remove(0); """, "lang": "painless" } } ``` Restart Kibana and now it should be migrated correctly and you shouldn't see any errors in your console. You should also see that the `Rule Monitoring` table and Rule Details `Failure History` table continue to function without error. ### Checklist Delete any items that are not applicable to this PR. - [ ] ~[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials~ - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### For maintainers - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Garrett Spong <spong@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary During the work here: elastic#113577 I accidentally have introduced a bug where on migration I was deleting the attributes of `ruleThrottle` and `alertThrottle` because I was not using splat correctly. Added unit and e2e tests to fix this. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Summary
Fixes #113278
siem-detection-engine-rule-actions
ruleAlertId
andactions
to saved object references arrayssiem-detection-engine-rule-actions
actionRef
to the mappingBefore migration you should see data structures like this if you query:
After migration you should see data structures like this:
Manual testing
There are e2e tests but for any manual testing or verification you can do the following:
If you have a 7.14.0 system and can migrate it forward that is the most straight forward way to ensure this does migrate correctly and forward. You should see that the legacy notification system still operates as expected.
If you are a developer off of master and want to test different scenarios then this section is for below as it is more involved and harder to do but goes into more depth:
_id
of them without their prefix:Mine was
"_id" : "action:879e8ff0-1be1-11ec-a722-83da1c22a481"
, so I will be copying the ID of879e8ff0-1be1-11ec-a722-83da1c22a481
Go to the file
detection_engine/scripts/legacy_notifications/one_action.json
and add this id to the file. Something like this:Query for an alert you want to add manually add back a legacy notification to it. Such as:
Grab the
_id
without the alert prefix. For mine this was933ca720-1be1-11ec-a722-83da1c22a481
Within the directory of detection_engine/scripts execute the script:
which is going to do a few things. See the file
detection_engine/routes/rules/legacy_create_legacy_notification.ts
for the definition of the route and what it does in full, but we should notice that we have now:Created a legacy side car action object of type
siem-detection-engine-rule-actions
you can see in dev tools:Take note that this actually creates the rule migrated since this PR updated the code to produce new side cars. So we have to use some scripting to change the actions to utilize the old format. However, before continuing you should verify that this does fire correctly and that the new format is working as expected. After that replace the structure with the older structure like so below and downgrade the migration version so that we can restart Kibana and ensure that this does migrate correctly forward:
Restart Kibana and now it should be migrated correctly and the system should fire the notifications as expected. You shouldn't see any errors in your console.
In the scripts folder execute the
find_rules.sh
and expect to see actions like so in the rule with theid
still in the REST interface and we shouldn't seeactionRef
within the actions:Take the rule id and query that as well using
./get_rule_by_id.sh
and verify that the action also looks the same and is present within the rule.You can also verify all of this within the UI's as well for rules to ensure the action is still present and as we expect it to be and work.
Checklist
Delete any items that are not applicable to this PR.