[SIEM][Detection Engine] Bulk REST API for create, update, and delete

[FrankHassanabad]

Summary

• Adds Bulk REST API and routes added for create, update, and delete
• Loops over data and calls alertClient until alerting team gives us bulk to push down
• Adds Unit tests

Testing/Usage:

Create in bulk:

POST /api/detection_engine/rules/_bulk_create
[{ ... rule_1}, { ... rule_2}]

see script

 ./post_rule_bulk.sh

Update in bulk:

PUT /api/detection_engine/rules/_bulk_update
[{ ... rule_1}, { ... rule_2}]

see script

 ./update_rule_bulk.sh

Delete in bulk:

DELETE /api/detection_engine/rules/_bulk_delete
[{"id": "rule_1"}, {"id": "rule_2"}]

or in case your client does not support bodies in DELETE

POST /api/detection_engine/rules/_bulk_delete
[{"id": "rule_1"}, {"id": "rule_2"}]

But try to use DELETE where possible

see script

./delete_bulk.sh

Caveats and error handling....If you do not validate correctly you will still get back a 400 return code. If you do validate correctly but one or more objects have errors involving a conflict or not found, you will get back a 200, but the body message will contain an array of errors or successes.

Examples:

If you delete in bulk for two objects but both do not have any values you get a response of 200 but then these errors in the array:

./delete_bulk.sh 
[
  {
    "id": "query-rule-id-1",
    "error": {
      "statusCode": 404,
      "message": "rule_id: \"query-rule-id-1\" not found"
    }
  },
  {
    "id": "query-rule-id-2",
    "error": {
      "statusCode": 404,
      "message": "rule_id: \"query-rule-id-2\" not found"
    }
  }
]

If one has a valid deleted value but the second does not you get a status of 200 and one object back as being deleted but the second as an error:

./delete_bulk.sh                        
[
  {
    "created_at": "2019-12-19T04:12:26.470Z",
    "updated_at": "2019-12-19T04:12:26.470Z",
    "created_by": "elastic_kibana",
    "description": "Query with a rule_id that acts like an external id",
    "enabled": true,
    "false_positives": [],
    "from": "now-6m",
    "id": "46d83e70-982a-4ba8-9ac1-fc386643c014",
    "immutable": false,
    "interval": "5m",
    "rule_id": "query-rule-id-1",
    "language": "kuery",
    "output_index": ".siem-signals-frank-hassanabad-default",
    "max_signals": 100,
    "risk_score": 1,
    "name": "Query with a rule id Number 1",
    "query": "user.name: root or user.name: admin",
    "references": [],
    "severity": "high",
    "updated_by": "elastic_kibana",
    "tags": [],
    "to": "now",
    "type": "query",
    "threats": [],
    "version": 1
  },
  {
    "id": "query-rule-id-2",
    "error": {
      "statusCode": 404,
      "message": "rule_id: \"query-rule-id-2\" not found"
    }
  }
]

Another example where an update has two errors because it tried to update two docs but both do not exist yet you get back a response of 200 but this array of errors:

./update_rule_bulk.sh                                                                                                                        <<<
[
  {
    "id": "query-rule-id-1",
    "error": {
      "statusCode": 404,
      "message": "rule_id: \"query-rule-id-1\" not found"
    }
  },
  {
    "id": "query-rule-id-2",
    "error": {
      "statusCode": 404,
      "message": "rule_id: \"query-rule-id-2\" not found"
    }
  }
]

If one is updatable but the other is not you get back a response of 200 and one update back but the other is an error:

./update_rule_bulk.sh
[
  {
    "created_at": "2019-12-19T04:15:43.739Z",
    "updated_at": "2019-12-19T04:16:01.633Z",
    "created_by": "elastic_kibana",
    "description": "Query with a rule_id that acts like an external id",
    "enabled": true,
    "false_positives": [],
    "from": "now-6m",
    "id": "c1fcea2c-cbc6-4f28-b7e7-32b4d7cb799d",
    "immutable": false,
    "interval": "5m",
    "rule_id": "query-rule-id-1",
    "language": "kuery",
    "output_index": ".siem-signals-frank-hassanabad-default",
    "max_signals": 100,
    "risk_score": 1,
    "name": "Rule id Number 1 with an updated name",
    "query": "user.name: root or user.name: admin",
    "references": [],
    "severity": "high",
    "updated_by": "elastic_kibana",
    "tags": [],
    "to": "now",
    "type": "query",
    "threats": [],
    "version": 2
  },
  {
    "id": "query-rule-id-2",
    "error": {
      "statusCode": 404,
      "message": "rule_id: \"query-rule-id-2\" not found"
    }
  }
]

Same thing goes with posts and any other bulk actions.

Checklist

Use strikethroughs to remove checklist items you don't feel are applicable to this PR.

- [ ] This was checked for cross-browser compatibility, including a check against IE11

- [ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support

- [ ] Documentation was added for features that require explanation or tutorials

• Unit or functional tests were updated or added to match the most common scenarios

- [ ] This was checked for keyboard-only and screenreader accessibility

For maintainers

- [ ] This was checked for breaking API changes and was labeled appropriately

• This includes a feature addition or change that requires a release note and was labeled appropriately

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 01669a9

History

• 💚 Build #16589 succeeded 7e12233

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream