Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose index connector and action in custom security detection configuration #110550

Closed
jethr0null opened this issue Aug 30, 2021 · 5 comments
Closed
Assignees
Labels
enhancement New value added to drive a business result epic Feature:Detection Rules Anything related to Security Solution's Detection Rules Feature:Rule Actions Security Solution Rule Actions feature Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v7.16.0

Comments

@jethr0null
Copy link

jethr0null commented Aug 30, 2021

Kibana alerting users are able to utilize the index connector and action when configuring alerts. We should explore exposing this capability when configuring custom security detections (rules).

This connector should index a document per alert (vs. a single document for all alerts generated by a rule execution).

Use cases for multi-document indexing:

  • Limit data access by applying DLS security on each per signal trigger.
  • Chain detection rules together (dynamic mapping is not allowed in siem signals and this is a workaround requiring individual documents)
@jethr0null jethr0null added enhancement New value added to drive a business result Feature:Detection Rules Anything related to Security Solution's Detection Rules Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Feature:Rule Actions Security Solution Rule Actions feature 7.16 candidate labels Aug 30, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@jethr0null jethr0null changed the title [Detections] Expose index connector and action in custom security detection configuration Expose index connector and action in custom security detection configuration Aug 30, 2021
@spong
Copy link
Member

spong commented Sep 10, 2021

Created a draft PR adding this functionality (it's a 1-liner 🙂) and in testing everything seems to be functional 👍.

Only thing of note here is that it doesn't appear to be possible to index multiple documents per action firing, so a rule execution could create 100 alerts (to .siem-signals-default), yet the user would only be able to index a single document using all the data from those 100 alerts. Perhaps it's possible to wrap the action template with the mustache looping syntax (i.e. {{#context.alerts}}{{/context.alerts}}), but I wasn't able to get any variant of this to pass validation in order to save the rule and test (API validation TBD, so maybe that way?).

Without the ability to write multiple documents per action firing, I question the usefulness of this feature and if it'll just prove to be more of a headache to users expecting a larger degree of flexibility here. That said, it can at least be leveraged to write data to arbitrary indices, so more functionality/flexibility than is available today.

At this point I think it's a Product/UX decision on if we want to move forward here as-is, or include prioritization of multi-document support as part of this effort. Would need to check with @elastic/kibana-alerting-services on LOE there since they own this connector, but wouldn't imagine this to be too large of an effort, just impactful since it would most likely change the accepted format from valid JSON.

cc @jethr0null @elastic/security-design

@mikecote
Copy link
Contributor

From doing a quick search, it seems correct, I don't think we have a workaround at this time other than solving the problem. The @elastic/kibana-alerting-services team is tracking such limitations under #89430.

@spong
Copy link
Member

spong commented Oct 11, 2021

As mentioned in closed draft PR (#111813), will hold off on implementing until dependency #89430 is resolved and we can support indexing multiple documents per action execution context.

Got 👍 from product to add for 7.16 -- merged in #111813.

spong added a commit that referenced this issue Oct 19, 2021
… Detection Actions (#111813)

## Summary

This PR enables the [Index Connector and Action](https://www.elastic.co/guide/en/kibana/master/index-action-type.html) for the detection engine, addressing #110550. 

<details><summary>Action type available in list:</summary>
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132774871-285ff387-eebe-44ee-9172-3143d0283b09.png" />
</p>
</details>

<details><summary>No Connector UI:</summary>
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132774935-e7e71061-e3b6-40ae-bc77-4adb76132de0.png" />
</p>
</details>


<details><summary>Create Connector UI:</summary>
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132774985-50980dcf-b022-4101-809a-a2d5a617a892.png" />
</p>
</details>

<details><summary>Connector Template:</summary>
<p>

``` json
{
    "rule_id": "{{context.rule.id}}",
    "rule_name": "{{context.rule.name}}",
    "alert_id": "{{alert.id}}",
    "context_message": "Threshold Results: {{#context.alerts}}{{#signal.threshold_result.terms}}{{value}}, {{/signal.threshold_result.terms}}{{/context.alerts}}"
}
```

<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132775311-3091ed5d-d7df-4dc1-89d6-c02a93c04779.png" />
</p>
</p>
</details>



<details><summary>Documents successfully written:</summary>
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132775457-128f0e62-0978-46a6-ae6e-bd951a1d6c96.png" />
</p>
</details>


---

If wanting to store the alert index timestamp, create index first with `timestamp` field and use `Define timefield for each document` option:


```
PUT .homemade-alerts-index
{
  "mappings" : {
    "dynamic": "true",
    "properties" : {
      "@timestamp": {
        "type": "date"
      }
    }
  }
}
```
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132775842-ac6534a7-289d-426f-851b-f5f2c2bb5716.png" />
</p>

<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132776120-2def172f-3bfa-4a7e-b041-155e817173ab.png" />
</p>




### Checklist

Delete any items that are not applicable to this PR.

- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials (will need to update documentation if we proceed with this PR)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Oct 19, 2021
… Detection Actions (elastic#111813)

## Summary

This PR enables the [Index Connector and Action](https://www.elastic.co/guide/en/kibana/master/index-action-type.html) for the detection engine, addressing elastic#110550. 

<details><summary>Action type available in list:</summary>
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132774871-285ff387-eebe-44ee-9172-3143d0283b09.png" />
</p>
</details>

<details><summary>No Connector UI:</summary>
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132774935-e7e71061-e3b6-40ae-bc77-4adb76132de0.png" />
</p>
</details>


<details><summary>Create Connector UI:</summary>
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132774985-50980dcf-b022-4101-809a-a2d5a617a892.png" />
</p>
</details>

<details><summary>Connector Template:</summary>
<p>

``` json
{
    "rule_id": "{{context.rule.id}}",
    "rule_name": "{{context.rule.name}}",
    "alert_id": "{{alert.id}}",
    "context_message": "Threshold Results: {{#context.alerts}}{{#signal.threshold_result.terms}}{{value}}, {{/signal.threshold_result.terms}}{{/context.alerts}}"
}
```

<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132775311-3091ed5d-d7df-4dc1-89d6-c02a93c04779.png" />
</p>
</p>
</details>



<details><summary>Documents successfully written:</summary>
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132775457-128f0e62-0978-46a6-ae6e-bd951a1d6c96.png" />
</p>
</details>


---

If wanting to store the alert index timestamp, create index first with `timestamp` field and use `Define timefield for each document` option:


```
PUT .homemade-alerts-index
{
  "mappings" : {
    "dynamic": "true",
    "properties" : {
      "@timestamp": {
        "type": "date"
      }
    }
  }
}
```
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132775842-ac6534a7-289d-426f-851b-f5f2c2bb5716.png" />
</p>

<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132776120-2def172f-3bfa-4a7e-b041-155e817173ab.png" />
</p>




### Checklist

Delete any items that are not applicable to this PR.

- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials (will need to update documentation if we proceed with this PR)
kibanamachine added a commit that referenced this issue Oct 19, 2021
… Detection Actions (#111813) (#115563)

## Summary

This PR enables the [Index Connector and Action](https://www.elastic.co/guide/en/kibana/master/index-action-type.html) for the detection engine, addressing #110550. 

<details><summary>Action type available in list:</summary>
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132774871-285ff387-eebe-44ee-9172-3143d0283b09.png" />
</p>
</details>

<details><summary>No Connector UI:</summary>
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132774935-e7e71061-e3b6-40ae-bc77-4adb76132de0.png" />
</p>
</details>


<details><summary>Create Connector UI:</summary>
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132774985-50980dcf-b022-4101-809a-a2d5a617a892.png" />
</p>
</details>

<details><summary>Connector Template:</summary>
<p>

``` json
{
    "rule_id": "{{context.rule.id}}",
    "rule_name": "{{context.rule.name}}",
    "alert_id": "{{alert.id}}",
    "context_message": "Threshold Results: {{#context.alerts}}{{#signal.threshold_result.terms}}{{value}}, {{/signal.threshold_result.terms}}{{/context.alerts}}"
}
```

<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132775311-3091ed5d-d7df-4dc1-89d6-c02a93c04779.png" />
</p>
</p>
</details>



<details><summary>Documents successfully written:</summary>
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132775457-128f0e62-0978-46a6-ae6e-bd951a1d6c96.png" />
</p>
</details>


---

If wanting to store the alert index timestamp, create index first with `timestamp` field and use `Define timefield for each document` option:


```
PUT .homemade-alerts-index
{
  "mappings" : {
    "dynamic": "true",
    "properties" : {
      "@timestamp": {
        "type": "date"
      }
    }
  }
}
```
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132775842-ac6534a7-289d-426f-851b-f5f2c2bb5716.png" />
</p>

<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/132776120-2def172f-3bfa-4a7e-b041-155e817173ab.png" />
</p>




### Checklist

Delete any items that are not applicable to this PR.

- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials (will need to update documentation if we proceed with this PR)

Co-authored-by: Garrett Spong <spong@users.noreply.github.com>
@spong spong closed this as completed Nov 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New value added to drive a business result epic Feature:Detection Rules Anything related to Security Solution's Detection Rules Feature:Rule Actions Security Solution Rule Actions feature Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v7.16.0
Projects
None yet
Development

No branches or pull requests

5 participants