Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] Assign proper diff algorithms to all rule fields #148191

Open
Tracked by #174168
banderror opened this issue Dec 29, 2022 · 2 comments
Open
Tracked by #174168

[Security Solution] Assign proper diff algorithms to all rule fields #148191

banderror opened this issue Dec 29, 2022 · 2 comments
Assignees
@dplumlee
Labels
8.15 candidate Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.

Comments

@banderror
Copy link
Contributor

banderror commented Dec 29, 2022 •
edited

Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168
Related to: #144060

Summary

In the Rule Customization RFC, we proposed different concrete diff algorithms for different types of rule fields. The goal of concrete diff algorithms is to improve the rule upgrade UX by trying to auto-merge user customizations with updates from Elastic.

Once the following algorithms are developed and assigned to a subset of rule fields: single-line string, multi-line string, number, array of objects, array of scalar values; we will need to review all the rule fields we have, which algorithms are used for which fields, and if there are any gaps.

For example, we might need to:

  • Assign already existing concrete algorithms to some fields that still use the simple diff algorithm.
  • Develop new concrete algorithms. It could be an algorithm for a specific field, e.g. MITRE mappings.

The end goal is to have all the rule fields assigned a proper diff algorithm.

Prior art
@banderror banderror added triage_needed Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules labels Dec 29, 2022
@banderror banderror self-assigned this Dec 29, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@banderror banderror removed their assignment Dec 29, 2022
@banderror banderror mentioned this issue Jan 5, 2023
Closed
@banderror banderror mentioned this issue Jan 27, 2023
Merged
banderror added a commit that referenced this issue Mar 8, 2023
@banderror
[Security Solution] Prebuilt rule upgrade and installation endpoints,…
03cdec3 
… initial implementation (#148392)

**Addresses:** #148181,
#148182,
#148185
**Partially addresses:**
#148183,
#148189

## Summary

Based on the [POC](#144060), this
PR adds 4 endpoints for the new upgrade and installation workflows for
prebuilt rules:

- `GET /internal/detection_engine/prebuilt_rules/status`
- `POST /internal/detection_engine/prebuilt_rules/upgrade/_review`
- `POST /internal/detection_engine/prebuilt_rules/installation/_review`
- `POST /internal/detection_engine/prebuilt_rules/_generate_assets`
(temporary helper endpoint for development and testing)

The new endpoints are hidden behind a feature flag and can be enabled by
the following config setting:

```yaml
xpack.securitySolution.enableExperimental: ['prebuiltRulesNewUpgradeAndInstallationWorkflowsEnabled']
```

## In the next episodes

Will be done later in follow-up PRs:

- Implementation of some additional response properties for the
`upgrade/_review` endpoint:
  - #148183
- Making base versions optional for diff calculation (we need to support
this in order to be able to still show diffs for rule assets coming from
packages without historical versions):
  - #148189
- Further development of the diff algorithm:
  - #148191
- Test coverage:
  - #148192
sloanelybutsurely pushed a commit to sloanelybutsurely/kibana that referenced this issue Mar 8, 2023
@banderror
[Security Solution] Prebuilt rule upgrade and installation endpoints,…
37bd66b 
… initial implementation (elastic#148392)

**Addresses:** elastic#148181,
elastic#148182,
elastic#148185
**Partially addresses:**
elastic#148183,
elastic#148189

## Summary

Based on the [POC](elastic#144060), this
PR adds 4 endpoints for the new upgrade and installation workflows for
prebuilt rules:

- `GET /internal/detection_engine/prebuilt_rules/status`
- `POST /internal/detection_engine/prebuilt_rules/upgrade/_review`
- `POST /internal/detection_engine/prebuilt_rules/installation/_review`
- `POST /internal/detection_engine/prebuilt_rules/_generate_assets`
(temporary helper endpoint for development and testing)

The new endpoints are hidden behind a feature flag and can be enabled by
the following config setting:

```yaml
xpack.securitySolution.enableExperimental: ['prebuiltRulesNewUpgradeAndInstallationWorkflowsEnabled']
```

## In the next episodes

Will be done later in follow-up PRs:

- Implementation of some additional response properties for the
`upgrade/_review` endpoint:
  - elastic#148183
- Making base versions optional for diff calculation (we need to support
this in order to be able to still show diffs for rule assets coming from
packages without historical versions):
  - elastic#148189
- Further development of the diff algorithm:
  - elastic#148191
- Test coverage:
  - elastic#148192
bmorelli25 pushed a commit to bmorelli25/kibana that referenced this issue Mar 10, 2023
@banderror @bmorelli25
[Security Solution] Prebuilt rule upgrade and installation endpoints,…
ebc623a 
… initial implementation (elastic#148392)

**Addresses:** elastic#148181,
elastic#148182,
elastic#148185
**Partially addresses:**
elastic#148183,
elastic#148189

## Summary

Based on the [POC](elastic#144060), this
PR adds 4 endpoints for the new upgrade and installation workflows for
prebuilt rules:

- `GET /internal/detection_engine/prebuilt_rules/status`
- `POST /internal/detection_engine/prebuilt_rules/upgrade/_review`
- `POST /internal/detection_engine/prebuilt_rules/installation/_review`
- `POST /internal/detection_engine/prebuilt_rules/_generate_assets`
(temporary helper endpoint for development and testing)

The new endpoints are hidden behind a feature flag and can be enabled by
the following config setting:

```yaml
xpack.securitySolution.enableExperimental: ['prebuiltRulesNewUpgradeAndInstallationWorkflowsEnabled']
```

## In the next episodes

Will be done later in follow-up PRs:

- Implementation of some additional response properties for the
`upgrade/_review` endpoint:
  - elastic#148183
- Making base versions optional for diff calculation (we need to support
this in order to be able to still show diffs for rule assets coming from
packages without historical versions):
  - elastic#148189
- Further development of the diff algorithm:
  - elastic#148191
- Test coverage:
  - elastic#148192
nkhristinin pushed a commit that referenced this issue Mar 22, 2023
@banderror @nkhristinin
[Security Solution] Prebuilt rule upgrade and installation endpoints,…
d6cdba7 
… initial implementation (#148392)

**Addresses:** #148181,
#148182,
#148185
**Partially addresses:**
#148183,
#148189

## Summary

Based on the [POC](#144060), this
PR adds 4 endpoints for the new upgrade and installation workflows for
prebuilt rules:

- `GET /internal/detection_engine/prebuilt_rules/status`
- `POST /internal/detection_engine/prebuilt_rules/upgrade/_review`
- `POST /internal/detection_engine/prebuilt_rules/installation/_review`
- `POST /internal/detection_engine/prebuilt_rules/_generate_assets`
(temporary helper endpoint for development and testing)

The new endpoints are hidden behind a feature flag and can be enabled by
the following config setting:

```yaml
xpack.securitySolution.enableExperimental: ['prebuiltRulesNewUpgradeAndInstallationWorkflowsEnabled']
```

## In the next episodes

Will be done later in follow-up PRs:

- Implementation of some additional response properties for the
`upgrade/_review` endpoint:
  - #148183
- Making base versions optional for diff calculation (we need to support
this in order to be able to still show diffs for rule assets coming from
packages without historical versions):
  - #148189
- Further development of the diff algorithm:
  - #148191
- Test coverage:
  - #148192
@banderror banderror mentioned this issue Jan 3, 2024
Open
@banderror banderror changed the title [Security Solution] Implement concrete field diff algorithms [Security Solution] Implement concrete field diff algorithms (DRAFT) Apr 17, 2024
@banderror banderror changed the title [Security Solution] Implement concrete field diff algorithms (DRAFT) [Security Solution] Assign proper diff algorithms to all rule fields May 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dplumlee dplumlee

Labels
8.15 candidate Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@banderror @elasticmachine @dplumlee