[Security Solution] Allow users to edit related_integrations field for custom rules #173595

approksiu · 2023-12-19T09:54:19Z

jpdjere · 2024-01-11T13:24:43Z

The related integrations field has the schema:

    RelatedIntegrationArray:
      type: array
      items:
        $ref: '#/components/schemas/RelatedIntegration'

    RelatedIntegration:
      type: object
      properties:
        package:
          $ref: '#/components/schemas/NonEmptyString'
        version:
          $ref: '#/components/schemas/NonEmptyString'
        integration:
          $ref: '#/components/schemas/NonEmptyString'
      required:
        - package
        - version

for example, for existing rules:

[
  {
    "name": "Linux Restricted Shell Breakout via Linux Binary(s)",
    "ri": [{ "package": "endpoint", "version": "^8.2.0" }]
  },
  {
    "name": "Potential Credential Access via DCSync",
    "ri": [
      { "package": "system", "version": "^1.6.4" },
      { "package": "windows", "version": "^1.5.0" }
    ]
  },
  {
    "name": "Microsoft Build Engine Using an Alternate Name",
    "ri": [
      { "package": "endpoint", "version": "^8.2.0" },
      { "package": "windows", "version": "^1.5.0" }
    ]
  },

  {
    "name": "Unusual AWS Command for a User",
    "ri": [{ "package": "aws", "version": "^2.0.0" }]
  },
  {
    "name": "First Time Seen AWS Secret Value Accessed in Secrets Manager",
    "ri": [
      { "package": "aws", "version": "^2.0.0", "integration": "cloudtrail" }
    ]
  },
  {
    "name": "AWS S3 Bucket Configuration Deletion",
    "ri": [
      { "package": "aws", "version": "^2.0.0", "integration": "cloudtrail" }
    ]
  }
]

Notice that the integration field is optional, and indeed some have it and some not. The integration field supercedes the package field what we show in the UI. For example, for the related integration:

 { "package": "aws", "version": "^2.0.0" }

We link to the page: /app/integrations/detail/aws-2.0.0/overview

while for the integration:

 { "package": "aws", "version": "^2.0.0", "integration": "cloudtrail" }

we link to: /app/integrations/detail/aws-2.0.0/overview?integration=cloudtrail

We can fetch all packages via the Fleet endpoint GET /kbn/api/fleet/epm/packages, and the response looks like:

{
    "items": [
        {
            "name": "aws",
            "title": "AWS",
            "version": "2.11.1",
            "release": "ga",
            "description": "Collect logs and metrics from Amazon Web Services (AWS) with Elastic Agent.",
            "type": "integration",
            "download": "/epr/aws/aws-2.11.1.zip",
            "path": "/package/aws/2.11.1",
            "icons": [
                {
                    "src": "/img/logo_aws.svg",
                    "path": "/package/aws/2.11.1/img/logo_aws.svg",
                    "title": "logo aws",
                    "size": "32x32",
                    "type": "image/svg+xml"
                }
            ],
            "policy_templates": [
                // more ingration policies
                {
                    "name": "cloudtrail",
                    "title": "AWS CloudTrail",
                    "description": "Collect AWS CloudTrail logs with Elastic Agent",
                    "icons": [
                        {
                            "src": "/img/logo_cloudtrail.svg",
                            "path": "/package/aws/2.11.1/img/logo_cloudtrail.svg",
                            "title": "AWS Cloudtrail logo",
                            "size": "32x32",
                            "type": "image/svg+xml"
                        }
                    ],
                    "categories": [
                        "security"
                    ]
                },
                {
                    "name": "cloudwatch",
                    "title": "AWS CloudWatch",
                    "description": "Use this integration to collect logs and metrics from Amazon CloudWatch with Elastic Agent, where no out of the box integration is available.",
                    "icons": [
                        {
                            "src": "/img/logo_cloudwatch.svg",
                            "path": "/package/aws/2.11.1/img/logo_cloudwatch.svg",
                            "title": "AWS CloudWatch logo",
                            "size": "32x32",
                            "type": "image/svg+xml"
                        }
                    ],
                    "categories": [
                        "observability",
                        "monitoring"
                    ]
                },
                // more ingration policies
            ],
            "conditions": {
                "kibana": {
                    "version": "^8.10.2"
                },
                "elastic": {
                    "subscription": "basic"
                }
            },
            "owner": {
                "type": "elastic",
                "github": "elastic/obs-ds-hosted-services"
            },
            "categories": [
                "aws"
            ],
            "signature_path": "/epr/aws/aws-2.11.1.zip.sig",
            "id": "aws",
            "status": "not_installed"
        },
        // more integration packages
        {
            "name": "windows",
            "title": "Windows",
            "version": "1.43.0",
            "release": "ga",
            "description": "Collect logs and metrics from Windows OS and services with Elastic Agent.",
            "type": "integration",
            "download": "/epr/windows/windows-1.43.0.zip",
            "path": "/package/windows/1.43.0",
            "icons": [
                {
                    "src": "/img/logo_windows.svg",
                    "path": "/package/windows/1.43.0/img/logo_windows.svg",
                    "title": "logo windows",
                    "size": "32x32",
                    "type": "image/svg+xml"
                }
            ],
            "policy_templates": [
                {
                    "name": "windows",
                    "title": "Windows logs and metrics",
                    "description": "Collect logs and metrics from Windows instances"
                }
            ],
            "conditions": {
                "kibana": {
                    "version": "^8.8.0"
                }
            },
            "owner": {
                "github": "elastic/elastic-agent-data-plane"
            },
            "categories": [
                "os_system",
                "security"
            ],
            "signature_path": "/epr/windows/windows-1.43.0.zip.sig",
            "id": "windows",
            "status": "not_installed"
        },
        
    ]
}

So:

the top level objects correspond to packages, which have info that we might need such as version and description.
each of those packages has a policy_templates array whose elements correspond to integrations in our schema. For example, AWS Cloudtrail.

Open questions:

How should the editing component for this field look like?
How do we handle the version? Do we just automatically select the previous minor version, major version? Do we allow the user to type in what they want? (This is important for the Related Integration component, where we show if the installation is installed or not)

approksiu · 2024-01-11T15:59:08Z

For installed integrations - select current version, for non-installed - latest version.
Do we show installed and non-installed integrations, and do we show only security category or others too?

approksiu · 2024-01-11T16:02:28Z

Consider showing packages and integrations in the "flat" drop-down.

approksiu · 2024-01-18T13:14:27Z

Do we show installed and non-installed integrations, and do we show only security category or others too?

We should show all security integrations. Would be nice to indicate which ones are installed/not installed - maybe as a follow up? cc @ARWNightingale
It should be possible to search by name

jpdjere · 2024-01-18T16:21:28Z

Great @approksiu .

So we just met with @ARWNightingale and discussed this.

I think it makes sense to give the user a dropdown of all possible related integrations, pulling the list from Fleet. At the same time, we should show the user which of those integrations are installed, with some kind of label on each integration's row, if it is actually installed. Maybe Alex you can think an example of how this should look like.

Also, integrations that are installed should be pushed up to the top of the list, because we can presume that the user will be selecting one of those with much higher probability.

Makes sense to you @approksiu ?

approksiu · 2024-01-19T10:34:31Z

@jpdjere totally, thanks!

elasticmachine · 2024-03-07T14:44:09Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2024-03-07T14:44:09Z

Pinging @elastic/security-solution (Team: SecuritySolution)

approksiu · 2024-03-26T10:16:01Z

@maximpn my preference is Design 2 option from the figma file. Let's pre-fill version value if possible for the user to accept or modify. Thanks!

maximpn · 2024-03-26T10:19:22Z

@approksiu thank you for an update 👍

I attach the current Design 2 Figma snapshot for quick reference:

banderror · 2024-04-17T10:33:17Z

@maximpn Since #178295 has been approved by reviewers (I'd like to take a look, but it shouldn't block the work on the ticket), I think we should notify the stakeholders that work on acceptance testing, exploratory testing, and documentation can start.

Once you have the PR deployed to Cloud for testing, please ping @approksiu, @ARWNightingale, and @vgomez-el in this thread and share credentials with them.

Please also open a documentation ticket for related_integrations similar to elastic/security-docs#4917 and elastic/security-docs#5029 and ping @joepeeples here in this thread.

It would be great if you could also try to align with the stakeholders on timing and plan a release date in Serverless.

maximpn · 2024-04-18T10:29:32Z

@approksiu, @ARWNightingale, and @vgomez-el could you perform testing of #178295?

ESS environment. Use default credentials or reach me in Slack if have any troubles.
~~Serverless environment. Credentials: https://p.elstc.co/paste/wEkkdXGx#9NCz27Z6HV1NfuNbkOvKn+5fLrauQCZM6SLEWD3DNAd.~~

@banderror there is a problem with automatic cloud deploy via tags. I had to use another approach to deploy my PR. It's only possible to deploy in ESS env atm.

UPDATE: Serverless cloud deploy works well. I updated the information above.

UPDATE 2: It seems Serverless cloud deploy stoped working.

approksiu · 2024-04-18T13:18:07Z

Great work Maxim, looks good to me!

ARWNightingale · 2024-04-26T13:18:12Z

Looks good to me!!
Just a note for future enhancements maybe we should look at disabling the add + option when we have an empty input so the user can not add more than 1 empty inputs, but this is across all similar interactions elsewhere too.

vgomez-el · 2024-04-29T11:25:52Z

@approksiu @banderror According to the issue title, user Story and @maximpn's PR description, I understand that this feature is only available for custom rules, but If I select another rule type rather than Custom rules, the related integrations option is displayed as well:
New Terms:

Indicator Match:

ES|QL:

Is that intended? In case it is, it should be reflected on any of the existing issues, because the current issues description can lead to confusion.

maximpn · 2024-04-29T15:07:41Z

Hi @vgomez-el!

Sorry for the confusion. Custom rules mean NOT Prebuilt rules since it's a prerequisite for Milestone 3 prebuilt rules customization. It's applicable to all rule types. Word custom is to only highlight we don't touch prebuilt rules.

vgomez-el · 2024-04-30T14:59:42Z

Sorry for not understanding it correctly, @maximpn. I have performed exploratory testing on your environment and the feature looks nice to me. I will perform the Kibana Upgrade scenarios and the CCS scenarios once I have the first build candidate of the 8.15 version. Please, Ping me if someone needs more details on the exploratory testing performed.

joepeeples · 2024-05-01T18:04:45Z

@maximpn and all, a docs PR is now open for serverless docs: https://github.com/elastic/staging-serverless-security-docs/pull/337

…r custom rules (#178295) **Resolves: #173595 ## Summary This PR adds an ability to add and edit custom rule's related integrations. Functionality is necessary to start working on [Prebuilt Rule Customization Epic Milestone 3](#174168). ## Details Rule's related integrations represent optional dependencies on [Elastic integrations](https://docs.elastic.co/en/integrations) to ingest data. Currently prebuilt rule's related integrations are shown on rule details page. This information contains integration's name, installation status and a version mismatch warning when related integration's version dependency doesn't match with an installed integration's version. A subset of [Semver](https://semver.org/) is used to specify version dependency. Elastic prebuilt rules use only caret syntax like `^1.2.3`. To make it possible to add and edit related integrations for custom rules the following has been done - New internal endpoint `/internal/detection_engine/fleet/integrations/all` has been added. It returns the full list of available integrations containing title, latest available version and installed version if available. This is necessary to display an options list where users can pick a desired integration. Since some Elastic Prebuilt rules depend not only on integrations from `security` category this endpoint returns all available integrations (not only related to Security Solution). - Rule create form has been adjusted by adding `Related Integrations` form controls - Rule edit form has been adjusted by adding `Related Integrations` form controls - Related integrations installation status has been adjusted to conform with the design - Functional Jest tests have been added - Functional tests have been added to make sure it's possible to (bulk) `create`/`patch`/`update`/`export`/`import` with related integrations - A limited number of Cypress tests have been added ### Integration installation status Integration installation status has been adjusted. There are following statuses shown - `Enabled` for installed and enabled integrations. Enabled integrations are detected by checking Elastic Agent policies for presence of such an integration. It's not guaranteed the policy is picked by agents and data is being ingested. - `Disabled` for installed and disabled integrations. An agent policy containing such an integration isn't found. - `Not installed` for not installed integrations. - Nothing is shown for unknown integrations. If there is no such integration found in `/internal/detection_engine/fleet/integrations/all` result it's considered as unknown. ### Version dependency [Semver](https://semver.org/) allows a wide range of version range declaration. Such flexibility will complicate constructing of an integration link on rule details page. Since Elastic Prebuilt rules use only caret version dependency like `^1.2.3` related integration's version dependency is limited to a subset of semver semantic. The following is supported - A plain version dependency e.g. `1.2.3` - Tilde version dependency e.g. `~1.2.3` - Caret version dependency e.g. `^1.2.3` ### Misc - #152408 has been fixed by this PR. - `/internal/detection_engine/fleet/integrations/installed` endpoint hasn't been removed. We need to make sure it's not needed anymore. - E2e testing of the current functionality is complicated by dependency on EPR and difficulties to mock it. EPR periodically may respond with an error resulting in flaky Cypress tests. ### Flaky test runner results - 🟢 [Create rule](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5632) (100 runs ESS and 100 runs in Serverless) - 🟢 [Rule Management related integrations](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5632) (100 runs ESS and 100 runs in Serverless) ### Screenshots ![Screenshot 2024-04-16 at 10 01 25](https://github.com/elastic/kibana/assets/3775283/f41574cb-c806-4e49-97bf-9b27bf4c0f39) ![Screenshot 2024-04-16 at 10 02 03](https://github.com/elastic/kibana/assets/3775283/cf15580e-169f-4823-a579-257509c806a4) ![Screenshot 2024-04-16 at 10 02 16](https://github.com/elastic/kibana/assets/3775283/03a21eea-1014-484f-b1d2-3db81c46b8ef) ![Screenshot 2024-04-16 at 10 04 19](https://github.com/elastic/kibana/assets/3775283/06385ef4-458f-4562-bb8f-d98db9bb1fe3) ![Screenshot 2024-04-16 at 10 02 33](https://github.com/elastic/kibana/assets/3775283/edec85bf-d020-4afb-a999-4eb21255c3b6) ![Screenshot 2024-04-16 at 10 04 40](https://github.com/elastic/kibana/assets/3775283/a21c55a8-9947-44b0-ba1f-6624cd410d3e) ![Screenshot 2024-04-16 at 10 05 03](https://github.com/elastic/kibana/assets/3775283/05928a15-961b-4f67-9968-d2a32ceb86dc)

…r custom rules (elastic#178295) **Resolves: elastic#173595 ## Summary This PR adds an ability to add and edit custom rule's related integrations. Functionality is necessary to start working on [Prebuilt Rule Customization Epic Milestone 3](elastic#174168). ## Details Rule's related integrations represent optional dependencies on [Elastic integrations](https://docs.elastic.co/en/integrations) to ingest data. Currently prebuilt rule's related integrations are shown on rule details page. This information contains integration's name, installation status and a version mismatch warning when related integration's version dependency doesn't match with an installed integration's version. A subset of [Semver](https://semver.org/) is used to specify version dependency. Elastic prebuilt rules use only caret syntax like `^1.2.3`. To make it possible to add and edit related integrations for custom rules the following has been done - New internal endpoint `/internal/detection_engine/fleet/integrations/all` has been added. It returns the full list of available integrations containing title, latest available version and installed version if available. This is necessary to display an options list where users can pick a desired integration. Since some Elastic Prebuilt rules depend not only on integrations from `security` category this endpoint returns all available integrations (not only related to Security Solution). - Rule create form has been adjusted by adding `Related Integrations` form controls - Rule edit form has been adjusted by adding `Related Integrations` form controls - Related integrations installation status has been adjusted to conform with the design - Functional Jest tests have been added - Functional tests have been added to make sure it's possible to (bulk) `create`/`patch`/`update`/`export`/`import` with related integrations - A limited number of Cypress tests have been added ### Integration installation status Integration installation status has been adjusted. There are following statuses shown - `Enabled` for installed and enabled integrations. Enabled integrations are detected by checking Elastic Agent policies for presence of such an integration. It's not guaranteed the policy is picked by agents and data is being ingested. - `Disabled` for installed and disabled integrations. An agent policy containing such an integration isn't found. - `Not installed` for not installed integrations. - Nothing is shown for unknown integrations. If there is no such integration found in `/internal/detection_engine/fleet/integrations/all` result it's considered as unknown. ### Version dependency [Semver](https://semver.org/) allows a wide range of version range declaration. Such flexibility will complicate constructing of an integration link on rule details page. Since Elastic Prebuilt rules use only caret version dependency like `^1.2.3` related integration's version dependency is limited to a subset of semver semantic. The following is supported - A plain version dependency e.g. `1.2.3` - Tilde version dependency e.g. `~1.2.3` - Caret version dependency e.g. `^1.2.3` ### Misc - elastic#152408 has been fixed by this PR. - `/internal/detection_engine/fleet/integrations/installed` endpoint hasn't been removed. We need to make sure it's not needed anymore. - E2e testing of the current functionality is complicated by dependency on EPR and difficulties to mock it. EPR periodically may respond with an error resulting in flaky Cypress tests. ### Flaky test runner results - 🟢 [Create rule](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5632) (100 runs ESS and 100 runs in Serverless) - 🟢 [Rule Management related integrations](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5632) (100 runs ESS and 100 runs in Serverless) ### Screenshots ![Screenshot 2024-04-16 at 10 01 25](https://github.com/elastic/kibana/assets/3775283/f41574cb-c806-4e49-97bf-9b27bf4c0f39) ![Screenshot 2024-04-16 at 10 02 03](https://github.com/elastic/kibana/assets/3775283/cf15580e-169f-4823-a579-257509c806a4) ![Screenshot 2024-04-16 at 10 02 16](https://github.com/elastic/kibana/assets/3775283/03a21eea-1014-484f-b1d2-3db81c46b8ef) ![Screenshot 2024-04-16 at 10 04 19](https://github.com/elastic/kibana/assets/3775283/06385ef4-458f-4562-bb8f-d98db9bb1fe3) ![Screenshot 2024-04-16 at 10 02 33](https://github.com/elastic/kibana/assets/3775283/edec85bf-d020-4afb-a999-4eb21255c3b6) ![Screenshot 2024-04-16 at 10 04 40](https://github.com/elastic/kibana/assets/3775283/a21c55a8-9947-44b0-ba1f-6624cd410d3e) ![Screenshot 2024-04-16 at 10 05 03](https://github.com/elastic/kibana/assets/3775283/05928a15-961b-4f67-9968-d2a32ceb86dc)

banderror · 2024-05-08T19:32:12Z

The feature went live in Serverless today 🚀 Thanks to everyone for your efforts!

approksiu added the Team:Detection Rule Management Security Detection Rule Management Team label Dec 19, 2023

approksiu mentioned this issue Jan 2, 2024

[Security Solution] Discuss rule fields in the context of the prebuilt rule upgrade workflow #147239

Closed

approksiu changed the title ~~[Security Solution] Allow users to edit related_integrations field for rules~~ [Security Solution] Allow users to edit related_integrations field for custom rules Jan 2, 2024

ARWNightingale self-assigned this Jan 8, 2024

jpdjere mentioned this issue Jan 11, 2024

[Security Solution] Users can Customize Prebuilt Detection Rules: Milestone 3 #174168

Open

banderror assigned maximpn Mar 4, 2024

banderror mentioned this issue Mar 7, 2024

[Security Solution] Allow users to edit setup field for custom rules #173626

Closed

8 tasks

maximpn mentioned this issue Mar 8, 2024

[Security Solution] Allow users to edit related_integrations field for custom rules #178295

Merged

banderror added the Feature:Rule Edit label Apr 13, 2024

maximpn added 8.15 candidate and removed 8.14 candidate labels Apr 15, 2024

maximpn mentioned this issue Apr 18, 2024

Edit related_integrations field for custom rules in UI and API elastic/security-docs#5099

Closed

banderror added the v8.15.0 label Apr 19, 2024

maximpn closed this as completed in #178295 May 2, 2024

maximpn reopened this May 2, 2024

maximpn mentioned this issue May 8, 2024

Updates related_integrations field API docs elastic/security-docs#5183

Merged

banderror closed this as completed May 8, 2024

maximpn mentioned this issue Jun 29, 2024

[Security Solution] Integration status isn't shown for some rules #187199

Open

mergify bot mentioned this issue Jul 5, 2024

[8.15] Updates related_integrations field API docs (backport #5183) elastic/security-docs#5509

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Allow users to edit related_integrations field for custom rules #173595

[Security Solution] Allow users to edit related_integrations field for custom rules #173595

approksiu commented Dec 19, 2023 •

edited by banderror

Loading

jpdjere commented Jan 11, 2024 •

edited

Loading

approksiu commented Jan 11, 2024

approksiu commented Jan 11, 2024

approksiu commented Jan 18, 2024 •

edited

Loading

jpdjere commented Jan 18, 2024 •

edited

Loading

approksiu commented Jan 19, 2024

elasticmachine commented Mar 7, 2024

elasticmachine commented Mar 7, 2024

approksiu commented Mar 26, 2024

maximpn commented Mar 26, 2024 •

edited

Loading

banderror commented Apr 17, 2024

maximpn commented Apr 18, 2024 •

edited

Loading

approksiu commented Apr 18, 2024

ARWNightingale commented Apr 26, 2024

vgomez-el commented Apr 29, 2024

maximpn commented Apr 29, 2024 •

edited

Loading

vgomez-el commented Apr 30, 2024 •

edited

Loading

joepeeples commented May 1, 2024

banderror commented May 8, 2024

[Security Solution] Allow users to edit related_integrations field for custom rules #173595

[Security Solution] Allow users to edit related_integrations field for custom rules #173595

Comments

approksiu commented Dec 19, 2023 • edited by banderror Loading

Summary

Background

User story

Acceptance criteria

Designs

Release progress

jpdjere commented Jan 11, 2024 • edited Loading

Open questions:

approksiu commented Jan 11, 2024

approksiu commented Jan 11, 2024

approksiu commented Jan 18, 2024 • edited Loading

jpdjere commented Jan 18, 2024 • edited Loading

approksiu commented Jan 19, 2024

elasticmachine commented Mar 7, 2024

elasticmachine commented Mar 7, 2024

approksiu commented Mar 26, 2024

maximpn commented Mar 26, 2024 • edited Loading

banderror commented Apr 17, 2024

maximpn commented Apr 18, 2024 • edited Loading

approksiu commented Apr 18, 2024

ARWNightingale commented Apr 26, 2024

vgomez-el commented Apr 29, 2024

maximpn commented Apr 29, 2024 • edited Loading

vgomez-el commented Apr 30, 2024 • edited Loading

joepeeples commented May 1, 2024

banderror commented May 8, 2024

approksiu commented Dec 19, 2023 •

edited by banderror

Loading

jpdjere commented Jan 11, 2024 •

edited

Loading

approksiu commented Jan 18, 2024 •

edited

Loading

jpdjere commented Jan 18, 2024 •

edited

Loading

maximpn commented Mar 26, 2024 •

edited

Loading

maximpn commented Apr 18, 2024 •

edited

Loading

maximpn commented Apr 29, 2024 •

edited

Loading

vgomez-el commented Apr 30, 2024 •

edited

Loading