[Detections Response] Finish moving remaining legacy FTRs #175837

yctercero · 2024-01-29T19:41:14Z

Resolves: #151902

Summary

After this PR, all D&R FTRs are moved to new folder where they can be run in ESS and serverless. Please see below table for a summary of what tests need revisiting by the teams. During the test migration there may have been some tests that failed on serverless, but not ESS. Some we were able to fix and get running on both, others are still marked as brokenInServerless and need triage.

Detection Engine FTRs

Test	ESS	Serverless	Notes
/actions/trial_license_complete_tier/check_privileges.ts	✅	❌
/exceptions/workflows/trial_license_complete_tier/create_rule_exceptions.ts	✅	❌	One test in suite is `brokenInServerless`
/exceptions/workflows/trial_license_complete_tier/role_based_add_edit_comments.ts	✅	❌
/exceptions/workflows/trial_license_complete_tier/role_based_rule_exceptions_workflows.ts	✅	❌	One test in suite is `brokenInServerless`

Rules Management FTRs

Test	ESS	Serverless	Notes
/rule_bulk_actions/trial_license_complete_tier/perform_bulk_action.ts	✅	❌
/rule_creation/trial_license_complete_tier/create_rules_bulk.ts	✅	❌
/rule_creation/trial_license_complete_tier/create_rules.ts	✅	❌	Three tests in suite is `brokenInServerless`
/rule_creation/trial_license_complete_tier/preview_rules.ts	✅	❌	Two tests in suite is `brokenInServerless`
/rule_delete/trial_license_complete_tier/delete_rules_bulk.ts	✅	❌
/rule_import_export/trial_license_complete_tier/export_rules.ts	✅	❌
/rule_import_export/trial_license_complete_tier/import_rules.ts	✅	❌
/rule_patch/basic_license_essentials_tier/patch_rules.ts	✅	❌	One test in suite is `brokenInServerless`
/rule_patch/trial_license_complete_tier/patch_rules_bulk.ts	✅	❌
/rule_patch/trial_license_complete_tier/patch_rules.ts	✅	❌	Four tests in suite is `brokenInServerless`
/rule_read/trial_license_complete_tier/read_rules.ts	✅	❌	Two tests in suite is `brokenInServerless`
/rule_update/basic_license_essentials_tier/update_rules.ts	✅	❌	One test in suite is `brokenInServerless`
/rule_update/trial_license_complete_tier/update_rules_bulk.ts	✅	❌
/rule_update/trial_license_complete_tier/update_rules.ts	✅	❌	Four tests in suite is `brokenInServerless`

Shared D&R FTRs

Test	ESS	Serverless	Notes
/telemetry/trial_license_complete_tier/usage_collector/detection_rules.ts	✅	❌	14 tests in suite is `brokenInServerless`
/user_roles/trial_license_complete_tier/read_privileges.ts	✅	❌

…ftrs

elasticmachine · 2024-01-29T19:41:52Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2024-01-29T19:41:52Z

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

elasticmachine · 2024-01-29T19:41:52Z

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine · 2024-01-29T19:41:52Z

Pinging @elastic/security-detection-engine (Team:Detection Engine)

…ftrs

banderror · 2024-01-31T12:59:32Z

Please see below table for a summary of what tests need revisiting by the teams. During the test migration there may have been some tests that failed on serverless, but not ESS. Some we were able to fix and get running on both, others are still marked as brokenInServerless and need triage.

Thank you for compiling this info into a table @yctercero, this will be very helpful in the future! Once the PR is merged, I'll open follow-up tickets for @elastic/security-detection-rule-management to revisit and unskip the tests that were marked as @brokenInServerless.

banderror

Restructuring tests is a huge undertaking, I really appreciate all the work you've done in this PR, in #175417, and other prior PRs. Thank you so much @yctercero!

The PR LGTM 👍 I left some concerns that I'm ok to address in follow-up PRs myself. Nothing blocking from my side.

I will defer approving it to @maximpn since he was assigned as a reviewer.

.buildkite/ftr_configs.yml

.buildkite/pipelines/security_solution/api_integration.yml

...ctions_response/rules_management/rule_creation/basic_license_essentials_tier/create_rules.ts

…ftrs

maximpn

@yctercero thank you for moving FTR tests in the new folder 🙏

I totally agree with Georgii's comment https://github.com/elastic/kibana/pull/175837/files#r1472814985 but I don't we should address all that issues in this PR. It should be focused on moving tests with minimal changes so it doesn't block any other PR requiring changing/adding FTR tests. The issues mentioned should be addressed in follow up PRs.

I see you made some little changes by using updateUsername() and similar. It doesn't look to be necessary (correct me if I'm wrong here) for this PR. Ideally we need to get rid of removeServerGeneratedProperties(), updateUsername(), getSimpleRuleOutputWithoutRuleId() and etc in favor of Jest expect with partial matching like expect().toMatchObject() and objectContaining to focus on the changes the test does. It's a huge amount of work so it looks logical do this refactoring only when a test touched (updated/fixed) or a new test added.

As I don't see any critical changes required I approve this PR.

x-pack/plugins/security_solution/common/experimental_features.ts

…fix'

…ftrs

…ary changes

yctercero · 2024-02-09T01:21:52Z

@elasticmachine merge upstream

…o move_remaining_ftrs

…ftrs

…-fix'

…ftrs

…o move_remaining_ftrs

…-fix'

yctercero · 2024-02-12T01:11:05Z

@elasticmachine merge upstream

rylnd

I had a few recommendations for cleaning up some of the cruft introduced by the expansion of these FTRs into the serverless era, but they were mostly just resurrecting @banderror's previous insights 😅 .

Kudos on finding and calling out the test duplication as well; that's easily overlooked. However, since these PRs are not introducing the duplication (rather, they're just not fixing it), I would agree that it can be fixed down the line.

Thanks for the helpful PR description and comment responses; this one was a lot so those are very much appreciated :)

rylnd · 2024-02-12T16:00:54Z

...s_response/rules_management/rule_creation/basic_license_essentials_tier/create_rules_bulk.ts

 export default ({ getService }: FtrProviderContext): void => {
  const esArchiver = getService('esArchiver');
  const supertest = getService('supertest');
  const log = getService('log');
  const es = getService('es');
-
-  describe('create_rules_bulk', () => {
+  // TODO: add a new service for loading archiver files similar to "getService('es')"


I agree with georgii that we probably don't need to keep adding this comment, but I appreciate the commitment!

rylnd · 2024-02-12T16:33:43Z

...ons_response/rules_management/rule_delete/basic_license_essentials_tier/delete_rules_bulk.ts

@@ -51,7 +55,9 @@ export default ({ getService }: FtrProviderContext): void => {
          .expect(200);

        const bodyToCompare = removeServerGeneratedProperties(body[0]);
-        expect(bodyToCompare).to.eql(getSimpleRuleOutput());
+        const expectedRule = updateUsername(getSimpleRuleOutput(), ELASTICSEARCH_USERNAME);


Seeing all these updateUsernames added here doesn't feel great; I'll reiterate @banderror's previous comment pointing out that we now modify both the actual value and the expected value, which does seem a little silly.

I really like expect.objectContaining as referenced, but it can definitely get a little verbose and hard to read; maybe we could hide some of that verbosity in a declarative matcher? It's not obvious how one would go about adding a matcher to kbn-expect, but we could certainly just write a function that compares the appropriate fields and ignores the rest:

assertRuleEquality({ actual: body[0], expected: getSimpleRuleOutput() })

I'll look to create some follow up issues to track these suggestions!

yctercero · 2024-02-13T05:57:36Z

@elasticmachine merge upstream

…o move_remaining_ftrs

kibana-ci · 2024-02-13T07:22:31Z

💚 Build Succeeded

Buildkite Build
Commit: e0cc448

Metrics [docs]

✅ unchanged

History

💔 Build #193099 failed 92f73fe
💛 Build #192777 was flaky 7519725
💔 Build #192745 failed 2c39a19
💔 Build #192721 failed f1901c1
💔 Build #192718 failed bc5748c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @yctercero

…5837) **Resolves: elastic#151902 ## Summary After this PR, all D&R FTRs are moved to new folder where they can be run in ESS and serverless. Please see below table for a summary of what tests need revisiting by the teams. During the test migration there may have been some tests that failed on serverless, but not ESS. Some we were able to fix and get running on both, others are still marked as `brokenInServerless` and need triage.

yctercero added 2 commits January 29, 2024 11:13

moved over tests remaining in legacy ftr folder

b820406

Merge branch 'main' of github.com:elastic/kibana into move_remaining_…

be22bc0

…ftrs

yctercero requested review from a team as code owners January 29, 2024 19:41

yctercero requested a review from maximpn January 29, 2024 19:41

yctercero self-assigned this Jan 29, 2024

jbudz approved these changes Jan 29, 2024

View reviewed changes

yctercero added 2 commits January 29, 2024 17:17

fix lint errors

9b9ac18

Merge branch 'main' of github.com:elastic/kibana into move_remaining_…

5af01a4

…ftrs

yctercero requested a review from a team as a code owner January 30, 2024 01:17

cnasikas approved these changes Jan 30, 2024

View reviewed changes

banderror reviewed Jan 31, 2024

View reviewed changes

yctercero added 2 commits February 2, 2024 10:31

clean up some more areas referencing old folder

71c7e59

Merge branch 'main' of github.com:elastic/kibana into move_remaining_…

4e585f0

…ftrs

maximpn approved these changes Feb 5, 2024

View reviewed changes

x-pack/plugins/security_solution/common/experimental_features.ts Outdated Show resolved Hide resolved

yctercero and others added 3 commits February 5, 2024 09:10

add new ftr to tsconfig

be97404

removed local tsconfig in favor of upper level tsconfig

7fd0844

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

1d5cac9

…fix'

yctercero requested a review from a team as a code owner February 8, 2024 18:32

yctercero requested a review from rylnd February 8, 2024 18:32

yctercero added 3 commits February 8, 2024 16:19

trying to fix linting errors...again

ebaf35d

Merge branch 'main' of github.com:elastic/kibana into move_remaining_…

a53b381

…ftrs

cleanup after reviewing all files touched, trying to minimize unecess…

b352c65

…ary changes

kibanamachine and others added 9 commits February 8, 2024 20:21

Merge branch 'main' into move_remaining_ftrs

39b91c0

cleanup remaining lint errors

b44cfb9

Merge branch 'move_remaining_ftrs' of github.com:yctercero/kibana int…

b10b37b

…o move_remaining_ftrs

Merge branch 'main' of github.com:elastic/kibana into move_remaining_…

bc5748c

…ftrs

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

f1901c1

…-fix'

running linter locally didnt catch these errors, hoping this ones it

227339d

Merge branch 'main' of github.com:elastic/kibana into move_remaining_…

3e7ea2a

…ftrs

Merge branch 'move_remaining_ftrs' of github.com:yctercero/kibana int…

c2493e0

…o move_remaining_ftrs

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

2c39a19

…-fix'

Merge branch 'main' into move_remaining_ftrs

7519725

rylnd approved these changes Feb 12, 2024

View reviewed changes

fix merge conflict

92f73fe

kibanamachine and others added 3 commits February 13, 2024 00:57

Merge branch 'main' into move_remaining_ftrs

472fe53

updating references after merge upstream

087fcd9

Merge branch 'move_remaining_ftrs' of github.com:yctercero/kibana int…

e0cc448

…o move_remaining_ftrs

yctercero merged commit f4f71d1 into elastic:main Feb 13, 2024
34 checks passed

kibanamachine added the backport:skip This commit does not require backporting label Feb 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Detections Response] Finish moving remaining legacy FTRs #175837

[Detections Response] Finish moving remaining legacy FTRs #175837

yctercero commented Jan 29, 2024 •

edited by banderror

elasticmachine commented Jan 29, 2024

elasticmachine commented Jan 29, 2024

elasticmachine commented Jan 29, 2024

elasticmachine commented Jan 29, 2024

banderror commented Jan 31, 2024 •

edited

banderror left a comment

maximpn left a comment

yctercero commented Feb 9, 2024

yctercero commented Feb 12, 2024

rylnd left a comment

rylnd Feb 12, 2024

rylnd Feb 12, 2024

yctercero Feb 12, 2024

yctercero commented Feb 13, 2024

kibana-ci commented Feb 13, 2024

[Detections Response] Finish moving remaining legacy FTRs #175837

[Detections Response] Finish moving remaining legacy FTRs #175837

Conversation

yctercero commented Jan 29, 2024 • edited by banderror

Summary

Detection Engine FTRs

Rules Management FTRs

Shared D&R FTRs

elasticmachine commented Jan 29, 2024

elasticmachine commented Jan 29, 2024

elasticmachine commented Jan 29, 2024

elasticmachine commented Jan 29, 2024

banderror commented Jan 31, 2024 • edited

banderror left a comment

Choose a reason for hiding this comment

maximpn left a comment

Choose a reason for hiding this comment

yctercero commented Feb 9, 2024

yctercero commented Feb 12, 2024

rylnd left a comment

Choose a reason for hiding this comment

rylnd Feb 12, 2024

Choose a reason for hiding this comment

rylnd Feb 12, 2024

Choose a reason for hiding this comment

yctercero Feb 12, 2024

Choose a reason for hiding this comment

yctercero commented Feb 13, 2024

kibana-ci commented Feb 13, 2024

💚 Build Succeeded

Metrics [docs]

History

yctercero commented Jan 29, 2024 •

edited by banderror

banderror commented Jan 31, 2024 •

edited