Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Exceptions Cypress tests #81759

Merged
merged 68 commits into from
Nov 30, 2020

Conversation

MadameSheema
Copy link
Member

@MadameSheema MadameSheema commented Oct 27, 2020

Summary

In this PR we are adding some tests in order to check that exceptions feature is working fine.

These tests were complex to develop so I will appreciate a deep look to it, because I'm afraid I can be messing or not properly checking things.

For these tests we have created different archives:

  • rule_for_exceptions
  • rule_for_exceptions_from_alert
  • auditbeat_for_exceptions
  • auditbeat_for_exceptions2
  • auditbeat_for_exceptions3
  • auditbeat_for_exceptions_from_alert
  • auditbeat_for_exceptions_from_alert2
  • auditbeat_for_exceptions_from_alert3

The rule_for_exceptions archive contains a custom rule with query host.name:* and index pattern exceptions-*. As on CI all the Kibana tests are executed in parallel, this was the selected index pattern in order to avoid collisions with different tests from different teams using a common index pattern (i.e. auditbeat).

The rule_for_exceptions_from_alert archive contains a rule with archive contains a custom rule with query host.name:* and index pattern exceptionsalert-*. This index pattern was selected from the same reason as the previous one and also, to prevent collisions with the previous test.

auditbeat_for_exceptions, auditbeat_for_exceptions2 and auditbeat_for_exceptions3, each archive contains 2 different alerts that matches the exceptions-* pattern.

auditbeat_for_exceptions_from_alert, auditbeat_for_exceptions_from_alert2, auditbeat_for_exceptions_from_alert3, each archive contains 2 different alerts that thames the exceptionsalert-* pattern.

Creates an exception from rule details and deletes the exception implementation:

  • We load the rule
  • We activate it
  • We load the first set of data
  • We wait for the rule to be executed
  • We deactivate the rule
  • We check that the rule has populated the expected alerts
  • We add an exception from the rule details for all the host.names that matches with the alerts on the archive, checking the closing alerts option
  • We activate the rule again
  • We load a new set of data
  • We wait for the rule to be executed again
  • We navigate to closed alerts in order to check that the alerts have been closed
  • We navigate to opened alerts and we check that no new rules have been populated
  • We delete the exception
  • We load a new set of data
  • We wait for the rule to be executed
  • We check that the rule has populated the expected alerts

Creates an exception from an existing alert and deletes the exception implementation:

  • We load the rule
  • We activate it
  • We load the first set of data
  • We wait for the rule to be executed
  • We deactivate the rule
  • We check that the rule has populated the expected alerts
  • We add an exception from the first existing alert for all the host.names that matches with the alerts on the archive, checking the closing alerts option
  • We activate the rule again
  • We load a new set of data
  • We wait for the rule to be executed again
  • We navigate to closed alerts in order to check that the alerts have been closed
  • We navigate to opened alerts and we check that no new rules have been populated
  • We delete the exception
  • We load a new set of data
  • We wait for the rule to be executed
  • We check that the rule has populated the expected alerts

MadameSheema and others added 30 commits September 15, 2020 21:46
# Conflicts:
#	x-pack/plugins/security_solution/cypress/integration/alerts_detection_rules_custom.spec.ts
#	x-pack/plugins/security_solution/cypress/integration/alerts_detection_rules_ml.spec.ts
#	x-pack/plugins/security_solution/cypress/integration/alerts_detection_rules_override.spec.ts
#	x-pack/plugins/security_solution/cypress/integration/alerts_detection_rules_threshold.spec.ts
#	x-pack/plugins/security_solution/cypress/screens/rule_details.ts
#	x-pack/plugins/security_solution/cypress/tasks/create_new_rule.ts
# Conflicts:
#	x-pack/plugins/security_solution/cypress/integration/alerts_detection_rules_custom.spec.ts
#	x-pack/plugins/security_solution/cypress/integration/alerts_detection_rules_ml.spec.ts
#	x-pack/plugins/security_solution/cypress/integration/alerts_detection_rules_override.spec.ts
#	x-pack/plugins/security_solution/cypress/integration/alerts_detection_rules_threshold.spec.ts
#	x-pack/plugins/security_solution/cypress/screens/rule_details.ts
#	x-pack/plugins/security_solution/cypress/tasks/create_new_rule.ts
@MadameSheema
Copy link
Member Author

@elasticmachine merge upstream

Copy link
Contributor

@peluja1012 peluja1012 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for added coverage and for helping debug the "conflict" issue!

@patrykkopycinski
Copy link
Contributor

@elasticmachine merge upstream

@MadameSheema
Copy link
Member Author

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 8.0MB 8.0MB +159.0B

Distributable file count

id before after diff
default 43134 43139 +5

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@MadameSheema MadameSheema merged commit 4546352 into elastic:master Nov 30, 2020
@MadameSheema MadameSheema deleted the exceptions branch November 30, 2020 09:37
MadameSheema added a commit to MadameSheema/kibana that referenced this pull request Nov 30, 2020
* improves 'Creates and activates a new custom rule' test

* fixes constant problem

* improves 'Creates and activates a new custom rule with override option' test

* improves 'Creates and activates a new threshold rule' test

* refactor

* fixes type check issue

* improves assertions

* removes unused code

* changes variables for constants

* improves 'waitForTheRuleToBeExecuted' test

* improves readability

* fixes jenkins error

* refactor

* blah

* more things

* finishes 'Creates an exception from rule details and deletes the excpetion' implementation

* implements 'Creates an exception from an alert and deletes the exception'

* updates VALUES_INPUT locator

* updates archiver

* refactor

* improves the code

* fixes CI error

* renames exceptions archive

* refactor

* fixes merge issue

* fixes CI issue

* debug

* refactor

* improves test data

* removes signals index after the execution

* removes unused line

* removes unused variable

* refactors 'numberOfauditbeatExceptionsAlerts' constant to camel case

* simplifies the archive

* waits for the rule to be executed after navigating to opened alerts tab

* cleaning data

* fixes tests flakiness

* cleans test data

* refactors code

* removes unsused archives

* cleans data

* simplifies data

* fixes CI issue

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
gmmorris added a commit to gmmorris/kibana that referenced this pull request Nov 30, 2020
* master:
  [Security Solution] Exceptions Cypress tests (elastic#81759)
  [ML] Fix spaces job ID check (elastic#84404)
  [Security Solution][Detections] Handle dupes when processing threshold rules (elastic#83062)
  skip flaky suite (elastic#84440)
  skip flaky suite (elastic#84445)
  [APM] Fix missing `service.node.name` (elastic#84269)
  Upgrade fp-ts to 2.8.6 (elastic#83866)
  Added data streams privileges to better control delete actions in UI (elastic#83573)
  Improve short-url redirect validation (elastic#84366)
  TSVB offsets (elastic#83051)
  [Discover] Fix navigating back when changing index pattern (elastic#84061)
  [Logs UI] Polish the UI for the log entry examples in the anomaly table (elastic#82139)
  [Logs UI] Limit the height of the "view in context" container (elastic#83178)
  [Application Usage] Update `schema` with new `fleet` rename (elastic#84327)
  fix identation in list (elastic#84301)
MadameSheema added a commit that referenced this pull request Nov 30, 2020
* improves 'Creates and activates a new custom rule' test

* fixes constant problem

* improves 'Creates and activates a new custom rule with override option' test

* improves 'Creates and activates a new threshold rule' test

* refactor

* fixes type check issue

* improves assertions

* removes unused code

* changes variables for constants

* improves 'waitForTheRuleToBeExecuted' test

* improves readability

* fixes jenkins error

* refactor

* blah

* more things

* finishes 'Creates an exception from rule details and deletes the excpetion' implementation

* implements 'Creates an exception from an alert and deletes the exception'

* updates VALUES_INPUT locator

* updates archiver

* refactor

* improves the code

* fixes CI error

* renames exceptions archive

* refactor

* fixes merge issue

* fixes CI issue

* debug

* refactor

* improves test data

* removes signals index after the execution

* removes unused line

* removes unused variable

* refactors 'numberOfauditbeatExceptionsAlerts' constant to camel case

* simplifies the archive

* waits for the rule to be executed after navigating to opened alerts tab

* cleaning data

* fixes tests flakiness

* cleans test data

* refactors code

* removes unsused archives

* cleans data

* simplifies data

* fixes CI issue

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
phillipb added a commit to phillipb/kibana that referenced this pull request Nov 30, 2020
…bana into add-metadata-to-node-details

* 'add-metadata-to-node-details' of github.com:phillipb/kibana:
  [APM] ML anomaly detection integration: Displaying anomaly job results in the Transaction duration chart is not as intended  (elastic#84415)
  Support for painless language autocomplete within monaco (elastic#80577)
  [Lens] Time scale ui (elastic#83904)
  removing beta callouts (elastic#84510)
  [Lens] (Accessibility) add aria-label to chart type icon (elastic#84493)
  Trusted Apps signer API. (elastic#83661)
  increase stdout max listeners for legacy logging (elastic#84497)
  [APM] Service overview: Add throughput chart (elastic#84439)
  [Discover] Unskip main functional tests (elastic#84300)
  Uptime overview overhaul (elastic#83406)
  [APM] Adjust time formats based on the difference between start and end (elastic#84470)
  [ML] Renaming saved object repair to sync (elastic#84311)
  [UsageCollection] Remove `formatBulkUpload` and other unused APIs (elastic#84313)
  [Visualizations] Adds visConfig.title and uiState to build pipeline function (elastic#84456)
  [Elasticsearch Migration] Update docs re UsageCollection (elastic#84322)
  TSVB field list performance issue on using annotations (elastic#84407)
  [Security Solution] Exceptions Cypress tests (elastic#81759)
  [ML] Fix spaces job ID check (elastic#84404)
  [Security Solution][Detections] Handle dupes when processing threshold rules (elastic#83062)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:skip Skip the PR/issue when compiling release notes Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v7.11.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants