[SecuritySolution] Failed to upgrade host risk score #144916
Assignees
Labels
bug
Fixes for quality problems that affect the customer experience
fixed
QA:Validated
Issue has been validated by QA
Team: SecuritySolution
Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Team:Threat Hunting:Explore
Team:Threat Hunting
Security Solution Threat Hunting Team
Comments
angorayc
added
Team:Threat Hunting
|
Pinging @elastic/security-threat-hunting (Team:Threat Hunting) |
angorayc
added
bug
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
2 tasks
angorayc
added a commit
that referenced
this issue
Nov 16, 2022
… scores (#145232) ## Summary Original issue: #144916 Users installed via https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md and https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md couldn't upgrade successfully. **Fixes**: 1. Remove all the legacy scripts and ingest pipelines with or without space name 2. Add version history to x-pack/plugins/security_solution/server/lib/risk_score/readme.md <img width="1459" alt="Screenshot 2022-11-15 at 13 49 43" src="https://user-images.githubusercontent.com/6295984/201936206-e73ab61c-9a0f-4cfe-8a01-9666217bb863.png"> <img width="1429" alt="Screenshot 2022-11-15 at 13 53 54" src="https://user-images.githubusercontent.com/6295984/201936751-c3a65f46-1f6e-4b2f-a04a-58f1f32a546f.png"> **Steps to reproduce**: Option 1: **Cypress**: Run `upgrade_risk_score.cy.ts` Option 2: **Manually**: 1. Follow the steps of https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md and https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md to install the module. 4. Back to `/app/security/entity_analytics` and click the upgrade buttons. 5. Observe if the installation success. ### Checklist Delete any items that are not applicable to this PR. - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>
angorayc
added a commit
to angorayc/kibana
that referenced
this issue
Nov 16, 2022
… scores (elastic#145232) ## Summary Original issue: elastic#144916 Users installed via https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md and https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md couldn't upgrade successfully. **Fixes**: 1. Remove all the legacy scripts and ingest pipelines with or without space name 2. Add version history to x-pack/plugins/security_solution/server/lib/risk_score/readme.md <img width="1459" alt="Screenshot 2022-11-15 at 13 49 43" src="https://user-images.githubusercontent.com/6295984/201936206-e73ab61c-9a0f-4cfe-8a01-9666217bb863.png"> <img width="1429" alt="Screenshot 2022-11-15 at 13 53 54" src="https://user-images.githubusercontent.com/6295984/201936751-c3a65f46-1f6e-4b2f-a04a-58f1f32a546f.png"> **Steps to reproduce**: Option 1: **Cypress**: Run `upgrade_risk_score.cy.ts` Option 2: **Manually**: 1. Follow the steps of https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md and https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md to install the module. 4. Back to `/app/security/entity_analytics` and click the upgrade buttons. 5. Observe if the installation success. ### Checklist Delete any items that are not applicable to this PR. - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co> (cherry picked from commit b6693bd) # Conflicts: # x-pack/plugins/security_solution/cypress/e2e/dashboards/enable_risk_score.cy.ts # x-pack/plugins/security_solution/cypress/e2e/dashboards/upgrade_risk_score.cy.ts # x-pack/plugins/security_solution/cypress/tasks/api_calls/risk_scores/index.ts # x-pack/plugins/security_solution/cypress/tasks/api_calls/risk_scores/saved_objects.ts # x-pack/plugins/security_solution/cypress/tasks/risk_scores/index.ts # x-pack/plugins/security_solution/cypress/tasks/risk_scores/ingest_pipelines.ts # x-pack/plugins/security_solution/cypress/tasks/risk_scores/stored_scripts.ts # x-pack/plugins/security_solution/cypress/tasks/risk_scores/transforms.ts # x-pack/plugins/security_solution/public/risk_score/components/risk_score_onboarding/utils.test.ts # x-pack/plugins/security_solution/public/risk_score/components/risk_score_onboarding/utils.ts
angorayc
added a commit
that referenced
this issue
Nov 17, 2022
…t risk scores (#145232) (#145352) # Backport This will backport the following commits from `main` to `8.5`: - [[SecuritySolution] Ingest pipelines conflict when upgrading host risk scores (#145232)](#145232) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Angela Chuang","email":"6295984+angorayc@users.noreply.github.com"},"sourceCommit":{"committedDate":"2022-11-16T12:26:56Z","message":"[SecuritySolution] Ingest pipelines conflict when upgrading host risk scores (#145232)\n\n## Summary\r\n\r\nOriginal issue: #144916 installed via\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md\r\nand\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md\r\ncouldn't upgrade successfully.\r\n\r\n**Fixes**:\r\n1. Remove all the legacy scripts and ingest pipelines with or without\r\nspace name\r\n2. Add version history to\r\nx-pack/plugins/security_solution/server/lib/risk_score/readme.md\r\n<img width=\"1459\" alt=\"Screenshot 2022-11-15 at 13 49 43\"\r\nsrc=\"https://user-images.githubusercontent.com/6295984/201936206-e73ab61c-9a0f-4cfe-8a01-9666217bb863.png\">\r\n\r\n<img width=\"1429\" alt=\"Screenshot 2022-11-15 at 13 53 54\"\r\nsrc=\"https://user-images.githubusercontent.com/6295984/201936751-c3a65f46-1f6e-4b2f-a04a-58f1f32a546f.png\">\r\n\r\n\r\n**Steps to reproduce**:\r\n\r\nOption 1: **Cypress**: Run `upgrade_risk_score.cy.ts`\r\n\r\nOption 2: **Manually**: \r\n1. Follow the steps of\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md\r\nand\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md\r\nto install the module.\r\n4. Back to `/app/security/entity_analytics` and click the upgrade\r\nbuttons.\r\n5. Observe if the installation success.\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Steph Milovic <stephanie.milovic@elastic.co>","sha":"b6693bd9260c1620ec5ad8f09141b534c3b02e81","branchLabelMapping":{"^v8.6.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","Team:Threat Hunting","Team: SecuritySolution","Team:Threat Hunting:Explore","v8.6.0","v8.5.2"],"number":145232,"url":"#145232 Ingest pipelines conflict when upgrading host risk scores (#145232)\n\n## Summary\r\n\r\nOriginal issue: #144916 installed via\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md\r\nand\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md\r\ncouldn't upgrade successfully.\r\n\r\n**Fixes**:\r\n1. Remove all the legacy scripts and ingest pipelines with or without\r\nspace name\r\n2. Add version history to\r\nx-pack/plugins/security_solution/server/lib/risk_score/readme.md\r\n<img width=\"1459\" alt=\"Screenshot 2022-11-15 at 13 49 43\"\r\nsrc=\"https://user-images.githubusercontent.com/6295984/201936206-e73ab61c-9a0f-4cfe-8a01-9666217bb863.png\">\r\n\r\n<img width=\"1429\" alt=\"Screenshot 2022-11-15 at 13 53 54\"\r\nsrc=\"https://user-images.githubusercontent.com/6295984/201936751-c3a65f46-1f6e-4b2f-a04a-58f1f32a546f.png\">\r\n\r\n\r\n**Steps to reproduce**:\r\n\r\nOption 1: **Cypress**: Run `upgrade_risk_score.cy.ts`\r\n\r\nOption 2: **Manually**: \r\n1. Follow the steps of\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md\r\nand\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md\r\nto install the module.\r\n4. Back to `/app/security/entity_analytics` and click the upgrade\r\nbuttons.\r\n5. Observe if the installation success.\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Steph Milovic <stephanie.milovic@elastic.co>","sha":"b6693bd9260c1620ec5ad8f09141b534c3b02e81"}},"sourceBranch":"main","suggestedTargetBranches":["8.5"],"targetPullRequestStates":[{"branch":"main","label":"v8.6.0","labelRegex":"^v8.6.0$","isSourceBranch":true,"state":"MERGED","url":"#145232 Ingest pipelines conflict when upgrading host risk scores (#145232)\n\n## Summary\r\n\r\nOriginal issue: #144916 installed via\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md\r\nand\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md\r\ncouldn't upgrade successfully.\r\n\r\n**Fixes**:\r\n1. Remove all the legacy scripts and ingest pipelines with or without\r\nspace name\r\n2. Add version history to\r\nx-pack/plugins/security_solution/server/lib/risk_score/readme.md\r\n<img width=\"1459\" alt=\"Screenshot 2022-11-15 at 13 49 43\"\r\nsrc=\"https://user-images.githubusercontent.com/6295984/201936206-e73ab61c-9a0f-4cfe-8a01-9666217bb863.png\">\r\n\r\n<img width=\"1429\" alt=\"Screenshot 2022-11-15 at 13 53 54\"\r\nsrc=\"https://user-images.githubusercontent.com/6295984/201936751-c3a65f46-1f6e-4b2f-a04a-58f1f32a546f.png\">\r\n\r\n\r\n**Steps to reproduce**:\r\n\r\nOption 1: **Cypress**: Run `upgrade_risk_score.cy.ts`\r\n\r\nOption 2: **Manually**: \r\n1. Follow the steps of\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md\r\nand\r\nhttps://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md\r\nto install the module.\r\n4. Back to `/app/security/entity_analytics` and click the upgrade\r\nbuttons.\r\n5. Observe if the installation success.\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Steph Milovic <stephanie.milovic@elastic.co>","sha":"b6693bd9260c1620ec5ad8f09141b534c3b02e81"}},{"branch":"8.5","label":"v8.5.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
benakansara
pushed a commit
to benakansara/kibana
that referenced
this issue
Nov 17, 2022
… scores (elastic#145232) ## Summary Original issue: elastic#144916 Users installed via https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md and https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md couldn't upgrade successfully. **Fixes**: 1. Remove all the legacy scripts and ingest pipelines with or without space name 2. Add version history to x-pack/plugins/security_solution/server/lib/risk_score/readme.md <img width="1459" alt="Screenshot 2022-11-15 at 13 49 43" src="https://user-images.githubusercontent.com/6295984/201936206-e73ab61c-9a0f-4cfe-8a01-9666217bb863.png"> <img width="1429" alt="Screenshot 2022-11-15 at 13 53 54" src="https://user-images.githubusercontent.com/6295984/201936751-c3a65f46-1f6e-4b2f-a04a-58f1f32a546f.png"> **Steps to reproduce**: Option 1: **Cypress**: Run `upgrade_risk_score.cy.ts` Option 2: **Manually**: 1. Follow the steps of https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md and https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/user-risk-score.md to install the module. 4. Back to `/app/security/entity_analytics` and click the upgrade buttons. 5. Observe if the installation success. ### Checklist Delete any items that are not applicable to this PR. - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>
|
we have validated this issue on 8.6.0 BC2 and found this issue to be fixed now ✔️ . Build Details: Screen-Cast
Transforms.-.Elastic.Mozilla.Firefox.2022-11-24.06-52-58.mp4
Transforms.-.Elastic.Mozilla.Firefox.2022-11-24.06-57-45.mp4Hence we are closing this issue and adding "QA:Validated" tag to it. c.c @angorayc |
![https://user-images.githubusercontent.com/6295984/201936206-e73ab61c-9a0f-4cfe-8a01-9666217bb863.png\">\r\n\r\n<img](https://user-images.githubusercontent.com/6295984/201936206-e73ab61c-9a0f-4cfe-8a01-9666217bb863.png\">\r\n\r\n<img){kind=link}
![https://user-images.githubusercontent.com/6295984/201936751-c3a65f46-1f6e-4b2f-a04a-58f1f32a546f.png\">\r\n\r\n\r\n**Steps](https://user-images.githubusercontent.com/6295984/201936751-c3a65f46-1f6e-4b2f-a04a-58f1f32a546f.png\">\r\n\r\n\r\n**Steps){kind=link}
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue
Some users are having issue with upgrade host risk score if they followed https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md
Work around
Delete the module manually:
elastic/security-docs#2477 (comment)
Root cause
At the very beginning (before 8.4) the ingest pipeline id for host risk score was
ml_hostriskscore_ingest_pipeline_<space-name>https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/host-risk-score.md#3-upload-ingest-pipeline
We found that the space name of ingest pipeline for host risk score was removed in 8.4, which became
ml_hostriskscore_ingest_pipeline, so we remove the ingest pipeline id without space name (ml_hostriskscore_ingest_pipeline) during upgrade, and trying to create a new one with space name appended (ml_hostriskscore_ingest_pipeline_<space-name>). Therefore the conflict happens if people hadml_hostriskscore_ingest_pipeline_<space-name>already.We are going to delete both
ml_hostriskscore_ingest_pipeline_<space-name>andml_hostriskscore_ingest_pipelineduring upgrade.The text was updated successfully, but these errors were encountered: