[ML] Fix not blocking forced downgrades/installation if cannot delete indices #159814

qn895 · 2023-06-15T14:53:32Z

Summary

Currently, doing a forced downgrade of a package requires deletion of the transform's destination index. However, it's possible that the kibana_system user does not have sufficient permission to do the deletion of the index for certain indices.

This PR uses the new parameter to also enable deletion . With this change, the destination index is deleted together with the transform regardless of whether the destination index was created by the transform or by the user.

Before:

Example 1 (Dest index is a system index. Does not fail for reinstallation)

Screen.Recording.2023-06-20.at.13.15.48.mov

Example 2 (Dest index is a not system index. Fail for reinstallation)

Screen.Recording.2023-06-20.at.13.20.02.mov

After:

Example 1 (Dest index is a system index. Does not fail for reinstallation)

Screen.Recording.2023-06-20.at.13.23.47.mov

Example 2 (Dest index is a not system index. Does not fail for reinstallation)

Screen.Recording.2023-06-20.at.14.56.17.mov

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

This was checked for breaking API changes and was labeled appropriately

…ocking-downgrades

elasticmachine · 2023-06-20T16:52:51Z

Pinging @elastic/ml-ui (:ml)

elasticmachine · 2023-06-20T16:52:53Z

Pinging @elastic/fleet (Team:Fleet)

nchaulet · 2023-06-20T19:00:05Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/transform/remove.ts

-        { transform_id: transformId },
-        { ignore: [404] }
-      );
+      await esClient.transform.getTransform({ transform_id: transformId }, { ignore: [404] });


What the point of that call now? as we ignore 404 it seems to check the transform exists

Great catch! Removed here 656dd9d (#159814)

nchaulet

One small questions but otherwise LGTM 🚀

…g-downgrades

kibana-ci · 2023-06-20T21:42:06Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 431f516

Failed CI Steps

FTR Configs #55

Test Failures

[job] [logs] FTR Configs #55 / cloud_security_posture GET internal/cloud_security_posture/rules/_find Should return 200 status code and filter rules by benchmarkId

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`enterpriseSearch`	13	15	+2
`securitySolution`	411	415	+4
total			+6

Total ESLint disabled count

id	before	after	diff
`enterpriseSearch`	14	16	+2
`securitySolution`	494	498	+4
total			+6

History

💔 Build #136716 failed 656dd9d
💛 Build #136612 was flaky fcfa3a9
💔 Build #135672 failed 806ecb4

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @qn895

alvarezmelissa87

LGTM ⚡

… indices (elastic#159814) (cherry picked from commit 7686163)

kibanamachine · 2023-06-21T00:22:08Z

💚 All backports created successfully

Status	Branch	Result
✅	8.8

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…delete indices (#159814) (#160089) Co-authored-by: Quynh Nguyen (Quinn) <43350163+qn895@users.noreply.github.com> fix",":ml","Team:Fleet","v8.9.0","v8.8.2"],"number":159814,"url":"#159814

qn895 added bug Fixes for quality problems that affect the customer experience :ml Team:Fleet Team label for Observability Data Collection Fleet team v8.9.0 labels Jun 15, 2023

qn895 self-assigned this Jun 15, 2023

Fix not blocking downgrades/installation if cannot delete indicesyarn s

806ecb4

qn895 force-pushed the ml-fleet-fix-not-blocking-downgrades branch from b0f209a to 806ecb4 Compare June 15, 2023 15:28

qn895 added 2 commits June 20, 2023 09:49

Merge remote-tracking branch 'upstream/main' into ml-fleet-fix-not-bl…

aed161b

…ocking-downgrades

r

fcfa3a9

qn895 added the release_note:fix label Jun 20, 2023

qn895 marked this pull request as ready for review June 20, 2023 16:52

qn895 requested a review from a team as a code owner June 20, 2023 16:52

qn895 added the v8.8.2 label Jun 20, 2023

nchaulet reviewed Jun 20, 2023

View reviewed changes

nchaulet approved these changes Jun 20, 2023

View reviewed changes

qn895 added 2 commits June 20, 2023 14:59

Merge remote-tracking branch 'upstream' into ml-fleet-fix-not-blockin…

3805a5e

…g-downgrades

Remove unnecessary calls

431f516

qn895 force-pushed the ml-fleet-fix-not-blocking-downgrades branch from 656dd9d to 431f516 Compare June 20, 2023 20:41

alvarezmelissa87 approved these changes Jun 20, 2023

View reviewed changes

qn895 merged commit 7686163 into elastic:main Jun 21, 2023
24 checks passed

kibanamachine mentioned this pull request Jun 21, 2023

[8.8] [ML] Fix not blocking forced downgrades/installation if cannot delete indices (#159814) #160089

Merged

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jun 21, 2023

[ML] Fix not blocking forced downgrades/installation if cannot delete…

8b2eaa9

… indices (elastic#159814) (cherry picked from commit 7686163)

walterra mentioned this pull request Sep 20, 2023

[ML] Transform: Use ES delete_dest_index instead of delete via Kibana #166796

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Fix not blocking forced downgrades/installation if cannot delete indices #159814

[ML] Fix not blocking forced downgrades/installation if cannot delete indices #159814

qn895 commented Jun 15, 2023 •

edited by kibanamachine

elasticmachine commented Jun 20, 2023

elasticmachine commented Jun 20, 2023

nchaulet Jun 20, 2023

qn895 Jun 20, 2023

nchaulet left a comment

kibana-ci commented Jun 20, 2023

ESLint disabled line counts

Total ESLint disabled count

alvarezmelissa87 left a comment

kibanamachine commented Jun 21, 2023

[ML] Fix not blocking forced downgrades/installation if cannot delete indices #159814

[ML] Fix not blocking forced downgrades/installation if cannot delete indices #159814

Conversation

qn895 commented Jun 15, 2023 • edited by kibanamachine

Summary

Checklist

Risk Matrix

For maintainers

elasticmachine commented Jun 20, 2023

elasticmachine commented Jun 20, 2023

nchaulet Jun 20, 2023

Choose a reason for hiding this comment

qn895 Jun 20, 2023

Choose a reason for hiding this comment

nchaulet left a comment

Choose a reason for hiding this comment

kibana-ci commented Jun 20, 2023

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

ESLint disabled line counts

Total ESLint disabled count

History

alvarezmelissa87 left a comment

Choose a reason for hiding this comment

kibanamachine commented Jun 21, 2023

💚 All backports created successfully

Questions ?

qn895 commented Jun 15, 2023 •

edited by kibanamachine