[Fleet] Catch errors on package install if saved object reference not found #142112
Assignees
Labels
bug
Fixes for quality problems that affect the customer experience
Team:Fleet
Team label for Observability Data Collection Fleet team
Comments
juliaElastic
added
enhancement
1 task
|
The download source error is already reported here: #139513 |
juliaElastic
added a commit
that referenced
this issue
Nov 28, 2022
…ge assets (#146274) ## Summary Fixes #142112 Improvements coming out of SDHs, to handle Fleet errors more robustly. ### 1. Catching package assets not found error and logging a warning, to prevent it blocking setup/install. ### 2. In assets cleanup logic, doing a check that the assets are not referenced by packages before deleting them. Verifying 1. and 2. hard, because I don't have a way to reproduce the assets not found error. Tried to install/delete different versions of a package, but didn't run into the issues. ### 3. Force removing a package deletes package policies as well To verify: - add at least one integration (e.g. system package) - force delete integration with API ``` DELETE http://elastic:changeme@localhost:5601/julia/api/fleet/epm/packages/system/1.20.4 kbn-xsrf: kibana { "force": true } ``` - verify that package policies are deleted as well - there is an info log added: ``` [2022-11-24T14:10:29.138+01:00][INFO ][plugins.fleet] deleting package policies of system package because force flag was enabled: 84a4e5d4-e363-4045-9240-9ab151f2b376,6d34a544-7467-48fd-a779-c2d30d808aa6 ``` ### 4. Catching saved object not found error when tagging package assets To verify: - follow the steps above to force delete a package - add system integration again - verify that it succeeds, and there is a warning log ``` [2022-11-24T14:07:47.960+01:00][WARN ][plugins.fleet] Error: Saved object [dashboard/system-01c54730-fee6-11e9-8405-516218e3d268] not found ``` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Nov 28, 2022
…ge assets (elastic#146274) ## Summary Fixes elastic#142112 Improvements coming out of SDHs, to handle Fleet errors more robustly. ### 1. Catching package assets not found error and logging a warning, to prevent it blocking setup/install. ### 2. In assets cleanup logic, doing a check that the assets are not referenced by packages before deleting them. Verifying 1. and 2. hard, because I don't have a way to reproduce the assets not found error. Tried to install/delete different versions of a package, but didn't run into the issues. ### 3. Force removing a package deletes package policies as well To verify: - add at least one integration (e.g. system package) - force delete integration with API ``` DELETE http://elastic:changeme@localhost:5601/julia/api/fleet/epm/packages/system/1.20.4 kbn-xsrf: kibana { "force": true } ``` - verify that package policies are deleted as well - there is an info log added: ``` [2022-11-24T14:10:29.138+01:00][INFO ][plugins.fleet] deleting package policies of system package because force flag was enabled: 84a4e5d4-e363-4045-9240-9ab151f2b376,6d34a544-7467-48fd-a779-c2d30d808aa6 ``` ### 4. Catching saved object not found error when tagging package assets To verify: - follow the steps above to force delete a package - add system integration again - verify that it succeeds, and there is a warning log ``` [2022-11-24T14:07:47.960+01:00][WARN ][plugins.fleet] Error: Saved object [dashboard/system-01c54730-fee6-11e9-8405-516218e3d268] not found ``` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 7198faf)
kibanamachine
added a commit
that referenced
this issue
Nov 30, 2022
… package assets (#146274) (#146411) # Backport This will backport the following commits from `main` to `8.6`: - [Catching saved object not found error, do not delete referenced package assets (#146274)](#146274) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Julia Bardi","email":"90178898+juliaElastic@users.noreply.github.com"},"sourceCommit":{"committedDate":"2022-11-28T15:07:20Z","message":"Catching saved object not found error, do not delete referenced package assets (#146274)\n\n## Summary\r\n\r\nFixes #142112 coming out of SDHs, to handle Fleet errors more robustly.\r\n\r\n### 1. Catching package assets not found error and logging a warning, to\r\nprevent it blocking setup/install.\r\n### 2. In assets cleanup logic, doing a check that the assets are not\r\nreferenced by packages before deleting them.\r\n\r\nVerifying 1. and 2. hard, because I don't have a way to reproduce the\r\nassets not found error. Tried to install/delete different versions of a\r\npackage, but didn't run into the issues.\r\n\r\n### 3. Force removing a package deletes package policies as well\r\n\r\nTo verify:\r\n- add at least one integration (e.g. system package)\r\n- force delete integration with API\r\n```\r\nDELETE http://elastic:changeme@localhost:5601/julia/api/fleet/epm/packages/system/1.20.4\r\nkbn-xsrf: kibana\r\n\r\n{ \"force\": true }\r\n```\r\n- verify that package policies are deleted as well\r\n- there is an info log added:\r\n```\r\n[2022-11-24T14:10:29.138+01:00][INFO ][plugins.fleet] deleting package policies of system package because force flag was enabled: 84a4e5d4-e363-4045-9240-9ab151f2b376,6d34a544-7467-48fd-a779-c2d30d808aa6\r\n```\r\n\r\n### 4. Catching saved object not found error when tagging package assets\r\nTo verify:\r\n- follow the steps above to force delete a package\r\n- add system integration again \r\n- verify that it succeeds, and there is a warning log\r\n```\r\n[2022-11-24T14:07:47.960+01:00][WARN ][plugins.fleet] Error: Saved object [dashboard/system-01c54730-fee6-11e9-8405-516218e3d268] not found\r\n```\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>","sha":"7198faf5ee61faf26c41c4549b48d83a7eff0069","branchLabelMapping":{"^v8.7.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","v8.7.0","v8.6.1"],"number":146274,"url":"#146274 saved object not found error, do not delete referenced package assets (#146274)\n\n## Summary\r\n\r\nFixes #142112 coming out of SDHs, to handle Fleet errors more robustly.\r\n\r\n### 1. Catching package assets not found error and logging a warning, to\r\nprevent it blocking setup/install.\r\n### 2. In assets cleanup logic, doing a check that the assets are not\r\nreferenced by packages before deleting them.\r\n\r\nVerifying 1. and 2. hard, because I don't have a way to reproduce the\r\nassets not found error. Tried to install/delete different versions of a\r\npackage, but didn't run into the issues.\r\n\r\n### 3. Force removing a package deletes package policies as well\r\n\r\nTo verify:\r\n- add at least one integration (e.g. system package)\r\n- force delete integration with API\r\n```\r\nDELETE http://elastic:changeme@localhost:5601/julia/api/fleet/epm/packages/system/1.20.4\r\nkbn-xsrf: kibana\r\n\r\n{ \"force\": true }\r\n```\r\n- verify that package policies are deleted as well\r\n- there is an info log added:\r\n```\r\n[2022-11-24T14:10:29.138+01:00][INFO ][plugins.fleet] deleting package policies of system package because force flag was enabled: 84a4e5d4-e363-4045-9240-9ab151f2b376,6d34a544-7467-48fd-a779-c2d30d808aa6\r\n```\r\n\r\n### 4. Catching saved object not found error when tagging package assets\r\nTo verify:\r\n- follow the steps above to force delete a package\r\n- add system integration again \r\n- verify that it succeeds, and there is a warning log\r\n```\r\n[2022-11-24T14:07:47.960+01:00][WARN ][plugins.fleet] Error: Saved object [dashboard/system-01c54730-fee6-11e9-8405-516218e3d268] not found\r\n```\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>","sha":"7198faf5ee61faf26c41c4549b48d83a7eff0069"}},"sourceBranch":"main","suggestedTargetBranches":["8.6"],"targetPullRequestStates":[{"branch":"main","label":"v8.7.0","labelRegex":"^v8.7.0$","isSourceBranch":true,"state":"MERGED","url":"#146274 saved object not found error, do not delete referenced package assets (#146274)\n\n## Summary\r\n\r\nFixes #142112 coming out of SDHs, to handle Fleet errors more robustly.\r\n\r\n### 1. Catching package assets not found error and logging a warning, to\r\nprevent it blocking setup/install.\r\n### 2. In assets cleanup logic, doing a check that the assets are not\r\nreferenced by packages before deleting them.\r\n\r\nVerifying 1. and 2. hard, because I don't have a way to reproduce the\r\nassets not found error. Tried to install/delete different versions of a\r\npackage, but didn't run into the issues.\r\n\r\n### 3. Force removing a package deletes package policies as well\r\n\r\nTo verify:\r\n- add at least one integration (e.g. system package)\r\n- force delete integration with API\r\n```\r\nDELETE http://elastic:changeme@localhost:5601/julia/api/fleet/epm/packages/system/1.20.4\r\nkbn-xsrf: kibana\r\n\r\n{ \"force\": true }\r\n```\r\n- verify that package policies are deleted as well\r\n- there is an info log added:\r\n```\r\n[2022-11-24T14:10:29.138+01:00][INFO ][plugins.fleet] deleting package policies of system package because force flag was enabled: 84a4e5d4-e363-4045-9240-9ab151f2b376,6d34a544-7467-48fd-a779-c2d30d808aa6\r\n```\r\n\r\n### 4. Catching saved object not found error when tagging package assets\r\nTo verify:\r\n- follow the steps above to force delete a package\r\n- add system integration again \r\n- verify that it succeeds, and there is a warning log\r\n```\r\n[2022-11-24T14:07:47.960+01:00][WARN ][plugins.fleet] Error: Saved object [dashboard/system-01c54730-fee6-11e9-8405-516218e3d268] not found\r\n```\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>","sha":"7198faf5ee61faf26c41c4549b48d83a7eff0069"}},{"branch":"8.6","label":"v8.6.1","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In a recent sdh, Fleet setup triggered a package reinstall, which contained a reference to a saved object that did not exist anymore.
This caused an error from which Fleet could not recover.
The code should catch errors when looking up saved objects, logging out errors as a warning, and should not throw an error.
See usages of
savedObjectsClient.gethere: https://github.com/elastic/kibana/blob/main/x-pack/plugins/fleet/server/services/epm/archive/storage.tsStack trace:
Further improvements from the same SDH:
epm-packagessaved object as well, not onlyepm-packages-assets. It looks like we leave orphaned references inepm-packagesobject now.Unable to initialize Fleet Default download source host is not setup.Check the migration logic to 8.4 to make sure the created download source is set as default."unable to remove package with existing package policy(s) in use by agent(s)"error.Error: No package for package policy undefined at /usr/share/kibana/x-pack/plugins/fleet/server/services/agent_policies/package_policies_to_agent_permissions.js:37:13The text was updated successfully, but these errors were encountered: