[Fleet] Allow uploading zip packages in integrations UI #148599
Comments
jen-huang
added
enhancement
|
Pinging @elastic/fleet (Team:Fleet) |
mukeshelastic
changed the title
|
After reviewing the issue, the linked improvements in #70582 are still needed. Started working on them.
Can't reproduce this issue:
This one I'll hold off on, as it is a change in package-spec repo.
|
|
@jen-huang @kpollich @jsoriano Do we have an example package that is not yet in the registry? The |
|
@juliaElastic you could use |
|
@hop-dev it worked, thanks. |
|
@jen-huang @kpollich I am wondering if this issue is still relevant: #82010 Not sure how to reproduce a cache miss. Do any of you know? I'm looking at the uploaded packages after kibana restart in local, and don't see any issues.
|
|
I think the above is likely no longer relevant. From what I remember, all installation paths add packages to the in-memory cache as expected. I'm going to close that issue as I don't think it's relevant to the current state of the EPM codebase. |
## Summary Related to #148599 Fixes #81995 Changed `GET /epm/packages` API to list uploaded packages as well that are not in registry. Testing steps: - Built a test package that is not in registry from kibana repo: ``` x-pack/test/fleet_api_integration/apis/fixtures/package_verification/packages/src/verified-1.0.0 elastic-package build --zip -v ``` - Uploaded the resulting zip [verified-1.0.0.zip](https://github.com/elastic/kibana/files/10664094/verified-1.0.0.zip) ``` curl -XPOST -H 'content-type: application/zip' -H 'kbn-xsrf: true' http://localhost:5601/julia/api/fleet/epm/packages -u elastic:changeme --data-binary @verified-1.0.0.zip ``` - Navigate to `Integrations` page, verify that the `Verified` package is displayed under `Browse` / `Installed` integrations, and the details are visible when clicking on the package. <img width="639" alt="image" src="https://user-images.githubusercontent.com/90178898/217000305-90c28293-bdfa-42d5-b30b-50f7a63aad11.png"> <img width="757" alt="image" src="https://user-images.githubusercontent.com/90178898/217000420-208d3b90-5341-4f1f-8ab8-554f78821bbf.png"> <img width="1147" alt="image" src="https://user-images.githubusercontent.com/90178898/217000541-06324345-4204-43cc-8411-6c961ec37d17.png"> ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
…0332) ## Summary Related to elastic#148599 Fixes elastic#81995 Changed `GET /epm/packages` API to list uploaded packages as well that are not in registry. Testing steps: - Built a test package that is not in registry from kibana repo: ``` x-pack/test/fleet_api_integration/apis/fixtures/package_verification/packages/src/verified-1.0.0 elastic-package build --zip -v ``` - Uploaded the resulting zip [verified-1.0.0.zip](https://github.com/elastic/kibana/files/10664094/verified-1.0.0.zip) ``` curl -XPOST -H 'content-type: application/zip' -H 'kbn-xsrf: true' http://localhost:5601/julia/api/fleet/epm/packages -u elastic:changeme --data-binary @verified-1.0.0.zip ``` - Navigate to `Integrations` page, verify that the `Verified` package is displayed under `Browse` / `Installed` integrations, and the details are visible when clicking on the package. <img width="639" alt="image" src="https://user-images.githubusercontent.com/90178898/217000305-90c28293-bdfa-42d5-b30b-50f7a63aad11.png"> <img width="757" alt="image" src="https://user-images.githubusercontent.com/90178898/217000420-208d3b90-5341-4f1f-8ab8-554f78821bbf.png"> <img width="1147" alt="image" src="https://user-images.githubusercontent.com/90178898/217000541-06324345-4204-43cc-8411-6c961ec37d17.png"> ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
|
Tested the @kpollich What is the expected behavior of the We could potentially add a drag and drop feature to upload the zip file again on the UI, though this is out of scope for this issue I think. Should we make the button disabled with a tooltip for uploaded packages?
|
Agreed on a both. A dropzone implementation could be nice, but seems needless for the corner case of reinstalling an uploaded package. Let's go forward with the disabled button + tooltip stating
|
…0332) ## Summary Related to elastic#148599 Fixes elastic#81995 Changed `GET /epm/packages` API to list uploaded packages as well that are not in registry. Testing steps: - Built a test package that is not in registry from kibana repo: ``` x-pack/test/fleet_api_integration/apis/fixtures/package_verification/packages/src/verified-1.0.0 elastic-package build --zip -v ``` - Uploaded the resulting zip [verified-1.0.0.zip](https://github.com/elastic/kibana/files/10664094/verified-1.0.0.zip) ``` curl -XPOST -H 'content-type: application/zip' -H 'kbn-xsrf: true' http://localhost:5601/julia/api/fleet/epm/packages -u elastic:changeme --data-binary @verified-1.0.0.zip ``` - Navigate to `Integrations` page, verify that the `Verified` package is displayed under `Browse` / `Installed` integrations, and the details are visible when clicking on the package. <img width="639" alt="image" src="https://user-images.githubusercontent.com/90178898/217000305-90c28293-bdfa-42d5-b30b-50f7a63aad11.png"> <img width="757" alt="image" src="https://user-images.githubusercontent.com/90178898/217000420-208d3b90-5341-4f1f-8ab8-554f78821bbf.png"> <img width="1147" alt="image" src="https://user-images.githubusercontent.com/90178898/217000541-06324345-4204-43cc-8411-6c961ec37d17.png"> ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
…150444) ## Summary Related to #148599 Closes #82007 Factored out common logic in `installPackageFromRegistry` and `installPackageByUpload` to a new function. This improves the upload flow as well to handle install failure. We might want to introduce a `force` flag for the upload API to reinstall a package, now the logic does not do anything if the installed version package is uploaded again. EDIT: found [here](#82007) that upload should work with an implicit `force` flag, so will update that. More manual and automated tests needed. ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
|
@joshdover I tried to reproduce the issue reported here on 8.7 but couldn't:
I uploaded the
|
|
@juliaElastic it must be fixed, I was testing this against 8.5 at the time which I think was before some of the changes we did for input packages were in place that may have fixed this. |
## Summary Disable Reinstall button and added tooltip to prevent clicking Reinstall for uploaded packages. Related to #148599 (comment) <img width="731" alt="image" src="https://user-images.githubusercontent.com/90178898/217497440-e36bad4d-d4dc-4d1c-b434-b17f0a4ead0a.png"> ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
## Summary Related to #148599 Added more integration tests. Added openapi spec. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
## Summary Related to elastic#148599 Added more integration tests. Added openapi spec. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit 486d56b)
# Backport This will backport the following commits from `main` to `8.7`: - [[Fleet] added more upload tests and openapi (#150743)](#150743) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Julia Bardi","email":"90178898+juliaElastic@users.noreply.github.com"},"sourceCommit":{"committedDate":"2023-02-10T12:07:23Z","message":"[Fleet] added more upload tests and openapi (#150743)\n\n## Summary\r\n\r\nRelated to #148599 more integration tests.\r\n\r\nAdded openapi spec.\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"486d56b33fcb137f6f2dc62a13acd58ea19a1e00","branchLabelMapping":{"^v8.8.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","Team:Fleet","v8.7.0","v8.8.0"],"number":150743,"url":"#150743 added more upload tests and openapi (#150743)\n\n## Summary\r\n\r\nRelated to #148599 more integration tests.\r\n\r\nAdded openapi spec.\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"486d56b33fcb137f6f2dc62a13acd58ea19a1e00"}},"sourceBranch":"main","suggestedTargetBranches":["8.7"],"targetPullRequestStates":[{"branch":"8.7","label":"v8.7.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.8.0","labelRegex":"^v8.8.0$","isSourceBranch":true,"state":"MERGED","url":"#150743 added more upload tests and openapi (#150743)\n\n## Summary\r\n\r\nRelated to #148599 more integration tests.\r\n\r\nAdded openapi spec.\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"486d56b33fcb137f6f2dc62a13acd58ea19a1e00"}}]}] BACKPORT--> Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>
## Summary Related to #148599 Added unit tests on parse package archive logic ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
## Summary Related to elastic#148599 Added unit tests on parse package archive logic ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit 3aca0c2)
… (#150908) # Backport This will backport the following commits from `main` to `8.7`: - [[Fleet] added unit tests on parse package archive logic (#150888)](#150888) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Julia Bardi","email":"90178898+juliaElastic@users.noreply.github.com"},"sourceCommit":{"committedDate":"2023-02-10T15:54:27Z","message":"[Fleet] added unit tests on parse package archive logic (#150888)\n\n## Summary\r\n\r\nRelated to #148599 unit tests on parse package archive logic\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"3aca0c2212d4caa16d8539e934210849d813aad1","branchLabelMapping":{"^v8.8.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","backport:prev-minor","v8.8.0"],"number":150888,"url":"#150888 added unit tests on parse package archive logic (#150888)\n\n## Summary\r\n\r\nRelated to #148599 unit tests on parse package archive logic\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"3aca0c2212d4caa16d8539e934210849d813aad1"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.8.0","labelRegex":"^v8.8.0$","isSourceBranch":true,"state":"MERGED","url":"#150888 added unit tests on parse package archive logic (#150888)\n\n## Summary\r\n\r\nRelated to #148599 unit tests on parse package archive logic\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"3aca0c2212d4caa16d8539e934210849d813aad1"}}]}] BACKPORT--> Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>
|
Tested uploads with
|
|
The API work is done, so closing the issue. |
|
I tried uploading a
@kpollich Could you let me know how to reproduce the issues mentioned in the description? Another thing, I noticed that the upload package logic skips verification. Is this something we want to support in the future? If so, might be worth creating a follow up issue. |
It's been a very long time, so I'm not sure. If gzip functionality is working in recent tests, then I'd expect work we've done to archive parsing logic in recent sprints/releases has resolved whatever issues we had initially.
Yes, it'd be great to support verification of uploaded packages if possible. I agree that a follow up issue would be appropriate here. I suppose we'll have to think about how to provide a signature file along with an uploaded package, right? |
Yes, we might have to change the API to support multiple files or create another API endpoint to verify an uploaded package. If we eventually add the |
|
I think having the upload API support multiple files would be ideal, but not sure on how much work that'd actually be. A separate API for uploading signatures would be a fallback. Another option could be to allow for a different archive directory structure that includes signature files at the root of the archive. We'd have to have some conditional logic that determines whether this is the case, though, because the current validation logic expects only a single top-level directory I believe. |
I don't think this would be feasible since the content of the signature itself depends on the content of the zip. +1 to supporting multiple files on the same API call. This is supported by multipart content-type I believe |
## Summary Related to #148599 Added missing `application/gzip` content type to openapi for packages upload API.
|
@kpollich & @juliaElastic is there a docs issue for this feature? |
|
@nimarezainia Not yet, perhaps we could add a section in the docs about how to upload packages around here: https://www.elastic.co/guide/en/fleet/master/air-gapped.html#air-gapped-diy-epr |
|
Thanks @juliaElastic. if there was a docs issue with enough details for the documentation to follow that would be great. Regarding the location where that content resides I would leave it to their expertise. (fyi @dedemorton & @karenzone ) |
|
@juliaElastic If this is an alternative to users hosting their own package registry, then, yes, it should be documented in the topic about air gapped environments. To get this in quickly, I would suggest opening a PR with the doc changes and tagging |
Supersedes #70582
Today we have the endpoint
POST /api/fleet/epm/packageswhich accepts a zip package archive and uploads it to Fleet in the same way that a regular package install would. This feature is important to reduce the total delivery time for package developers. While much of the functionality is complete through enabling bundled packages with Kibana, there are still some outstanding tasks before this API is ready for documentation and wider use:installPackageByUpload,parseAndVerifyArchive)Sample test instructions can be found at elastic/integrations#4597:
Related enhancements that can be done once the API is finished:
elastic-package: Add support to upload packages directly to Kibana/Fleet elastic-package#1084The text was updated successfully, but these errors were encountered: