File delivery

[pzl]

What is the problem this PR solves?

Delivering user-uploaded files (via kibana) to integrations (specifically, endpoint use-case)

How does this PR solve the problem?

an API route was added, /api/fleet/file/[ID]. Authenticated calls to this will locate file metadata documents in elasticsearch for the presence of a file, and that the authenticated agent is present in the file's file.Meta.target_agents field (i.e. only allowed client IDs can download). Fleet server iterates through the file contents to send on the response pipe

How to test this PR locally

• Create destination indices in Elasticsearch matching these templates (until the PR is merged): [Fleet] Add File Delivery index templates elasticsearch#96504
• Use a recent Dev Endpoint that includes file-requesting feature
• In kibana, create a send-file action to send a file to an Endpoint

Alternatively, a simpler setup would be to check out this PR, and modify the authentication section in internal/pkg/api/handleFileDelivery.go line 48. Comment out the authAgent call (or otherwise mock it), and put a different, perhaps static ID as the last parameter to the FindFileForAgent call a few lines down (e.g. ... fileID, "mockID")). As long as a file is uploaded to the right indices (.fleet-filedelivery-meta-endpoint and .fleet-filedelivery-data-endpoint) with that matching agent_id in the target_agents field, then a command-line curl, wget or similar to https://localhost:8221/api/fleet/file/[ID] should retrieve the file like any other file-based HTTP url.

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-06-09T19:23:13.075+0000

• Duration: 42 min 53 sec

Test stats 🧪

Test	Results
Failed	0
Passed	706
Skipped	1
Total	707

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[mergify]

This pull request is now in conflicts. Could you fix it @pzl? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b file-delivery upstream/file-delivery
git merge upstream/main
git push upstream file-delivery

[michel-laterman]

looks good so far.

Can you also add a test case for the e2e tests? (or a follow up item to do so)

[kevinlog]

I built this and ran Fleet Server locally with a Kibana stack to test everything e2e and it worked!

I can upload multi-chunk files and Endpoint will be able to download them from ES through this API.

In use in Response Console:

The file ends up on the host in the expected folder from Endpoint and is executable by root:

from a functionality POV, LGTM

[michel-laterman]

LGTM.

Please follow it up with e2e tests. you can ping me if you need help writing them

[pzl]

Thanks, adding plenty of tests in a followup. Going to merge this to get builds going over the weekend, for other teams to start integrating