Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fleet] Add File Delivery index templates #96504

Merged
merged 6 commits into from
Jun 8, 2023
Merged

Conversation

pzl
Copy link
Member

@pzl pzl commented Jun 1, 2023

Adds two new index templates (.fleet-filedevliery-meta-*-* and .fleet-filedelivery-data-*-*) for fleet usage, supporting a limited file-delivery feature through fleet.

This follows a similar path as #91413 with somewhat different usage patterns.

@pzl pzl added >feature :Core/Infra/Plugins Plugin API and infrastructure Team:Core/Infra Meta label for core/infra team Team:Fleet v8.9.0 labels Jun 1, 2023
@pzl pzl requested a review from hop-dev June 1, 2023 17:09
@elasticsearchmachine elasticsearchmachine added the external-contributor Pull request authored by a developer outside the Elasticsearch team label Jun 1, 2023
@pzl pzl marked this pull request as ready for review June 1, 2023 21:27
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (Team:Core/Infra)

@pzl pzl mentioned this pull request Jun 2, 2023
8 tasks
@kevinlog
Copy link
Contributor

kevinlog commented Jun 7, 2023

I checked this out and tested it in combination with the following PRs to ensure that files uploaded from Kibana to ES and are downloaded by the Endpoint.

PRs:
elastic/fleet-server#2666
elastic/kibana#158773

Templates get installed by Fleet:
image

When files get uploaded to Kibana, relevant docs are streamed in
image

@@ -0,0 +1,25 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can probably delete these ilm policies since we know we need to use another method for this in serverless

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the latest recommendation is to keep using ILM for stateful: https://github.com/elastic/ingest-dev/issues/1604#issuecomment-1536053175

@@ -0,0 +1,42 @@
{
"index_patterns": [
".fleet-filedelivery-data-*-*"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we decide to move to data streams, these indices should be converted as well.

Copy link
Contributor

@juliaElastic juliaElastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, see comments.

@pzl pzl merged commit bcd25c8 into elastic:main Jun 8, 2023
11 checks passed
@pzl pzl deleted the fleet-file-delivery branch June 8, 2023 14:15
paul-tavares added a commit to elastic/kibana that referenced this pull request Jun 9, 2023
…ents (#158773)

## Summary

- Adds `const`'s for index patterns for files that are delivered to
hosts/agents
- modifies `createFilesClient.toHost()` (exposed in `Plugin.start()`) to
use these new index names
- Adds code to the package install flow to ensure new file delivery
indexes are created
- The files client factory now validates if the integration name is
allowed to use fleet files functionality.


FYI: PR that will add index mappings to ES:
elastic/elasticsearch#96504
HiDAl pushed a commit to HiDAl/elasticsearch that referenced this pull request Jun 14, 2023
saarikabhasi pushed a commit to saarikabhasi/kibana that referenced this pull request Jun 14, 2023
…ents (elastic#158773)

## Summary

- Adds `const`'s for index patterns for files that are delivered to
hosts/agents
- modifies `createFilesClient.toHost()` (exposed in `Plugin.start()`) to
use these new index names
- Adds code to the package install flow to ensure new file delivery
indexes are created
- The files client factory now validates if the integration name is
allowed to use fleet files functionality.


FYI: PR that will add index mappings to ES:
elastic/elasticsearch#96504
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Core/Infra/Plugins Plugin API and infrastructure external-contributor Pull request authored by a developer outside the Elasticsearch team >feature Team:Core/Infra Meta label for core/infra team Team:Fleet v8.9.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants