Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Elastic Agent does not pass down input id to filestream #30300

Closed
jlind23 opened this issue Feb 9, 2022 · 9 comments · Fixed by #30386 or elastic/elastic-agent#196
Closed

Elastic Agent does not pass down input id to filestream #30300

jlind23 opened this issue Feb 9, 2022 · 9 comments · Fixed by #30386 or elastic/elastic-agent#196
Assignees
@ph
Labels
backport-v7.17.0 Automated backport with mergify backport-v8.0.0 Automated backport with mergify backport-v8.1.0 Automated backport with mergify impact:critical Immediate priority; high value or cost to the product. Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

Comments

@jlind23
Copy link
Collaborator

jlind23 commented Feb 9, 2022

When using filestream with multiple input you can specify for each input an id such as follow:

` - type: filestream
enabled: true
id: id1
paths:
- /var/log/k8sapps/myapp/*.log
fields:
app.name: "myapp"
fields_under_root: true

- type: filestream
  enabled: true
  id: id2
  paths:
   - /var/log/k8sapps/secondapp/*.log
  fields:
   app.name: "secondapp"
  fields_under_root: true`

For now there is a bug and Elastic Agent does not pass down this id parameter to its filestream inputs
@jlind23 jlind23 added the impact:critical Immediate priority; high value or cost to the product. label Feb 9, 2022
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 9, 2022
@jlind23 jlind23 mentioned this issue Feb 9, 2022
Closed
@jlind23 jlind23 added backport-v8.0.0 Automated backport with mergify backport-v8.1.0 Automated backport with mergify Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team labels Feb 9, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 9, 2022
@ph
Copy link
Contributor

ph commented Feb 9, 2022

@jlind23 I see this labeled as critical what is the effect of not propagating the id down to the input?

@jlind23
Copy link
Collaborator Author

jlind23 commented Feb 9, 2022 •
edited

It causes data duplication when multiple inputs are set in filestream config.
#30061

@ph
Copy link
Contributor

ph commented Feb 9, 2022

We can populate down, but I do agree with @kvch we would need to investigate the root cause.

@jlind23
Copy link
Collaborator Author

jlind23 commented Feb 9, 2022

Yes, but even if the rootcause is found, it is still a bug on our end to not populate parameters.

@ph ph mentioned this issue Feb 14, 2022
Merged
6 tasks
@ph
Copy link
Contributor

ph commented Feb 14, 2022 •
edited

Yes, but even if the rootcause is found, it is still a bug on our end to not populate parameters.

@kvch does every input now support an ID?

@kvch
Copy link
Contributor

kvch commented Feb 15, 2022

Yes, all inputs support IDs in Filebeat. Passing down IDs is also important for log, not only filestream. Otherwise, we cannot collect the same file from multiple inputs.

@ph
Copy link
Contributor

ph commented Feb 15, 2022

@kvch Ok, well it's only done for FB input in the linked PR.

@ph
Copy link
Contributor

ph commented Feb 15, 2022

@kvch file from multiple inputs I didn't know that, before we an error statement in the log, can you tell me went this change got introduced? is that available in 7.12?

@ph ph mentioned this issue Mar 10, 2022
Merged
@jlind23 jlind23 added the backport-v7.17.0 Automated backport with mergify label Mar 14, 2022
ph added a commit to ph/beats that referenced this issue Mar 14, 2022
@ph
Propage input id from the Agent policy into Filebeat configuration
751172c 
This allow an ID defined on the Agent Policy for any inputs supported by
Filebeat to be propaged down to the Filebeat configuration.

Note: No validation is done on the field, if the field is present it will sent to
Filebeat.

Fixes: elastic#30300
ph added a commit that referenced this issue Mar 16, 2022
@ph
Propagate input id from the Agent policy into Filebeat configuration (#…
b318f6e 
…30386)

* Propage input id from the Agent policy into Filebeat configuration

This allow an ID defined on the Agent Policy for any inputs supported by
Filebeat to be propaged down to the Filebeat configuration.

Note: No validation is done on the field, if the field is present it will sent to
Filebeat.

Fixes: #30300
mergify bot pushed a commit that referenced this issue Mar 16, 2022
@ph
Propagate input id from the Agent policy into Filebeat configuration (#…
0b21773 
…30386)

* Propage input id from the Agent policy into Filebeat configuration

This allow an ID defined on the Agent Policy for any inputs supported by
Filebeat to be propaged down to the Filebeat configuration.

Note: No validation is done on the field, if the field is present it will sent to
Filebeat.

Fixes: #30300
(cherry picked from commit b318f6e)

# Conflicts:
#	x-pack/elastic-agent/pkg/agent/program/supported.go
mergify bot pushed a commit that referenced this issue Mar 16, 2022
@ph
Propagate input id from the Agent policy into Filebeat configuration (#…
9f3fc45 
…30386)

* Propage input id from the Agent policy into Filebeat configuration

This allow an ID defined on the Agent Policy for any inputs supported by
Filebeat to be propaged down to the Filebeat configuration.

Note: No validation is done on the field, if the field is present it will sent to
Filebeat.

Fixes: #30300
(cherry picked from commit b318f6e)
mergify bot pushed a commit that referenced this issue Mar 16, 2022
@ph
Propagate input id from the Agent policy into Filebeat configuration (#…
9610ee7 
…30386)

* Propage input id from the Agent policy into Filebeat configuration

This allow an ID defined on the Agent Policy for any inputs supported by
Filebeat to be propaged down to the Filebeat configuration.

Note: No validation is done on the field, if the field is present it will sent to
Filebeat.

Fixes: #30300
(cherry picked from commit b318f6e)
ph added a commit that referenced this issue Mar 17, 2022
@mergify @ph
[7.17](backport #30386) Propagate input id from the Agent policy into…
f9c6121 
… Filebeat configuration (#30862)

* Propagate input id from the Agent policy into Filebeat configuration (#30386)


This allows an ID defined on the Agent Policy for any inputs supported by
Filebeat to be propagated down to the Filebeat configuration.

Note: No validation is done on the field, if the field is present it will be sent to
Filebeat.

Fixes: #30300
(cherry picked from commit b318f6e)

Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
ph added a commit that referenced this issue Mar 17, 2022
@mergify @ph
Propagate input id from the Agent policy into Filebeat configuration (#…
65d3d20 
…30386) (#30863)

This allows an ID defined on the Agent Policy for any inputs supported by
Filebeat to be propagated down to the Filebeat configuration.

Note: No validation is done on the field, if the field is present it will be sent to
Filebeat.

Fixes: #30300
(cherry picked from commit b318f6e)

Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
ph added a commit that referenced this issue Mar 17, 2022
@ph
Propagate input id from the Agent policy into Filebeat configuration (#…
a4e546e 
…30386)

* Propage input id from the Agent policy into Filebeat configuration

This allow an ID defined on the Agent Policy for any inputs supported by
Filebeat to be propaged down to the Filebeat configuration.

Note: No validation is done on the field, if the field is present it will sent to
Filebeat.

Fixes: #30300
(cherry picked from commit b318f6e)
ph added a commit that referenced this issue Mar 18, 2022
@mergify @ph
[8.1](backport #30386) Propagate input id from the Agent policy into …
4389079 
…Filebeat configuration (#30864)

* Propagate input id from the Agent policy into Filebeat configuration (#30386)

* Propage input id from the Agent policy into Filebeat configuration

This allow an ID defined on the Agent Policy for any inputs supported by
Filebeat to be propaged down to the Filebeat configuration.

Note: No validation is done on the field, if the field is present it will sent to
Filebeat.

Fixes: #30300
(cherry picked from commit b318f6e)

* changelog

Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
kush-elastic pushed a commit to kush-elastic/beats that referenced this issue May 2, 2022
@ph @kush-elastic
Propagate input id from the Agent policy into Filebeat configuration (e…
ded5456 
…lastic#30386)

* Propage input id from the Agent policy into Filebeat configuration

This allow an ID defined on the Agent Policy for any inputs supported by
Filebeat to be propaged down to the Filebeat configuration.

Note: No validation is done on the field, if the field is present it will sent to
Filebeat.

Fixes: elastic#30300
leweafan pushed a commit to leweafan/beats that referenced this issue Apr 28, 2023
@mergify @ph
Propagate input id from the Agent policy into Filebeat configuration (e…
09bf064 
…lastic#30386) (elastic#30863)

This allows an ID defined on the Agent Policy for any inputs supported by
Filebeat to be propagated down to the Filebeat configuration.

Note: No validation is done on the field, if the field is present it will be sent to
Filebeat.

Fixes: elastic#30300
(cherry picked from commit 24f6449)

Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
chrisberkhout pushed a commit that referenced this issue Jun 1, 2023
@ph @chrisberkhout
Propagate input id from the Agent policy into Filebeat configuration (#…
8c5e14d 
…30386)

* Propage input id from the Agent policy into Filebeat configuration

This allow an ID defined on the Agent Policy for any inputs supported by
Filebeat to be propaged down to the Filebeat configuration.

Note: No validation is done on the field, if the field is present it will sent to
Filebeat.

Fixes: #30300
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ph ph

Labels
backport-v7.17.0 Automated backport with mergify backport-v8.0.0 Automated backport with mergify backport-v8.1.0 Automated backport with mergify impact:critical Immediate priority; high value or cost to the product. Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team
Projects
None yet
Milestone
No milestone
4 participants
@ph @kvch @elasticmachine @jlind23