Propagate input id from the Agent policy into Filebeat configuration #30386

ph · 2022-02-14T22:12:57Z

This allow an ID defined on the Agent Policy for any inputs supported by
Filebeat to be propaged down to the Filebeat configuration.

Note: No validation is done on the field, if the field is present it will sent to
Filebeat.

Fixes: #30300

What does this PR do?

Why is it important?

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine · 2022-02-14T22:12:59Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

elasticmachine · 2022-02-15T00:02:48Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-03-14T18:46:57.478+0000
Duration: 93 min 25 sec

Test stats 🧪

Test	Results
Failed	0
Passed	6349
Skipped	14
Total	6363

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

Backwards compatibility

kvch

First, I accidentally approved the PR. The code looks good to me, and this should help with position tracking going forward.

My concern is given that IDs are used for finding state information in the registry file, changing an ID of an input means that Filebeat will not find the existing states. Thus, the files for existing inputs will be resent.

I do think passing down the IDs are needed otherwise we cannot support reading from the same file from multiple inputs. We just have to be careful how we roll it out.

ph · 2022-02-15T13:18:36Z

@kvch Yes, this is a valid concern, I don't think this is something I can enforce by any means, I would prefer we fix it on the filestream. This is sadly the best I can do on my side. In older log input we were using the file inode as the main identification of the files, with the downside of possible inode reuse on the mounted drive. I don't what is different in the filestream implementation.

Concerning the ID from Fleet, I would expect the ID to be consistent, I would assume that when a user add an integration to an agent policy and it edits the configuration the ID will stay the same until the integration is removed from a policy and added again. @joshdover or @nchaulet can you clarify the stability of the id for input in fleet?

ph · 2022-02-15T13:33:43Z

@kvch Would you consider the stability of the ID and passing the ID two different issues? In that case, you should test/review/approve this pull request.

AndersonQ · 2022-02-15T13:52:45Z

x-pack/elastic-agent/spec/filebeat.yml

    - copy_all_to_list:
        to: streams
        on_conflict: noop
-        except: ["streams", "id", "enabled", "processors"]
+        except: ["streams", "enabled", "processors"]


I imagine that there is a test that checks that all the fields, except these are passes to filebeat. Am I correct?

Yes, this is exactly that. If you look at the golden files in this PR, you see how an agent policy is transformed into a filebeat policy.

blakerouse · 2022-02-15T14:30:46Z

@ph @kvch I think the open question here is about backwards compatibility. What will Filebeat do? If Filebeat would take the inode as the main id first then merge the id passed into the configuration, that would allow an ID to be added to an input after the fact.

ph · 2022-02-15T14:41:58Z

@blakerouse Yes, I think this is an interesting question, I don't know exactly the changes that went into the filestream input with the disambiguation.

mergify · 2022-02-15T16:57:50Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b propagate-id-from-agent-to-filebeat upstream/propagate-id-from-agent-to-filebeat
git merge upstream/main
git push upstream propagate-id-from-agent-to-filebeat

kvch · 2022-02-15T17:17:54Z

I think the open question here is about backwards compatibility. What will Filebeat do? If Filebeat would take the inode as the main id first then merge the id passed into the configuration, that would allow an ID to be added to an input after the fact.

At the moment there is no such support. But we can implement some heuristics for backwards compatibility. We are also considering forcing users to set IDs in Filebeat. If we decide to do it we have to migrate old state IDs to the new ones. I have to sleep on it. :D But it is something that should be resolved in Filebeat, not in Agent. That is for sure.

kvch · 2022-02-16T12:50:41Z

We decided to enforce IDs for filestream inputs. We will take care of migrating existing entries in the registry. So adding this change to Agent should be safe. Once that is merged, I will approve this PR.

kvch · 2022-02-16T14:59:36Z

Please add a changelog entry.

ph · 2022-02-16T18:12:44Z

I just learned that id can be the same for different inputs, this is a bummer.

ph · 2022-02-17T16:34:37Z

rebased waiting on ci.

ph · 2022-02-22T13:55:45Z

yes, we can hold on.

…

On Tue, Feb 22, 2022 at 6:16 AM Noémi Ványi ***@***.***> wrote: Can we hold on with merging this PR? There is no support for migrating old IDs in Filebeat at the moment. — Reply to this email directly, view it on GitHub <#30386 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAFAEKCKMJ7A5UNXKXAVDU4NWANANCNFSM5OMXLKLQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- -- ph

mergify · 2022-03-10T14:07:00Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b propagate-id-from-agent-to-filebeat upstream/propagate-id-from-agent-to-filebeat
git merge upstream/8.1
git push upstream propagate-id-from-agent-to-filebeat

ph · 2022-03-10T19:06:16Z

done in elastic/elastic-agent#196 in the Elastic Agent repository for 8.2

ph · 2022-03-14T18:42:52Z

@mergify.io update

This allow an ID defined on the Agent Policy for any inputs supported by Filebeat to be propaged down to the Filebeat configuration. Note: No validation is done on the field, if the field is present it will sent to Filebeat. Fixes: elastic#30300

jlind23 · 2022-03-16T07:53:54Z

/package

ph · 2022-03-16T14:30:59Z

I am going to merge this, I need to investigate the failures, looking at the test this seems to be an issue with the way test are bootstrapped.

…30386) * Propage input id from the Agent policy into Filebeat configuration This allow an ID defined on the Agent Policy for any inputs supported by Filebeat to be propaged down to the Filebeat configuration. Note: No validation is done on the field, if the field is present it will sent to Filebeat. Fixes: #30300 (cherry picked from commit b318f6e) # Conflicts: # x-pack/elastic-agent/pkg/agent/program/supported.go

…30386) * Propage input id from the Agent policy into Filebeat configuration This allow an ID defined on the Agent Policy for any inputs supported by Filebeat to be propaged down to the Filebeat configuration. Note: No validation is done on the field, if the field is present it will sent to Filebeat. Fixes: #30300 (cherry picked from commit b318f6e)

… Filebeat configuration (#30862) * Propagate input id from the Agent policy into Filebeat configuration (#30386) This allows an ID defined on the Agent Policy for any inputs supported by Filebeat to be propagated down to the Filebeat configuration. Note: No validation is done on the field, if the field is present it will be sent to Filebeat. Fixes: #30300 (cherry picked from commit b318f6e) Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>

…30386) (#30863) This allows an ID defined on the Agent Policy for any inputs supported by Filebeat to be propagated down to the Filebeat configuration. Note: No validation is done on the field, if the field is present it will be sent to Filebeat. Fixes: #30300 (cherry picked from commit b318f6e) Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>

…30386) * Propage input id from the Agent policy into Filebeat configuration This allow an ID defined on the Agent Policy for any inputs supported by Filebeat to be propaged down to the Filebeat configuration. Note: No validation is done on the field, if the field is present it will sent to Filebeat. Fixes: #30300 (cherry picked from commit b318f6e)

…Filebeat configuration (#30864) * Propagate input id from the Agent policy into Filebeat configuration (#30386) * Propage input id from the Agent policy into Filebeat configuration This allow an ID defined on the Agent Policy for any inputs supported by Filebeat to be propaged down to the Filebeat configuration. Note: No validation is done on the field, if the field is present it will sent to Filebeat. Fixes: #30300 (cherry picked from commit b318f6e) * changelog Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>

…lastic#30386) * Propage input id from the Agent policy into Filebeat configuration This allow an ID defined on the Agent Policy for any inputs supported by Filebeat to be propaged down to the Filebeat configuration. Note: No validation is done on the field, if the field is present it will sent to Filebeat. Fixes: elastic#30300

…lastic#30386) (elastic#30863) This allows an ID defined on the Agent Policy for any inputs supported by Filebeat to be propagated down to the Filebeat configuration. Note: No validation is done on the field, if the field is present it will be sent to Filebeat. Fixes: elastic#30300 (cherry picked from commit 24f6449) Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>

…30386) * Propage input id from the Agent policy into Filebeat configuration This allow an ID defined on the Agent Policy for any inputs supported by Filebeat to be propaged down to the Filebeat configuration. Note: No validation is done on the field, if the field is present it will sent to Filebeat. Fixes: #30300

ph added backport-v8.0.0 Automated backport with mergify Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team backport-v8.1.0 Automated backport with mergify backport-v8.2.0 Automated backport with mergify labels Feb 14, 2022

ph requested review from kvch and a team February 14, 2022 22:12

ph self-assigned this Feb 14, 2022

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Feb 14, 2022

kvch previously approved these changes Feb 15, 2022

View reviewed changes

kvch reviewed Feb 15, 2022

View reviewed changes

AndersonQ reviewed Feb 15, 2022

View reviewed changes

ph changed the title ~~Propage input id from the Agent policy into Filebeat configuration~~ Propagate input id from the Agent policy into Filebeat configuration Feb 15, 2022

kvch approved these changes Feb 16, 2022

View reviewed changes

belimawr mentioned this pull request Feb 17, 2022

input/filestream: filestream input ID is required #30401

Merged

5 tasks

ph force-pushed the propagate-id-from-agent-to-filebeat branch from c7f6564 to 97e1951 Compare February 17, 2022 16:34

ph changed the base branch from main to 8.1 March 10, 2022 14:06

ph requested review from a team as code owners March 10, 2022 14:06

ph changed the base branch from 8.1 to main March 10, 2022 14:07

v1v removed the request for review from a team March 10, 2022 14:39

ph added 2 commits March 14, 2022 14:46

Add changelog entry

4aa7e62

ph force-pushed the propagate-id-from-agent-to-filebeat branch from 5a7b924 to 4aa7e62 Compare March 14, 2022 18:46

ph added the backport-v7.17.0 Automated backport with mergify label Mar 16, 2022

ph merged commit b318f6e into elastic:main Mar 16, 2022

mergify bot mentioned this pull request Mar 16, 2022

[7.17](backport #30386) Propagate input id from the Agent policy into Filebeat configuration #30862

Merged

mergify bot mentioned this pull request Mar 16, 2022

[8.0](backport #30386) Propagate input id from the Agent policy into Filebeat configuration #30863

Merged

mergify bot mentioned this pull request Mar 16, 2022

[8.1](backport #30386) Propagate input id from the Agent policy into Filebeat configuration #30864

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Propagate input id from the Agent policy into Filebeat configuration #30386

Propagate input id from the Agent policy into Filebeat configuration #30386

ph commented Feb 14, 2022

elasticmachine commented Feb 14, 2022

elasticmachine commented Feb 15, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

kvch left a comment

ph commented Feb 15, 2022 •

edited

Loading

ph commented Feb 15, 2022

AndersonQ Feb 15, 2022

ph Feb 15, 2022

blakerouse commented Feb 15, 2022

ph commented Feb 15, 2022

mergify bot commented Feb 15, 2022

kvch commented Feb 15, 2022

kvch commented Feb 16, 2022

kvch commented Feb 16, 2022

ph commented Feb 16, 2022

ph commented Feb 17, 2022

ph commented Feb 22, 2022 via email

mergify bot commented Mar 10, 2022

ph commented Mar 10, 2022

ph commented Mar 14, 2022

jlind23 commented Mar 16, 2022

ph commented Mar 16, 2022

Propagate input id from the Agent policy into Filebeat configuration #30386

Propagate input id from the Agent policy into Filebeat configuration #30386

Conversation

ph commented Feb 14, 2022

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Feb 14, 2022

elasticmachine commented Feb 15, 2022 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

kvch left a comment

Choose a reason for hiding this comment

ph commented Feb 15, 2022 • edited Loading

ph commented Feb 15, 2022

AndersonQ Feb 15, 2022

Choose a reason for hiding this comment

ph Feb 15, 2022

Choose a reason for hiding this comment

blakerouse commented Feb 15, 2022

ph commented Feb 15, 2022

mergify bot commented Feb 15, 2022

kvch commented Feb 15, 2022

kvch commented Feb 16, 2022

kvch commented Feb 16, 2022

ph commented Feb 16, 2022

ph commented Feb 17, 2022

ph commented Feb 22, 2022 via email

mergify bot commented Mar 10, 2022

ph commented Mar 10, 2022

ph commented Mar 14, 2022

jlind23 commented Mar 16, 2022

ph commented Mar 16, 2022

elasticmachine commented Feb 15, 2022 •

edited by jenkins-beats-ci bot

Loading

ph commented Feb 15, 2022 •

edited

Loading