[Custom Log Integration] Add tags & processors to Custom Log datastreams

[nimarezainia]

Background:

As we strive to make Elastic Agent a replacement for the beats, there are some global fields, generally applied on an input datastream, which need to also be available in our integrations. Currently there’s little to no consistency between these integrations, causing a lot of frustration amongst our users.

For more details please refer to: https://github.com/elastic/ingest-dev/issues/995

[fyi @akshay-saraswat ]

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[fearful-symmetry]

@nimarezainia I'm a tad confused by the scope of this. Is this issue for adding a tags and processors field to all integrations? Just a certain subset of logging integrations?

[faec]

[drive-by as I review sprint status] This task looks underspecified -- the design doc linked through the meta issue isn't finalized, and it's unclear what the technical deliverables for this specific issue are, so I don't think "estimation:Day" is a realistic tag.

[nimarezainia]

@fearful-symmetry this is just about adding tags and processors to the "Custom Log" integration. this was done for the system integration also: #2865 (comment)

@kpollich in context of elastic/elastic-package#949 - how should we be pursuing this?

[kpollich]

@nimarezainia It would be best to follow suit with #2865 here and add tags + processors to the Custom Logs integration.

Regarding elastic/elastic-package#949, let's not let this block any work in the short term. We can discuss a path forward and what the priority looks like for managing these fields in a global way elsewhere.

[fearful-symmetry]

@nimarezainia it looks like the main codeowners for that don't work here anymore, so I can probably just add it for this one. In the future if we want to add this to more integrations it should probably be done by the code owners of the given integrations, since the .yml.hbs files are so fragile and CI tends to not catch basic errors, so someone with knowledge of a given integration would have an easier time manually testing in-depth.