[UnifiedLogs] Add data_stream.dataset option to the unifiedlogs input package

[leandrojmp]

Similarly to other input packages, we want to add the ability to customize the dataset name for better data organization.

eg https://github.com/elastic/integrations/blob/main/packages/winlog/manifest.yml#L33

Edited, original issue below

Integration Name

Custom macOS Unified Logs [unifiedlogs]

Dataset Name

No response

Integration Version

0.4.0

Agent Version

It is not relevant

Agent Output Type

elasticsearch

Elasticsearch Version

It is not relevant

OS Version and Architecture

It is not relevant

Software/API Version

No response

Error Message

The Index Template for the UnifiedLogs integration is not using the correct index pattern, the template logs-unifiedlogs is using the index pattern as logs-unifiedlogs-*, this index pattern is wrong and does not follows the data stream naming scheme of integrations.

By using the wrong index pattern, the data from the integration will match the catch all index template logs, and will ignore both the integration custom template logs-unifiedlogs@custom and also the integration ingest pipeline, it will also not have the correct index.default_pipeline and index.final_pipeline set.

Event Original

No response

What did you do?

Add the Custom UnifiedLogs integration to an ingest pipeline.

What did you see?

The index pattern does not match the data stream name

What did you expect to see?

The index pattern should match the correct data stream name, so it needs to be changed to logs-unifiedlogs.*.

Anything else?

No response

[elasticmachine]

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)

[marc-gr]

unifiedlogs is an input package, so it does not have datastreams. This means the index template pattern is as expected and should not cause issues. Have you experienced any?

OTOH we allow to customize the datastream in other input packages for better organization, so I still think this is a valid enhancement to do.

I will edit this issue to this ER if that is fine.

[leandrojmp]

Hello @marc-gr

As mentioned in the original issue, the template does not work, it does not match the datastream created by the integration, I'm not sure what you mean with it does not have datastreams.

Also the changed description is unrelated to the original issue, the user already can customize the dataset name.

When configuring the integration you select the data stream type and the data set name, so it will create a datastream following the naming scheme for integration datastreams.

Those settings will create a datastream with the name: logs-unifiedlogs.unifiedlogs-<namespace>.

The issue I reported is that the template logs-unifiedlogs does not match this datastream, it will match logs-unifiedlogs-*, it should match logs-unifiedlogs.*-* or logs-unifiedlogs.*

After editing the template and adding logs-unifiedlogs.* to the index patterns, the template and the ingest pipeline started working.

I think that the issue name should be reverted to reflect the original problem.

[leandrojmp]

Hello @marc-gr any input on this? Can I revert back the title of the issue? It is not related to the user being able to set the dataset name, this is alreayd possible, it is related to an error in the template.