[Logs+] Retroactively annotating logs with service.name

[sorenlouv]

Background

The concept of "services" was introduced with APM but should not be limited to this domain. The plan is to make logs "service-aware" (where applicable like application logs), and thus make it easier to investigate logs per service. If APM instrumentation is enabled it'll be possible to correlate logs to traces.

To do this, logs must be annotated with service.name. Currently, logs are ingested without any reference to services, and users have to manually use the add_fields processor or similar to annotate their logs.

In the new Logs onboarding flow users will be able to provide a service name which will then be used to crate the add_fields processor automatically. This will reduce the barrier for adding service name but it will only work for new integrations.

Suggestion

It should be possible to retroactively specify a service name by navigating to an existing logs integration or logs in Discover, and specifying a service name. This should update the integration (intentionally handwavy) so that new logs are automatically annotated with service name. Bonus points if existing logs are also updated to include the specified service name.

Related:

• [Discuss] Add service.name to service based integrations integrations#6295
• Support service.name as first class citizen elastic-agent#2724

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[ruflin]

@sqren From a technical perspective, can I follow the assumption that logs in a dataset will belong to the same service? Or you expect some more specific queries on top of a dataset?

[sorenlouv]

can I follow the assumption that logs in a dataset will belong to the same service

I think that is correct but I'm not the right person to ask. Perhaps @felixbarny has an opinion/idea about this.

[sorenlouv]

can I follow the assumption that logs in a dataset will belong to the same service

@ruflin Forget what I said above. There is an ongoing discussion around this here (I see you are already engaged). The tldr: customers with 1000s of services most likely wont have a datastream per service because of scalability issue this causes.

[felixbarny]

A dataset can contain logs from multiple services.

more specific queries on top of a dataset

It's important that service.name is not just a derived field but an actual field in the documents. That way we can efficiently filter logs for a specific service and list all services with a terms_enum aggregation.

[ruflin]

A dataset can contain logs from multiple services.

This is a key statement. It means we can't just add a processor to the ingest pipeline for the dataset.

Any ideas on how to map some existing field to specific service names?

[felixbarny]

It means we can't just add a processor to the ingest pipeline for the dataset.

I don't think that's true. We can add a set (or rename if the target field should be deleted) processor to copy a field to service.name that ignores the source field to be missing. For example:

{
  "set": {
    "field": "service.name",
    "copy_from": "app_name",
    "ignore_empty_value": true
  }
}

You can also add multiple similar processors with different values for copy_from.

If there's not a field that can be simply renamed, we can also add a set processor with a condition:

{
  "set": {
    "if": "ctx.container?.name?.contains('com.example/myservice')"
    "field": "service.name",
    "value": "my-service"
  }
}

[ruflin]

Any ideas on how to map some existing field to specific service names?

What @felixbarny put above, answers this question. If we have potential fields that contain the relevant info just in the wrong place / more dynamic, agree it can be done.

[felixbarny]

I think Sørens point was that if the relevant info is not present, there should be an easy way for users to add the service.name. For example, when using the custom logs integration, there should be an easy way to change the config to add a service.name.

In addition to that, we should add a service.name to all existing integrations. And for the k8s integration, we can read out well-known labels, such as app.kubernetes.io/name to provide a service name.

Logs without a service.name should become the exception rather than the norm.

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)