OTLP: Support receiving logs in Loki using OpenTelemetry OTLP

[jpkrohling]

What this PR does / why we need it:
This change brings support to ingesting OTLP as part of the main
ingester. This uses the standard OpenTelemetry Collector pdata
Logs gRPC server as the interface and the translator package
from the contrib repository. Once the data is translated from OTLP
to a Loki's PushRequest, we call use the Ingester's Push function.

Which issue(s) this PR fixes:
Fixes #5346

Special notes for your reviewer:
Most of this change is about bringing new dependencies. The code itself is quite simple, just making bridges. I tested this with an OpenTelemetry Collector Contrib with the following configuration:

receivers:
  journald:
    directory: /var/log/journal/a04e3a44cdd740f88d6a7ae3bb8c70cf

exporters:
  logging:
    loglevel: debug
  otlp:
    endpoint: http://localhost:9095
    tls:
      insecure: true

processors:
  attributes:
    actions:
    - action: insert
      key: loki.attribute.labels
      value: http_status_code

    # the following attributes are added manually here in the example, but would
    # probably be added by other processors or straight from the source
    - action: insert
      key: http_status_code
      value: 500

  resource:
    attributes:
    - action: insert
      key: loki.resource.labels
      value: host_name, pod_name

    # the following attributes are added manually here in the example, but would
    # probably be added by other processors or straight from the source
    - action: insert
      key: host_name
      value: guarana
    - action: insert
      key: pod_name
      value: guarana-pod-01

extensions:

service:
  extensions:
  pipelines:
    logs:
      receivers: [journald]
      processors: [resource, attributes]
      exporters: [logging, otlp]

Checklist

• Reviewed the CONTRIBUTING.md guide
• Documentation added
• Tests updated
• CHANGELOG.md updated
• Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

Signed-off-by: Juraci Paixão Kröhling juraci@kroehling.de

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[jpkrohling]

I'm not sure what's the best place to document this. There aren't configuration options, the only things to mention would be:

• state that OTLP is accepted in the main gRPC port
• provide an example configuration file, like the one I posted in the issue description

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0.1%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0.1%

[jpkrohling]

There's something wrong with the CLA check. I signed, it says I'm signed, but the check is still failing:

[MichelHollands]

Suggested change

	// into Loki's PushRequest and forwards the requeste to the provided PusherServer.
	// into Loki's PushRequest and forwards the request to the provided PusherServer.

[MichelHollands]

Are all these version changes needed for oltp? If not I'd prefer it if they were in a separate PR.

[jpkrohling]

I'm closing this as we decided not to pursue this at the moment.

[evantorrie]

I'm closing this as we decided not to pursue this at the moment.

This is still something that is high on our list of desired features.

Is there still a desire on Grafana's part to deliver native OTLP ingestion for Loki?

[jpkrohling]

Is there still a desire on Grafana's part to deliver native OTLP ingestion for Loki?

Right now, we are working on a proxy for Grafana Cloud that would be able to ingest all three signals. Native OTLP ingestion for Loki will eventually land here, but some other base work has to be completed before.

[slim-bean]

Native OTLP ingestion is definitely on our list, but we are still working out how best to store the additional structured data sent with OTLP.

The resource data should map well to labels in Loki, however the attributes and other fields like trace and span may/will be too high cardinality to put in the index.

This PR takes the same approach as the Loki exporter which allows selective mapping of attributes to labels but this only will work for low cardinality attributes.

We recommend using the collector and exporter until we can natively support and store all the data in the OTLP payload directly in Loki, however to do this we need to make some store changes to make a place for structured data.

[feldentm-SAP]

@slim-bean are there any plans to extend the Loki data store to handle attributes like traceID/spanID or other correlators efficiently? I mean this would be required before dumping all the OTLP-compliant data into Loki.

[slim-bean]

Yes there are plans for this! Most of us are just getting back up to speed after some holiday time off and I know there are a few meetings scheduled to talk about this in the coming weeks.

Optimistically I hope we can have something started in Q1 this year.

[frzifus]

Is there some kind of public roadmap?