add OTLP receiver input package

[graphaelli]

Proposed commit message

add OTLP receiver input package

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

[graphaelli]

Unfortunately we end up with signal (log here) transforms that shouldn't be there. Is there any way to eliminate those?

service:
  pipelines:
    logs/otlp_receiver-otelcol-otelcol-otlp_receiver-otlp_receiver:
      receivers:
        - otlp/otlp_receiver-otelcol-otelcol-otlp_receiver-otlp_receiver
      processors:
        - >-
          transform/otlp_receiver-otelcol-otelcol-otlp_receiver-otlp_receiver-routing
    metrics/otlp_receiver-otelcol-otelcol-otlp_receiver-otlp_receiver:
      receivers:
        - otlp/otlp_receiver-otelcol-otelcol-otlp_receiver-otlp_receiver
      processors:
        - >-
          transform/otlp_receiver-otelcol-otelcol-otlp_receiver-otlp_receiver-routing
    traces/otlp_receiver-otelcol-otelcol-otlp_receiver-otlp_receiver:
      receivers:
        - otlp/otlp_receiver-otelcol-otelcol-otlp_receiver-otlp_receiver
      processors:
        - >-
          transform/otlp_receiver-otelcol-otelcol-otlp_receiver-otlp_receiver-routing
processors:
  transform/otlp_receiver-otelcol-otelcol-otlp_receiver-otlp_receiver-routing:
    log_statements:
      - context: log
        statements:
          - set(attributes["data_stream.type"], "logs")
          - set(attributes["data_stream.dataset"], "otlp_receiver")
          - set(attributes["data_stream.namespace"], "default")

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 845b31a

History

• 💚 Build #34307 succeeded 812c711
• 💔 Build #34269 failed 1e2dce2
• 💔 Build #34268 failed 5bc146d
• 💔 Build #34264 failed dc60d11

[graphaelli]

A few other issues, will need to get these over to fleet:

• we end up only with permissions to write to the data type this input requests and there doesn't appear to be a way to write to all of logs + metrics + traces + profiles
• can't configure elasticapm as the exporter even though it's a connector - https://github.com/elastic/integrations/pull/16003/files#diff-480324e221eb30e92a88eeeb9a01340a857d6e29d757fc7d9dabf91ae369da6fR36-R38

[jsoriano]

Is there any way to eliminate those?

It is not possible to eliminate these processors with current implementation. These are used to route the data to a data stream matching with the index template managed by Fleet, as configured by users.

As input packages work now, they are expected to write to an specific data stream. The user can configure the dataset and namespace, and Fleet configures the template for them, allowing customizations through the @custom component templates and so on.

• we end up only with permissions to write to the data type this input requests and there doesn't appear to be a way to write to all of logs + metrics + traces + profiles

Yes, this is the expected behavior for current implementation, each input package policy is only expected to collect one type of data.

can't configure elasticapm as the exporter even though it's a connector - https://github.com/elastic/integrations/pull/16003/files#diff-480324e221eb30e92a88eeeb9a01340a857d6e29d757fc7d9dabf91ae369da6fR36-R38

For Fleet-managed inputs and integrations, the exporters, or the outputs, are expected to be managed by Fleet, and not included in configuration templates.

connectors could work though, it would be interesting to complete the support for them. Even if not possible to define exporters, it should be possible to define connectors and include them in pipelines, but the truth is that I don't think we have tested this. It would create one connector per policy in any case.

Btw, out of curiosity, this elasticapm connector is used to enrich data, why wasn't it implemented as a processor? To ensure that multiple pipelines can reuse the same instance?

---

This package looks pretty particular, I guess that the idea is to enable the OTLP endpoint and allow the ingestion of any kind of data, that would get routed on ingestion?

I guess that for this case we could add some setting in packages that disables all the logic for index and permission management and the related UI components. It is difficult to estimate the scope of this because many things assume that each policy collects an specific kind of data in an specific data stream, we don't have anything like that yet.

[kpollich]

Yes, this is the expected behavior for current implementation, each input package policy is only expected to collect one type of data.

We will need to address this then as a gap. IIUC OTel receivers can collect multiple signal types and we'll want to support document routing in these input packages https://www.elastic.co/docs/reference/edot-collector/components/elasticsearchexporter#document-routing

We could either generate logs_index + traces_index + metrics_index properties for the Elasticsearch exporter in Fleet based on the input package configuration or rely on dynamic routing here. Not sure which is preferred.

[jsoriano]

Issue created: elastic/package-spec#1023

[graphaelli]

Thank you both.

Btw, out of curiosity, this elasticapm connector is used to enrich data, why wasn't it implemented as a processor? To ensure that multiple pipelines can reuse the same instance?

The elasticapm connector is used to calculate metrics from the various signals. There is a separate elasticapm processor that is used to enrich spans.

Created #16069 to capture requirements for this input since it's going to be more involved than just the pull request. I'll close this and let's move sub issues and further discussion there.