Add Elastic APM Intake OpenTelemetry Input Package#17114
Conversation
This package enables the EDOT Collector to receive APM data from Elastic APM agents using the elasticapmintakereceiver. It allows existing Elastic APM agents to send telemetry data to an OpenTelemetry Collector instead of directly to APM Server. Key features: - Accepts APM Intake V2 protocol data (traces, metrics, logs) - Configurable endpoint for APM agent connections - Optional TLS/mTLS support for encrypted connections - Optional agent configuration fetching from Elasticsearch Closes #17113 Co-authored-by: Julien Lind <jlind23@users.noreply.github.com>
|
@claude review this please |
|
Claude finished @jlind23's task in 1m 35s —— View job PR Review CompleteI've reviewed this PR which adds a new Overall: The package structure follows the established patterns for OTel input packages. I have a few observations and one issue to address. 🟡 Medium - Changelog Link References Wrong PR NumberFile: The changelog references PR #17113, but this is PR #17114. The link should point to the current PR. link: https://github.com/elastic/integrations/pull/17113Should be: link: https://github.com/elastic/integrations/pull/17114⚪ Low - Naming Inconsistency (Author's Discretion)Files: There's a minor inconsistency between the folder name and the package name in the manifest:
The 💬 Nitpicks / Observations
SummaryThis is a well-structured package. The only actionable item is fixing the changelog link to reference the correct PR number (#17114 instead of #17113). Recommendation: COMMENT (non-blocking) - fix the changelog link before merge. Why is Claude responding? | Type Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
The package structure looks good and follows the existing patterns for OTel input packages. One issue to address:
Changelog link points to wrong URL: The changelog links to issue #17113 instead of this PR #17114.
The package passes elastic-package check validation. The TLS configuration follows the same pattern as other packages (universal_profiling_collector, apm).
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
|
@mrodm The policy test was failing inconsistently because Fleet/Kibana assigns component IDs non-deterministically when there are multiple signal types (traces, metrics, logs). The component IDs like logs/componentid-2, metrics/componentid-1, traces/componentid-0 would change between test runs. Any chance there is a way to set fixed component id? |
There was a problem hiding this comment.
The agent configuration feature (agent_config_enabled) is incomplete. The elasticapmintakereceiver requires an agent_config.elasticsearch configuration block with connection settings to fetch APM Central Configurations from Elasticsearch. Without exposing these settings, enabling agent_config will cause runtime failures.
Options to address:
- Add elasticsearch connection variables (endpoint, API key/credentials) to the manifest and template
- Remove the agent_config feature until it can be fully implemented
- Document that agent_config requires additional manual configuration outside of this package
Additionally, consider making TLS cert/key fields visible (show_user: true) when TLS is enabled, as users need to provide these paths.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
|
Converting this back to draft for now since this is blocked on multi-signal support: elastic/package-spec#1023 |
@jlind23 taking a look. |
@jlind23 Looking at the test policy, I think the expected test case should be updated to set first traces, then metrics and finally logs in the --- packages/elasticapm_input_otel/_dev/test/policy/test-default.expected
+++ packages/elasticapm_input_otel/_dev/test/policy/test-default.expected
@@ -36,12 +36,12 @@ receivers:
secret_references: []
service:
pipelines:
- logs:
+ traces:
exporters:
- elasticsearch/componentid-0
receivers:
- forward
- logs/componentid-0:
+ traces/componentid-0:
exporters:
- forward
processors:
@@ -62,12 +62,12 @@ service:
- transform/componentid-1
receivers:
- elasticapmintake/componentid-0
- traces:
+ logs:
exporters:
- elasticsearch/componentid-0
receivers:
- forward
- traces/componentid-2:
+ logs/componentid-2:
exporters:
- forward
processors:That is the same order that appears in the I think the --- packages/elasticapm_input_otel/agent/input/input.yml.hbs
+++ packages/elasticapm_input_otel/agent/input/input.yml.hbs
@@ -16,12 +16,12 @@ processors:
detectors: ["system"]
service:
pipelines:
- traces/0:
+ traces:
receivers: [elasticapmintake]
processors: [resourcedetection/system]
- metrics/0:
+ metrics:
receivers: [elasticapmintake]
processors: [resourcedetection/system]
- logs/0:
+ logs:
receivers: [elasticapmintake]
processors: [resourcedetection/system] |
andrewkroh
added
New Integration
- Bump manifest format version to 3.6.0. - Add support for API key authentication and related configuration options. - Introduce new variables for TLS client CA file and API key validation endpoint. - Update README to include API key authentication details and examples. - Add new test policies for authentication and TLS configurations.
…sue-17113-20260129-0904
✅ Vale Linting ResultsNo issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
There was a problem hiding this comment.
The package has one issue that must be fixed before merge:
elastic-package check fails: The docs/README.md file contains a troubleshooting bullet point that doesn't exist in the source template (_dev/build/docs/README.md). Either add the missing content to the source template and rebuild, or remove it from docs/README.md by running elastic-package build.
Run elastic-package check locally to verify the fix.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| traces: | ||
| receivers: [elasticapmintake] | ||
| processors: [resourcedetection/system] | ||
|
|
There was a problem hiding this comment.
💬 NITPICK Trailing whitespace and blank line
Why: Template has a trailing blank line with spaces at the end
| processors: [resourcedetection/system] |
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
…README.md to streamline troubleshooting section.
There was a problem hiding this comment.
Package Validation Failures
elastic-package check fails with two errors:
format_version: 3.6.0is still unreleased (currently "3.6.0-next" in package-spec)dynamic_signal_typesproperty is not allowed in the current spec
Fix: Either downgrade to format_version: 3.5.0 and remove dynamic_signal_types, or wait for the 3.6.0 spec release.
Additionally, the test plan in the PR description shows policy tests are unchecked - these should be verified before merge.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
There was a problem hiding this comment.
Package Validation Fails
elastic-package check fails with validation errors due to using unreleased package spec features:
Error: checking package failed: linting package failed: found 2 validation errors:
1. file "manifest.yml" is invalid: field conditions: agent is required
2. file "manifest.yml" is invalid: field policy_templates.0: Additional property dynamic_signal_types is not allowed
Root Cause
The package uses format_version: 3.6.0 which is still 3.6.0-next (unreleased) in the package-spec. The dynamic_signal_types field was added in version 3.6.0 but is not yet available.
Options to Fix
-
Wait for package-spec 3.6.0 release - Coordinate with the package-spec maintainers to ensure 3.6.0 is released before merging this PR.
-
Downgrade to format_version 3.5.6 - Remove the
dynamic_signal_typesfield and updateformat_versionto 3.5.6. The package may lose automatic signal type handling, but will be valid.
Other Notes
- TLS and API key authentication options reference conditionally required fields (cert/key files, ES endpoint) that aren't validated at schema level. This is acceptable given Fleet UI limitations, but ensure documentation clearly communicates these dependencies.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
…t Elastic Stack version 9.4.0 or later in manifest and documentation.
|
Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as |
andrewkroh
removed
the
New Integration
…sue-17113-20260129-0904
…ceiver - Removed unsupported agent configuration options from the manifest. - Enhanced descriptions for endpoint, TLS settings, and API key authentication in the manifest and README. - Added a new optional field for Data Stream Dataset to allow customization of the dataset used for routing APM data. - Updated expected test outputs to reflect changes in the configuration structure. - Improved clarity in README regarding connection settings for remote APM agents.
…M Input - Introduced sample_event_logs.json, sample_event_metrics.json, and sample_event_traces.json to provide example data structures for logging, metrics, and tracing. - Updated sample_event.json with new timestamps, IDs, and runtime information. - Added new configuration files for testing logs, metrics, and traces, ensuring a minimum count of 50 for assertions. - Adjusted the default configuration to reflect the new testing requirements.
…APM Input - Updated sample_event_logs.json, sample_event_metrics.json, and sample_event_traces.json with new timestamps, IDs, and runtime information. - Adjusted sample_event.json to reflect changes in error attributes and stack traces. - Modified Dockerfile to use Go 1.26 and Alpine 3.22, ensuring compatibility with updated dependencies. - Updated go.mod and go.sum to reflect new versions of Elastic APM modules and other dependencies.
- Corrected wording from "older protocols" to "earlier protocols" for clarity. - Changed example index search instruction from "e.g." to "for example" to enhance readability.
| /packages/elastic_connectors @elastic/search-extract-and-transform | ||
| /packages/elastic_package_registry @elastic/ecosystem | ||
| /packages/elastic_security @elastic/security-service-integrations | ||
| /packages/elasticapm_input_otel @elastic/obs-ds-intake-services |
There was a problem hiding this comment.
i dont recall if this was correct or ecosystem should take ownership for betas?
There was a problem hiding this comment.
I think it would be ecosystem here, @kpollich ?
| receivers: | ||
| elasticapmintake: |
There was a problem hiding this comment.
In the documentation is mentioned to add agent_config:
receivers:
elasticapmintake:
agent_config:
enabled: falseShould we add that unconditionally ?
Reviewing at the code, maybe it can be left for now out ? Since it looks like that by default would be false ? Please, could you check this ?
There was a problem hiding this comment.
As default was false i left it out. I also dont see any documentation regarding the use of the option. Looking around the docs and elastic I see this is opt-in to make changes on the agent management, more than the intake mechanism of it...
the go struct is a bool https://github.com/elastic/opentelemetry-collector-components/blob/8da56e7f281694a146e0f3cc76477ff5ff16dc64/receiver/elasticapmintakereceiver/config.go#L37 so when option is not set, it gets zero value false also.
We can add it, with a note, referring to the official docs, to make it redundant and input developers aware of this option. wdyt?
There was a problem hiding this comment.
We can add it, with a note, referring to the official docs, to make it redundant and input developers aware of this option. wdyt?
I think this option could be good for now. We could add it as hard-coded settings in the agent template. Looking at the code and as you mentioned, setting that to false should be the default behavior.
There was a problem hiding this comment.
updated on latest commit this, policies expected too and codeowners
| /packages/elastic_connectors @elastic/search-extract-and-transform | ||
| /packages/elastic_package_registry @elastic/ecosystem | ||
| /packages/elastic_security @elastic/security-service-integrations | ||
| /packages/elasticapm_input_otel @elastic/obs-ds-intake-services |
There was a problem hiding this comment.
I think it would be ecosystem here, @kpollich ?
- Redesigned the logo SVG to enhance visual clarity and modernize the appearance. - Adjusted dimensions and added new graphical elements for improved aesthetics.
- Changed the default dataset name from 'elasticapmintakereceiver' to 'generic' to align with updated configuration standards. - This change ensures compatibility with new data routing requirements.
…M Input - Updated the dataset name from 'elasticapmintakereceiver' to 'generic' in expected test output files to ensure consistency across tests. - This change aligns with recent updates to configuration standards and improves compatibility with data routing requirements.
…c APM Input - Updated timestamps in sample_event_logs.json, sample_event_metrics.json, sample_event_traces.json, and sample_event.json to reflect the latest data. - Changed IDs and dataset names to 'generic' for consistency across all sample files. - Adjusted host information to ensure alignment with recent changes in the environment.
- Changed the owner in the CODEOWNERS file for the elasticapm_input_otel package from @elastic/obs-ds-intake-services to @elastic/ecosystem for better alignment with project ownership. - Updated the owner in the manifest.yml file for the elasticapm_input_otel package to reflect the new ownership structure. - Added agent_config settings in expected test output files to disable the agent configuration by default, ensuring consistency across test cases.
💚 Build Succeeded
History
|
andrewkroh
added
the
New Integration
|
Package elasticapm_input_otel - 0.1.0 containing this change is available at https://epr.elastic.co/package/elasticapm_input_otel/0.1.0/ |
andrewkroh
added
the
Integration:elasticapm_input_otel
5 participants
Proposed commit message
Add Elastic APM Intake OpenTelemetry Input Package
WHAT: Introduces a new input package
elasticapm_input_otel(v0.1.0) that configures the EDOT (Elastic Distribution of OpenTelemetry) Collector to receive APM data from Elastic APM agents using the upstreamelasticapmintakereceiver. The package is multi-signal —dynamic_signal_types: trueadvertises traces, metrics, and logs in a single policy template, matching the receiver's registered consumers (WithLogs/WithMetrics/WithTraces, all alpha stability per the upstreammetadata.yaml).User-facing configuration exposed in the manifest:
endpoint(defaultlocalhost:8200) — bind address for the intake listener.tls_enabled+tls_cert_file/tls_key_file/tls_client_ca_file— TLS, including mTLS via the optional client CA file. The template renders cert/key/CA fields conditionally on the corresponding values being set, so a misconfiguration surfaces as a schema error rather than an empty path.auth_enabled+auth_elasticsearch_endpoint— gates theapikeyauthextension, which validates Elastic APM API keys against an Elasticsearch cluster (apm/event:writeprivileges, resources[-]).data_stream.dataset(defaultelasticapmintakereceiver) — lets users route APM data into an alternate dataset, mirroring the convention used byotlp_input_otel.Pipeline shape: the rendered collector config wires
elasticapmintakeinto separatelogs,metrics, andtracespipelines, all running throughresourcedetection/system. The transform processor injected by Fleet routesspaneventcontext into the logs data stream alongside standalone APM logs.Test fixture:
_dev/test/policy/):test-default,test-auth(apikeyauth + ES endpoint),test-tls(cert + key) — all with regenerated*.expectedsnapshots._dev/test/system/):test-default-config.yml(smoke, any data lands) plus three per-signal configs —test-only-logs-config.yml(signal_types: [logs]),test-only-metrics-config.yml(signal_types: [metrics]), andtest-only-traces-config.yml(signal_types: [traces, logs], paired because span events route into the logs stream). All four usemin_count: 50and rely onservice_notify_signal: SIGHUPfor startup synchronization._dev/deploy/docker/): a Go HTTP server instrumented with the Elastic APM Go agent. Bumped to Go 1.26, alpine 3.22,go.elastic.co/apm/v2 v2.7.8(andapmhttp/v2,apmzap/v2),go.uber.org/zap v1.28.0. The service emits all three signal types — apmhttp transactions plus three custom child spans (db / cache / external) per request, custom counter/histogram metrics through a registered metrics gatherer, and apmzap Error logs (with trace context). It blocks onSIGHUPuntil the agent applies the policy, then runs continuous load untilSIGTERM.sample_event_logs.json,sample_event_metrics.json,sample_event_traces.json, plussample_event.jsonfor backward compatibility.WHY: Enables existing Elastic APM agents to send telemetry to an OpenTelemetry Collector instead of directly to APM Server, supporting OTel-based observability pipelines. APM data can be processed alongside other OTel sources before export to Elasticsearch, without retrofitting agents to native OTLP.
Known limitations (documented in the README): central agent configuration fetching (
AgentConfig.Elasticsearchon the receiver) is not exposed by this package and remains disabled. RUM intake and pre-V2 protocols are out of scope — only APM Intake V2 is supported.Checklist
changelog.ymlfile.Author's Checklist
elastic-package checkpasses locally._dev/test/policy/*.expected.logs,metrics,traces) config against a local stack.How to test this PR locally
cd packages/elasticapm_input_otel && elastic-package check.elastic-package stack up -v -d.elastic-package test policy -v.elastic-package test system -v. Each per-signal config asserts that its data stream —logs-*-*,metrics-*-*, ortraces-*-*— receives at least 50 events from the Go test app.0.0.0.0:8200if you intend to send from another host.ELASTIC_APM_SERVER_URL=http://<host>:8200).traces-*,metrics-*, andlogs-*withdata_stream.dataset=elasticapmintakereceiver(or a custom dataset if overridden).tls_enabled: trueand provide bothtls_cert_fileandtls_key_file; for API-key auth, setauth_enabled: trueand supply anauth_elasticsearch_endpointreachable from the agent.Related issues
Screenshots