Add the EDOT collector service by ezimuel · Pull Request #67 · elastic/start-local

ezimuel · 2025-07-08T10:46:13Z

This PR add the EDOT (Elastic Distribution of OpenTelemetry) collector in start-local using the option --edot. This is done using the edot_collector configuration reported here.
This PR should address the request in #55.

To test the --edot option we can run the following command:

curl -fsSL https://raw.githubusercontent.com/elastic/start-local/refs/heads/feature/otel/start-local.sh | sh -s -- --edot

To be done:

test the EDOT collector in tests
document the --edot option in README.md

start-local.sh

xrmx · 2025-07-08T12:32:05Z

start-local.sh

+
+      exporters:
+        elasticsearch:
+          endpoint: http://elasticsearch:9200


Suggested change

endpoint: http://elasticsearch:9200

endpoints: [ "http://elasticsearch:9200" ]

Shouldn't this be plural?

I just copy & paste from here where it's singular.

The source of truth should be the collector documentation:
https://www.elastic.co/docs/reference/opentelemetry/edot-collector/config/default-config-standalone

This is the example config for the collector in agent mode (i.e. will export to ES) that will also collect logs and generate metrics from the host https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v9.0.3/internal/pkg/otel/samples/linux/logs_metrics_traces.yml

I think it's a bit too much but maybe hostmetrics are a good source of data to do some smoke testing? i.e. you can check that you can query them in elasticsearch

I need help on this configuration for the docker-compose.yml. The goal is to offer a default settings for EDOT Collector (Standalone) to be used locally for start using the Observability stack in Elastic. I need the config here for the service and the docker specification here.

@rogercoll let's chat to get your support on generating a configuration that can enable us to deploy EDOT collector as a gateway for the start-local effort. So far I have referenced the one we are using for K8s gateway, but we might need to hardcode the endpoints and enrichment processors.

Both endpoint and endpoints configurations are supported in the Elasticsearch exporter. The first one is because the exporter embedding the upstream confighttp configuration: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/elasticsearchexporter/config.go#L69C13-L69C25

The latest was previously added because alignment with elasticsearch-go client configuration: https://github.com/elastic/go-elasticsearch/blob/main/elasticsearch.go#L71

Not a strong opinion, but maybe use endpoints as being the one used in our public docs: https://www.elastic.co/docs/reference/opentelemetry/edot-collector/config/default-config-standalone#data-export

start-local.sh

SylvainJuge · 2025-07-08T14:11:38Z

Do you plan to also generate a dedicated API key for the otel ingestion like it's currently provided for ES clients ? Without that users would have to perform another manual step to generate the API key before being able to send otel data to the otel collector.

Co-authored-by: Riccardo Magliocchetti <riccardo.magliocchetti@gmail.com>

ezimuel · 2025-07-08T14:44:27Z

@SylvainJuge it this API key different from the ES one? Can you point me to any documentation about it? Thanks.

I don't know if it's different, but there is dedicated UI for in in Kibana, also you can use https://www.elastic.co/docs/api/doc/kibana/operation/operation-createagentkey to create a new one for APM agents (which also includes EDOT SDKs).

xeraa · 2025-07-09T16:17:37Z

start-local.sh

-  if  [ -z "${esonly:-}" ]; then
+  if [ "$otel" = "true" ]; then
+    cat >> .env <<- EOM
+ES_LOCAL_JAVA_OPTS="-Xms2g -Xmx2g"


Do we set that Xms to 2GB intentionally? I don't think we'll compare well to Grafana's stack and others like that ("big", "bloated",...). We should IMO still initialize as small as possible with the necessary room to grow if needed. Even if we take a small performance hit when needing to increase the heap size.

@xeraa this is the default configuration proposed here. I'm not an expert of EDOT collector and I asked to the OTel team to help on this.

Yeah, IMO this can (should) use the same settings and general approach used for the other options. We still want this to start really lightweight.

mlunadia

added comments about imaged used and collector config.

mlunadia · 2025-07-21T15:12:47Z

start-local.sh

+  if  [ "$otel" = "true" ]; then
+    cat >> uninstall.sh <<- EOM
+    if docker rmi docker.elastic.co/elastic-agent/elastic-otel-collector:${es_version} >/dev/null 2>&1; then
+      echo "Image docker.elastic.co/elastic-agent/elastic-otel-collector:${es_version} removed successfully"


@ezimuel we should be using the image of Elastic Agent and trigger otel mode, there is a flag that enables the Elastic Agent container to start in otel mode = EDOT

@mlunadia Are we interested in any other Elastic Agent binaries/features except the Otel collector? If we are only interested in the EDOT collector I would recommend using the elastic-agent/elastic-otel-collector image as being smaller than the main elastic-agent image: elastic/elastic-agent#7173

just EDOT collector for start-local, for edge we might add examples later

mlunadia · 2025-07-21T15:18:57Z

start-local.sh

+  # Add the OTLP configs in docker-compose.yml
+  cat >> docker-compose.yml <<-'EOM'
+configs:
+  # This is the minimal yaml configuration needed to listen on all interfaces


We'll need to work on this config as we are missing some of the core processors like batch. As reference we can begin using this configuration with the exception of the inframetrics processor which is marked for removal in it. https://github.com/elastic/elastic-agent/blob/main/internal/pkg/otel/samples/linux/gateway.yml

mlunadia · 2025-07-21T15:21:52Z

start-local.sh

+
+      exporters:
+        elasticsearch:
+          endpoint: http://elasticsearch:9200


@rogercoll let's chat to get your support on generating a configuration that can enable us to deploy EDOT collector as a gateway for the start-local effort. So far I have referenced the one we are using for K8s gateway, but we might need to hardcode the endpoints and enrichment processors.

start-local.sh

rogercoll · 2025-07-30T09:42:33Z

start-local.sh

+
+      exporters:
+        elasticsearch:
+          endpoint: http://elasticsearch:9200


Both endpoint and endpoints configurations are supported in the Elasticsearch exporter. The first one is because the exporter embedding the upstream confighttp configuration: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/elasticsearchexporter/config.go#L69C13-L69C25

The latest was previously added because alignment with elasticsearch-go client configuration: https://github.com/elastic/go-elasticsearch/blob/main/elasticsearch.go#L71

Not a strong opinion, but maybe use endpoints as being the one used in our public docs: https://www.elastic.co/docs/reference/opentelemetry/edot-collector/config/default-config-standalone#data-export

rogercoll · 2025-07-30T10:05:50Z

start-local.sh

+      connectors:
+        elasticapm:
+
+      processors:


Could we add a couple of batch processor configurations? batch for logs and traces pipelines while batch/metrics for metrics pipelines, sample configuration: https://github.com/elastic/elastic-agent/blob/main/internal/pkg/otel/samples/linux/gateway.yml#L38-L44

(note that they need to be referenced in the pipeline's configuration too)

rogercoll · 2025-07-30T10:24:20Z

start-local.sh

+          logs_dynamic_index:
+            enabled: true
+          metrics_dynamic_index:
+            enabled: true
+          traces_dynamic_index:
+            enabled: true


This should be removed as being deprecated:

No-op. Documents are now always routed dynamically unless logs_index is not empty. Will be removed in a future version.

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/elasticsearchexporter/README.md#elasticsearch-document-routing

rogercoll · 2025-07-30T10:28:32Z

start-local.sh

  # Kibana settings container name
  kibana_settings_container_name="kibana-local-settings${ES_LOCAL_DIR:+-${ES_LOCAL_DIR}}"
+  # OTEL container name
+  otel_container_name="otel-collector${ES_LOCAL_DIR:+-${ES_LOCAL_DIR}}"


Wdyt of renaming the container name to elastic-otel-collector or edot-collector to differentiate from the upstream image?

Because we can have different services based on the version (--v option) and to avoid conflict we need to have different names.

Sorry, I should have worded that better. My suggestion would be to include the edot keyword on the container name prefix:

Suggested change

otel_container_name="otel-collector${ES_LOCAL_DIR:+-${ES_LOCAL_DIR}}"

otel_container_name="edot-collector${ES_LOCAL_DIR:+-${ES_LOCAL_DIR}}"

(EDOT = Elastic Distributions of OpenTelemetry)

rogercoll · 2025-07-30T10:45:24Z

start-local.sh

+      processors:
+        elastictrace:
+
+      exporters:


Could we add the debug exporter with the default configuration? That is really helpful to quickly check if data is being received by the collector, similar to https://github.com/elastic/elastic-agent/blob/main/internal/pkg/otel/samples/linux/gateway.yml#L48

Co-authored-by: Roger Coll <roger.coll@elastic.co>

ezimuel · 2025-09-12T10:13:02Z

@rogercoll and @mlunadia I updated the PR with the requested changes. Thanks.

rogercoll

LGTM! 🎉

rogercoll · 2025-09-12T11:04:54Z

start-local.sh

+          mapping:
+            mode: otel
+
+      service:


Suggested change

service:

service:

telemetry:

metrics:

readers:

- pull:

exporter:

prometheus:

host: '0.0.0.0'

port: 8888

One way to test the container is up and running would be exposing the collector's internal metrics through a Prometheus exporter interface. Then, we can assert the response code of curl http://edot-collector:8888/metrics

Another approach, that would not require exposing the collector's internal metrics, would be sending a testing payload to the already available http OTLP endpoint. Example:

$ curl -X POST http://localhost:4318/v1/logs \ -H "Content-Type: application/json" \ -d '{ "Timestamp": "1634630400000", "ObservedTimestamp": "1634630401000", "TraceId": "abcd1234", "SpanId": "efgh5678", "SeverityText": "DEBUG", "SeverityNumber": "5", "Body": "Testing log to assert collector OTLP endpoint", "Resource": { "service.name": "start-local-testing" }, "InstrumentationScope": {}, "Attributes": {} }' \ -o /dev/null -s -w "%{http_code}\n" 200

Thanks @rogercoll, I think we'll choose this approach since this does not require changing the configuration.

All suggestions have been applied

WIP: Add the OTEL collector service

9bcd20f

xrmx reviewed Jul 8, 2025

View reviewed changes

ezimuel and others added 2 commits July 8, 2025 16:40

Update start-local.sh

264b921

Co-authored-by: Riccardo Magliocchetti <riccardo.magliocchetti@gmail.com>

Update start-local.sh

7ff6274

Co-authored-by: Riccardo Magliocchetti <riccardo.magliocchetti@gmail.com>

xeraa reviewed Jul 9, 2025

View reviewed changes

mlunadia previously requested changes Jul 21, 2025

View reviewed changes

rogercoll reviewed Jul 30, 2025

View reviewed changes

ezimuel and others added 2 commits August 7, 2025 16:34

Update start-local.sh

4376d88

Co-authored-by: Roger Coll <roger.coll@elastic.co>

Update start-local.sh

8f96eb4

Co-authored-by: Roger Coll <roger.coll@elastic.co>

ezimuel changed the title ~~WIP: Add the OTEL collector service~~ WIP: Add the EDOT collector service Aug 22, 2025

ezimuel added 2 commits September 10, 2025 14:47

Updated the EDOT docker configuration

ac88aed

Merge remote-tracking branch 'origin/feature/otel' into feature/otel

474f21d

ezimuel changed the title ~~WIP: Add the EDOT collector service~~ Add the EDOT collector service Sep 10, 2025

ezimuel marked this pull request as ready for review September 10, 2025 12:52

rogercoll approved these changes Sep 12, 2025

View reviewed changes

ezimuel added 4 commits September 15, 2025 17:03

Added EDOT test

d3da2cf

Updated README about --edot option

26c43d1

Removed unused var in edot test

f3d433f

Fixed trailing space in README

7101aa6

ezimuel merged commit 83ef641 into main Sep 15, 2025
11 checks passed

ezimuel mentioned this pull request Sep 16, 2025

start EDOT collector #55

Closed

	endpoint: http://elasticsearch:9200
	endpoints: [ "http://elasticsearch:9200" ]

	otel_container_name="otel-collector${ES_LOCAL_DIR:+-${ES_LOCAL_DIR}}"
	otel_container_name="edot-collector${ES_LOCAL_DIR:+-${ES_LOCAL_DIR}}"

-      service:
+      service:
+        telemetry:
+          metrics:
+            readers:
+              - pull:
+                  exporter:
+                    prometheus:
+                      host: '0.0.0.0'
+                      port: 8888

Conversation

ezimuel commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xrmx Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SylvainJuge commented Jul 8, 2025

Uh oh!

ezimuel commented Jul 8, 2025 • edited by SylvainJuge Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mlunadia left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mlunadia Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ezimuel commented Sep 12, 2025

Uh oh!

rogercoll left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rogercoll Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

ezimuel commented Jul 8, 2025 •

edited

Loading

xrmx Jul 8, 2025 •

edited

Loading

ezimuel commented Jul 8, 2025 •

edited by SylvainJuge

Loading

mlunadia Aug 7, 2025 •

edited

Loading

rogercoll Sep 15, 2025 •

edited

Loading