OpenSearch Exporter #23045

MaxKsyunz · 2023-06-02T18:29:18Z

Adding collector exporter for OpenSearch. Currently only sends traces to OpenSearch. Supports simple schema for observability.

Here's a demo of it all working together using examples/demo app.

OpenSearch.Otel.Exporter.mp4

References

Resolves #7905.

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

linux-foundation-easycla · 2023-06-05T23:24:15Z

The committers listed above are authorized under a signed CLA.

✅ login: MaxKsyunz / name: Max Ksyunz (dc8b519, 73e7fde, a4d6945, 213893d, 972f8b2, 2b0848b, d9be8eb, e757a70, a24c245, 4d90f42, a0e4c11, 84134ef, 96e322d, 38da590, 1977ab6, 37269f9, 9047dd6, 618a396)
✅ login: MitchellGale / name: Mitchell Gale (1cdb910, e7fbb0e)

MitchellGale · 2023-06-09T18:08:05Z

exporter/opensearchexporter/model.go

+// encodeModel tries to keep the event as close to the original open telemetry semantics as is.
+// No fields will be mapped by default.
+//
+// Field deduplication and dedotting of attributes is supported by the encodeModel.


What is dedotting?

Have a look at exporter/opensearchexporter/internal/objmodel/objmodel.go. This explains a lot about it. In short, this is something, that is required for OpenSearch due to its interpretation of dots in field names (keys).

KarstenSchnitter

Thanks for this great contribution. It seems you have implemented traces and logs. Can you adjust the README.md to reflect this?

I am concerned about the compatibility between the data generated with this exporter when compared with using DataPrepper for the same job. There is real nice support for traces in OpenSearch Dashboards with the observability plugin, which builds on the data model provided by DataPrepper. Will this be useable with the OpenTelemetry OpenSearch exporter as well?

A particular feature of the DataPrepper approach we like, is the common handling of attributes and resource attributes, which can be shared between traces and logs. Having the name schema allows to create dashboards and visualisations in OpenSearch feeding from different indices for logs, traces, and metrics utilising common filters. This should also be able with the exported of this PR. As I understand it, the different naming schemes will render indices created by the OpenTelemetry Collector and DataPrepper incompatible with regards to filtering, e.g. by trace id or service name. Can this be improved?

KarstenSchnitter · 2023-06-11T09:19:10Z

exporter/opensearchexporter/README.md

+- `traces_index`: The
+  [index](https://opensearch.org/docs/latest/opensearch/index-data/)
+  or [datastream](https://opensearch.org/docs/latest/opensearch/data-streams/)
+  name to publish trace events to. The default value is `traces-generic-default`. Note: To better differentiate between log indexes and traces indexes, `index` option are deprecated and replaced with below `logs_index`


Why does this not point to otel-v1-apm-span-000001 or similar, which would be used by DataPrepper to provide data for the OpenSearch Observability Trace plugin? Is there some kind of incompatibility between the data?

data-prepper is currently also undergoing the movement to data-stream so that both ingestion concepts would use the same mechanism

In addition the catalog repository recently added enables the consolidation and sharing of knowledge and physical mapping for all the plugins across the org

KarstenSchnitter · 2023-06-11T09:22:22Z

exporter/opensearchexporter/README.md

+| Status                   |               |
+| ------------------------ |---------------|
+| Stability                | [development] |
+| Supported pipeline types | traces        |


The example below contains a configuration for logs besides the traces pipeline. How does this relate to this field, which states, that only traces is supported?

KarstenSchnitter · 2023-06-11T09:23:28Z

exporter/opensearchexporter/README.md

+- `read_buffer_size` (default=0): Read buffer size.
+- `write_buffer_size` (default=0): Write buffer size used when.
+- `timeout` (default=90s): HTTP request time limit.
+- `headers` (optional): Headers to be send with each HTTP request.


Does the exported use compression, e.g. gzip, by default? Can this be added as a setting?

KarstenSchnitter · 2023-06-11T09:25:43Z

exporter/opensearchexporter/config.go

+
+	// Headers allows users to configure optional HTTP headers that
+	// will be send with each HTTP request.
+	Headers map[string]string `mapstructure:"headers,omitempty"`


Wouldn't it be better to use a secure string for the header values, as they could contain sensitive information similar to the password of the AuthenticationSettings?

KarstenSchnitter · 2023-06-11T09:30:02Z

exporter/opensearchexporter/README.md

+- `mapping`: Events are encoded to JSON. The `mapping` allows users to
+  configure additional mapping rules.


Config validation looks for particular values none, no, and sso. Can you add documentation for these values, so that it is clear, what they mean?

KarstenSchnitter · 2023-06-11T10:01:29Z

exporter/opensearchexporter/internal/objmodel/objmodel_test.go

+			},
+			want: Document{[]field{{"a", IntValue(1)}, {"z", IntValue(26)}}},
+		},
+		"sorting is stable": {


Is this a required quality, as long as the sorting provides the same result given the same input?

KarstenSchnitter · 2023-06-11T10:08:35Z

exporter/opensearchexporter/model.go

+// encodeModel tries to keep the event as close to the original open telemetry semantics as is.
+// No fields will be mapped by default.
+//
+// Field deduplication and dedotting of attributes is supported by the encodeModel.


Have a look at exporter/opensearchexporter/internal/objmodel/objmodel.go. This explains a lot about it. In short, this is something, that is required for OpenSearch due to its interpretation of dots in field names (keys).

KarstenSchnitter · 2023-06-11T10:12:49Z

exporter/opensearchexporter/model.go

+	document.AddTimestamp("@timestamp", span.StartTimestamp()) // We use @timestamp in order to ensure that we can index if the default data stream logs template is used.
+	document.AddTimestamp("EndTimestamp", span.EndTimestamp())
+	document.AddTraceID("TraceId", span.TraceID())
+	document.AddSpanID("SpanId", span.SpanID())
+	document.AddSpanID("ParentSpanId", span.ParentSpanID())
+	document.AddString("Name", span.Name())
+	document.AddString("Kind", traceutil.SpanKindStr(span.Kind()))
+	document.AddInt("TraceStatus", int64(span.Status().Code()))
+	document.AddString("Link", spanLinksToString(span.Links()))
+	document.AddAttributes("Attributes", span.Attributes())
+	document.AddAttributes("Resource", resource.Attributes())


This looks very different to the structure DataPrepper would generate. This is important, since the OpenSearch Observability Trace Analytics build on that data model. Here is an example from an OpenSearch index:

"_source": { "kind": "SPAN_KIND_INTERNAL", "traceId": "eaa69f00b4ab63fd618eadffb00e60d7", "spanId": "75d7d735cb2b4972", "parentSpanId": "84f4ad4086932693", "traceState": "", "traceGroup": null, "serviceName": "sample", "name": "LogController.generateLog", "startTime": "2023-06-09T14:32:34.444486691Z", "endTime": "2023-06-09T14:32:34.446411315Z", "durationInNanos": 1924624, "status.code": 0, "events": [], "droppedEventsCount": 0, "traceGroupFields": { "endTime": null, "durationInNanos": null, "statusCode": null }, "links": [], "droppedLinksCount": 0, "instrumentationScope.version": "1.25.0-alpha", "instrumentationScope.name": "io.opentelemetry.spring-webmvc-3.1", "resource.attributes.process@pid": 7, "resource.attributes.host@arch": "amd64", "resource.attributes.telemetry@sdk@version": "1.25.0", "resource.attributes.service@name": "sample", "resource.attributes.process@runtime@name": "Java(TM) SE Runtime Environment", "resource.attributes.os@type": "linux", "resource.attributes.telemetry@sdk@language": "java", "resource.attributes.host@name": "18232480-822d-40e2-5ded-ebb2", "resource.attributes.process@runtime@version": "8.1.094", "resource.attributes.telemetry@sdk@name": "opentelemetry", "resource.attributes.telemetry@auto@version": "1.25.0", "resource.attributes.os@description": "Linux 5.19.0-41-generic", "span.attributes.thread@id": 36, "span.attributes.thread@name": "http-nio-8080-exec-8", "droppedAttributesCount": 0 }

How does the two naming schemes align in OpenSearch. Is there already support for both?

@KarstenSchnitter yes this was based on the elastic exporter. I will be changing the schema shortly.

@KarstenSchnitter could you please open a DataPrepper issue to make any necessary changes?

@kkondaka what would be the concrete changes required in DataPrepper? It is not clear to me, what naming would be preferable. Some are just capitalization, some are specific to DataPrepper (extra serviceName and traceGroup) and others dilute understanding of the original OTel fields like @timestamp and TraceStatus (to a lesser extend). Attributes is in plural, where Link and Resource are not.

I would prefer to not change the names in DataPrepper, since we use it in production. If this is necessary for alignment between the two tools, so be it.

@KarstenSchnitter we definitely do not want to make any changes that would break any existing deployment. We may want to add new fields and slowly deprecate old ones

KarstenSchnitter · 2023-06-11T10:21:43Z

exporter/opensearchexporter/opensearch_bulk.go

+
+		case resp.Status == 0 && err != nil:
+			// Encoding error. We didn't even attempt to send the event
+			logger.Error("Drop docs: failed to add docs to the bulk request buffer.",


I am not sure on the severity of dropped docs. Since the collector remains running and connecting to OpenSearch, I would see this with a lower severity. My thinking is, that dropping some of the docs is less severe as dropping all of them due to misconfiguration or network partition. Especially indexing errors (further down) can occur due to conflicts within a particular document.

Does this instrumentation expose metrics on the number of dropped docs? This would really help observability of the Collector.

KarstenSchnitter · 2023-06-11T10:24:43Z

exporter/opensearchexporter/trace_exporter.go

+	return multierr.Combine(errs...)
+}
+
+func (e *opensearchTracesExporter) pushTraceRecord(ctx context.Context, resource pcommon.Resource, span ptrace.Span) error {


Do you plan adding support for service maps as well?

Service map is not planned currently in this exporter - we are thinking of using the existing OTEL contribute service graph processor and consolidate that functionality with data prepper

jpkrohling · 2023-06-12T18:15:48Z

Note that this component has NOT been accepted yet. There are no sponsors for this component. Until you find an sponsor who can commit to review and co-own this component, this is not going to be accepted.

#7905 (comment)

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

…porter Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> # Conflicts: # cmd/otelcontribcol/builder-config.yaml # cmd/otelcontribcol/go.mod # cmd/otelcontribcol/go.sum # cmd/oteltestbedcol/builder-config.yaml # cmd/oteltestbedcol/go.mod # cmd/oteltestbedcol/go.sum

MaxKsyunz · 2023-06-13T07:33:08Z

Note that this component has NOT been accepted yet. There are no sponsors for this component. Until you find an sponsor who can commit to review and co-own this component, this is not going to be accepted.

#7905 (comment)

Yes, very true.

How can I get in touch with maintainer group to begin looking for a sponsor?

MaxKsyunz · 2023-06-13T09:23:39Z

Thanks for this great contribution. It seems you have implemented traces and logs. Can you adjust the README.md to reflect this?

I am concerned about the compatibility between the data generated with this exporter when compared with using DataPrepper for the same job. There is real nice support for traces in OpenSearch Dashboards with the observability plugin, which builds on the data model provided by DataPrepper. Will this be useable with the OpenTelemetry OpenSearch exporter as well?

A particular feature of the DataPrepper approach we like, is the common handling of attributes and resource attributes, which can be shared between traces and logs. Having the name schema allows to create dashboards and visualisations in OpenSearch feeding from different indices for logs, traces, and metrics utilising common filters. This should also be able with the exported of this PR. As I understand it, the different naming schemes will render indices created by the OpenTelemetry Collector and DataPrepper incompatible with regards to filtering, e.g. by trace id or service name. Can this be improved?

Thank you for a thorough review @KarstenSchnitter.

Yes, this schema is odd -- I'll replace it shortly. FWIW I started with it because that's what elasticsearchexporter emits.

The end-goal of this PR is to export otel signals to OpenSearch in a schema compatible with OpenTelemetry standard. The specific schema is here.

This is part of a larger effort to improve OpenSearch Observability plugin. See here for details.

derek-ho · 2023-06-13T14:25:39Z

exporter/opensearchexporter/config.go

+	switch candidate {
+	case "none":
+	case "no":
+	case "sso":


We changed the naming convention from SSO to SS4O to avoid confusion with single sign on, can we change all the references to SS4O?

Yep, coming up.

@MaxKsyunz plz update to the current location of the Observability mapping schema

…porter # Conflicts: # cmd/otelcontribcol/go.mod # cmd/otelcontribcol/go.sum

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

YANG-DB · 2023-06-17T04:09:51Z

@KarstenSchnitter has raised many important concerns regarding compatibility between simple schema and data-prepper traces.

Recently @kkondaka and myself worked together for data-prepper & SimpleSchema to share the same structure for Metrics (this has already been released in the latest data-prepper version).
We are very much in favour of following this process for traces as well, since traces was completed prior to simple schema arriving there are few (minor) differences that we will consolidate in the next versions.

Regarding logs - the recent announcement by OTEL and ECS for converging both protocols into one is a large step forward that we at OpenSearch would very much like to adopt. This is why we are currently postponing the Logs support in the exporter until OpenTelemetry will release the first stable version we can reference and follow.

This will be probably true for data-prepper as well and therefore I think we need to continue our current work on traces and wait for the consolidated logs protocol to be published.

Regarding Observability-Dashboards and trace analytics dialog - we are adding support for custom selection of trace index source so that the user can select the source of the trace data.

Since there is already a large overlap between data-prepper and simple schema - they both would work similarly

jpkrohling · 2023-06-27T18:56:47Z

This is why we are currently postponing the Logs support in the exporter until OpenTelemetry will release the first stable version we can reference and follow.

Isn't the ECS merger only about semantic conventions? I think that work has completed already.

github-actions · 2023-07-12T05:20:59Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

MaxKsyunz · 2023-07-12T06:19:00Z

Continued in separate PRs as per contributor guidelines.

Adding initial set-up for OpenSearch exporter addition. Future PRs will include adding functionality to the exporter, code coverage and e2e tests. Broken up for easier review. [New component proposal.](#23611) Start of resolution of #7905. Will come in future PR to 80+% coverage. Break-up of #23045 --------- Signed-off-by: Max Ksyunz <max.ksyunz@improving.com>

github-actions bot added cmd/configschema configschema command cmd/otelcontribcol otelcontribcol command cmd/oteltestbedcol labels Jun 2, 2023

github-actions bot requested review from dmitryax, mx-psi and pmcollins June 2, 2023 18:29

atoulme added the Sponsor Needed New component seeking sponsor label Jun 2, 2023

MaxKsyunz and others added 5 commits June 2, 2023 13:51

OpenTelemetry to OpenSearch Exporter (#1)

dc8b519

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

Minor code cleanup

73e7fde

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

Update dependencies

a4d6945

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

Update to match current build checks

213893d

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

Add opensearchexporter to otelcontribcol and oteltestbedcol

972f8b2

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

MaxKsyunz force-pushed the integ/opensearch-exporter branch from f8a1475 to 972f8b2 Compare June 3, 2023 09:15

Add missing replace go.mod directives

2b0848b

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

github-actions bot added the testbed label Jun 3, 2023

MaxKsyunz added 2 commits June 3, 2023 02:57

make crosslink

d9be8eb

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

update dependencies and a unit test

e757a70

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

MaxKsyunz force-pushed the integ/opensearch-exporter branch from 10e4b87 to df4a1e8 Compare June 5, 2023 22:08

Removing opensearchexporter registration

a24c245

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

MaxKsyunz force-pushed the integ/opensearch-exporter branch from df4a1e8 to a24c245 Compare June 5, 2023 22:12

Add a codeowner for opensearchexporter

4d90f42

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

MaxKsyunz force-pushed the integ/opensearch-exporter branch from 63be6d9 to 4d90f42 Compare June 5, 2023 22:18

MaxKsyunz changed the title ~~[WIP] OpenSearch Exporter~~ OpenSearch Exporter Jun 8, 2023

MitchellGale reviewed Jun 9, 2023

View reviewed changes

MitchellGale approved these changes Jun 9, 2023

View reviewed changes

KarstenSchnitter reviewed Jun 11, 2023

View reviewed changes

MitchellGale added 2 commits June 12, 2023 18:59

Adding Opensearch change log for SSO support addding to OS exporter

1cdb910

Removed comments and shortened note in opensearch-exporter.yaml

e7fbb0e

MaxKsyunz added 3 commits June 12, 2023 18:59

Set appropriate stability status

a0e4c11

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

Register opensearchexporter component (#12)

84134ef

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

MaxKsyunz force-pushed the integ/opensearch-exporter branch from fc4e8d2 to 96e322d Compare June 13, 2023 08:38

derek-ho reviewed Jun 13, 2023

View reviewed changes

MaxKsyunz added 5 commits June 13, 2023 10:50

Merge remote-tracking branch 'upstream/main' into integ/opensearch-ex…

38da590

…porter # Conflicts: # cmd/otelcontribcol/go.mod # cmd/otelcontribcol/go.sum

Rerun genotelcontribcol and oteltestbedcol

1977ab6

Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>

Export traces in SSO schema

37269f9

Improve error message

9047dd6

Updated README.md

618a396

YANG-DB mentioned this pull request Jun 16, 2023

Add OpenSearch OTEL exporter opensearch-project/opentelemetry-demo#30

Closed

MitchellGale mentioned this pull request Jun 28, 2023

[Exporter/OpenSearch] OpenSearch exporter setup #23819

Merged

github-actions bot added the Stale label Jul 12, 2023

MaxKsyunz closed this Jul 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenSearch Exporter #23045

OpenSearch Exporter #23045

MaxKsyunz commented Jun 2, 2023 •

edited

linux-foundation-easycla bot commented Jun 5, 2023 •

edited

MitchellGale Jun 9, 2023

KarstenSchnitter Jun 11, 2023

KarstenSchnitter left a comment

KarstenSchnitter Jun 11, 2023

YANG-DB Jun 13, 2023

KarstenSchnitter Jun 11, 2023

KarstenSchnitter Jun 11, 2023

KarstenSchnitter Jun 11, 2023

KarstenSchnitter Jun 11, 2023

KarstenSchnitter Jun 11, 2023

KarstenSchnitter Jun 11, 2023

KarstenSchnitter Jun 11, 2023

MaxKsyunz Jun 13, 2023

kkondaka Jun 15, 2023

KarstenSchnitter Jun 19, 2023

kkondaka Jun 19, 2023

KarstenSchnitter Jun 11, 2023

KarstenSchnitter Jun 11, 2023

KarstenSchnitter Jun 11, 2023

YANG-DB Jun 13, 2023

jpkrohling commented Jun 12, 2023

MaxKsyunz commented Jun 13, 2023

MaxKsyunz commented Jun 13, 2023

derek-ho Jun 13, 2023

MaxKsyunz Jun 13, 2023

YANG-DB Jun 13, 2023 •

edited

YANG-DB commented Jun 17, 2023 •

edited

jpkrohling commented Jun 27, 2023

github-actions bot commented Jul 12, 2023

MaxKsyunz commented Jul 12, 2023

		- `mapping`: Events are encoded to JSON. The `mapping` allows users to
		configure additional mapping rules.

OpenSearch Exporter #23045

OpenSearch Exporter #23045

Conversation

MaxKsyunz commented Jun 2, 2023 • edited

References

linux-foundation-easycla bot commented Jun 5, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KarstenSchnitter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpkrohling commented Jun 12, 2023

MaxKsyunz commented Jun 13, 2023

MaxKsyunz commented Jun 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YANG-DB Jun 13, 2023 • edited

Choose a reason for hiding this comment

YANG-DB commented Jun 17, 2023 • edited

jpkrohling commented Jun 27, 2023

github-actions bot commented Jul 12, 2023

MaxKsyunz commented Jul 12, 2023

MaxKsyunz commented Jun 2, 2023 •

edited

linux-foundation-easycla bot commented Jun 5, 2023 •

edited

YANG-DB Jun 13, 2023 •

edited

YANG-DB commented Jun 17, 2023 •

edited