Conflicting requirements in OTLP JSON field names

[tigrannajaryan]

Problem

JSON Protobuf Encoding section in OTLP protocol specification says:

The trace_id and span_id byte arrays are represented as ...

Note that trace_id and span_id fields use lower snake case, with underscores, using original field names from the proto.

However, in the same section of the spec we also have this:

The keys of JSON objects are field names converted to lowerCamelCase. Original field names are not valid to use as keys for JSON objects. For example, this is a valid JSON representation of a Resource: { "attributes": {...}, "droppedAttributesCount": 123 }, and this is NOT a valid representation: { "attributes": {...}, "dropped_attributes_count": 123 }.

These 2 clauses conflict with each other. Which one is true? Should we use lower camel case, i.e. traceId/spanId or the original names trace_id/span_id?

We ended up in this broken situation after this PR, where we decided that lowerCamelCase names only must be used.

The spec contradicts with itself and it needs fixing.

Chosen Solution

• We recognize this a bug that needs to be fixed.
• We choose camelCase as canonical JSON field casing. Use correct lowerCameCase for JSON field references #468 fixed it in the spec.
• We make sure Collector's OTLP receiver accepts both capitalizations for one year to give everyone time to transition.
• All SDK exporters and Collector's OTLP exporter will be immediately updated to use the camelCase (if not already).

Alternate solutions considered

Allow trace_id/span_id names as an exception

We can explicitly call out in the spec that the names of these 2 fields are an exception from the rule about lowerCamelCase. We will end up with a bit of ugliness and inconsistency in the OTLP JSON spec.

If anyone can think of a better solution please comment.

[tigrannajaryan]

@open-telemetry/specs-approvers FYI.

[jack-berg]

The trace_id and span_id byte arrays are represented as case-insensitive hex-encoded strings; they are not base64-encoded as is defined in the standard Protobuf JSON Mapping. Hex encoding is used for trace_id and span_id fields in all OTLP Protobuf messages, e.g., the Span, Link, LogRecord, etc. messages. For example, the trace_id field in a Span can be represented like this: { "trace_id": "5B8EFFF798038103D269B633813FC60C", ... }

The reference to snake case here seems like a typo. This clause is referencing the encoding of the strings, and is not concerns with the casing of keys. I believe the snake case was probably in reference to the field names in the proto definition.

I'm in favor of the first option, but would like to understand more about which components would be disrupted by this.

[tigrannajaryan]

I'm in favor of the first option, but would like to understand more about which components would be disrupted by this.

Any existing sender implementation of OTLP JSON which is compliant with the current spec will be no longer able to send spans to any other receiver implementation of OTLP JSON that is compliant with the modified Otel spec (and vice versa).

This is a catastrophic failure mode for me, the worst kind of a failure to deliver on our stability promise since the only way to avoid being affected by it is co-ordinated updating of all participants (producers and consumers) or a distributed network, which in any non-trivial deployment is either hard or impossible to do.

[jack-berg]

Any existing sender implementation of OTLP JSON which is compliant with the current spec will be no longer able to send spans to any other receiver implementation of OTLP JSON that is compliant with the modified Otel spec (and vice versa).

Doesn't that assume that a sender producing trace_id / span_id is in fact compliant? As you point out, the language is contradictory. You can make the case that the later bullet point means that sending trace_id / span_id instead of traceId / spanId means the sender is not compliant.

[tigrannajaryan]

You are right. It is a matter of which of the contradicting clauses we think was the overriding one and I don't think there is a clear answer to that.

I updated the description of the issue to reflect the fact that depending on what we think was the overriding clause one or the other proposed solution is breaking.

[joaopgrassi]

Does the merge of #468 changes anything in this issue?

[punya]

Note that the example JSON files in the repository actually use both types of casing in the same file!

• Snake case: resource_spans, scope_spans
• Camel case: stringValue, parentSpanId

[tigrannajaryan]

I added this to 1.0 pre-release checklist. We need to resolve this.

[tigrannajaryan]

Does the merge of #468 changes anything in this issue?

Yes, it went with camelCase. But I think there is a bit more work to do before we close this.

[tigrannajaryan]

Note that the example JSON files in the repository actually use both types of casing in the same file!

• Snake case: resource_spans, scope_spans
• Camel case: stringValue, parentSpanId

Those are different fields, but I agree they need to be fixed to camelCase too.

[tigrannajaryan]

@open-telemetry/specs-approvers

• We recognize this a bug that needs to be fixed.
• We choose camelCase as canonical JSON field casing. Use correct lowerCameCase for JSON field references #468 fixed it in the spec.
• We make sure Collector's OTLP receiver accepts both capitalizations for one year to give everyone time to transition.
• All SDK exporters and Collector's OTLP exporter will be immediately updated to use the camelCase (if not already).

Thoughts?

[tigrannajaryan]

• We make sure Collector's OTLP receiver accepts both capitalizations for one year to give everyone time to transition.

This is already the case:
https://github.com/open-telemetry/opentelemetry-collector/blob/e79db959c31a7356a676bccd596f2f66068807bf/pdata/ptrace/json.go#L166
https://github.com/open-telemetry/opentelemetry-collector/blob/e79db959c31a7356a676bccd596f2f66068807bf/pdata/plog/json.go#L118
https://github.com/open-telemetry/opentelemetry-collector/blob/e79db959c31a7356a676bccd596f2f66068807bf/pdata/pmetric/json.go#L414

[tigrannajaryan]

Also related to this is "Trace Context in non-OTLP Log Formats" document: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/logging_trace_context.md

Should we update it from snake_case to camelCase too?

@open-telemetry/specs-logs-approvers what do you think?

[UPDATE] I created a separate issue for this, let's discuss there: open-telemetry/opentelemetry-specification#3518

[tigrannajaryan]

I opened issues in every language repo to ensure OTLP/JSON exporters use correct capitalization. Closing this issue, there is nothing else to do.