Requesting TC review for JS metrics API/SDK spec compliance

[dyladan]

Requesting a compliance review from @open-telemetry/technical-committee prior to GA release. Documentation work is still underway. If there are questions you can ask myself or any of the other JS maintainers, especially @legendecas as he as done most of our metrics maintenance work or @pichlermarc since he has also done a lot of work on the metrics SDK.

[tigrannajaryan]

I added this to the next TC meeting agenda.

[reyang]

I can do it, @dyladan @legendecas would you provide a permanent link to the doc/code? Is this about the API and SDK only, or it covers some exporters (e.g. OTLP)?

[dyladan]

@reyang it covers API and SDK. Here is a full list of packages:

@opentelemetry/api-metrics

package readme: https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-api-metrics

reference: https://open-telemetry.github.io/opentelemetry-js/modules/_opentelemetry_api_metrics.html

@opentelemetry/exporter-metrics-otlp-grpc

proto/grpc exporter

package readme: https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-exporter-metrics-otlp-grpc

reference: https://open-telemetry.github.io/opentelemetry-js/modules/_opentelemetry_exporter_metrics_otlp_grpc.html

@opentelemetry/exporter-metrics-otlp-http

json over http exporter

package readme https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-exporter-metrics-otlp-http

reference: https://open-telemetry.github.io/opentelemetry-js/modules/_opentelemetry_exporter_metrics_otlp_http.html

@opentelemetry/exporter-metrics-otlp-proto

protobuf over http exporter

package readme https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-exporter-metrics-otlp-proto

reference: https://open-telemetry.github.io/opentelemetry-js/modules/_opentelemetry_exporter_metrics_otlp_proto.html

@opentelemetry/exporter-prometheus

package readme https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-exporter-prometheus

reference: https://open-telemetry.github.io/opentelemetry-js/modules/_opentelemetry_exporter_prometheus.html

@opentelemetry/sdk-metrics

This package is probably the most important SDK package as it contains all classes and types.

package readme https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-sdk-metrics

reference: https://open-telemetry.github.io/opentelemetry-js/modules/_opentelemetry_sdk_metrics.html

@opentelemetry/sdk-node

This package is used for both tracing and metrics

package readme https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-sdk-node

reference https://open-telemetry.github.io/opentelemetry-js/modules/_opentelemetry_sdk_node.html

[reyang]

The opentelemetry-js repo has issue templates which makes it harder to file issues found during the review, to make it easier, I'll post my findings in individual comments here.

[reyang]

API not mentioned in the spec but provided by the implementation:

• https://github.com/open-telemetry/opentelemetry-js/blob/d154066354863e8c682b4aa488d79c7e914bf91c/api/src/metrics/Meter.ts#L142
• https://github.com/open-telemetry/opentelemetry-js/blob/d154066354863e8c682b4aa488d79c7e914bf91c/api/src/metrics/Meter.ts#L156

[reyang]

API spec requires a state object to be passed to the callback, I couldn't find how it is supported in opentelemetry-js, e.g. https://github.com/open-telemetry/opentelemetry-js/blob/d154066354863e8c682b4aa488d79c7e914bf91c/api/src/metrics/Meter.ts#L110.

The API SHOULD provide some way to pass state to the callback.

[reyang]

Nit: not part of the spec compliance review but I noticed this comment:

https://github.com/open-telemetry/opentelemetry-js/blob/d154066354863e8c682b4aa488d79c7e914bf91c/api/src/metrics/Metric.ts#L67

Do we want Inputs may not be negative. or Inputs must not be negative.? (if we look at histogram, it is using the must wording https://github.com/open-telemetry/opentelemetry-js/blob/d154066354863e8c682b4aa488d79c7e914bf91c/api/src/metrics/Metric.ts#L81)

[reyang]

Minor: since 1.13.0, the API spec allows optional attributes in the GetMeter API:

https://github.com/open-telemetry/opentelemetry-js/blob/d154066354863e8c682b4aa488d79c7e914bf91c/api/src/metrics/MeterProvider.ts#L32

[reyang]

Exemplar in the SDK spec is not yet Stable, need to be removed or hidden if the intention is to ship the initial stable JS package https://github.com/open-telemetry/opentelemetry-js/tree/d154066354863e8c682b4aa488d79c7e914bf91c/experimental/packages/opentelemetry-sdk-metrics/src/exemplar.

[reyang]

MetricReader doesn't have ForceFlush according to the SDK spec.

https://github.com/open-telemetry/opentelemetry-js/blob/d154066354863e8c682b4aa488d79c7e914bf91c/experimental/packages/opentelemetry-sdk-metrics/src/export/MetricReader.ts#L171

[reyang]

I haven't looked deeper so I could be wrong, it seems the ConsoleExporter is only supporting CUMULATIVE? https://github.com/open-telemetry/opentelemetry-js/blob/d154066354863e8c682b4aa488d79c7e914bf91c/experimental/packages/opentelemetry-sdk-metrics/src/export/ConsoleMetricExporter.ts#L41

The specification requires both CUMULATIVE and DELTA to be supported https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk_exporters/stdout.md#opentelemetry-metrics-exporter---standard-output.

InMemoryExporter seems to be covering both https://github.com/open-telemetry/opentelemetry-js/blob/d154066354863e8c682b4aa488d79c7e914bf91c/experimental/packages/opentelemetry-sdk-metrics/src/export/InMemoryMetricExporter.ts#L68.

[reyang]

So far I've finished reviewing the following parts:

• @opentelemetry/api-metrics
• @opentelemetry/sdk-metrics
• @opentelemetry/sdk-node

I'll cover these parts next week:

• @opentelemetry/exporter-metrics-otlp-grpc
• @opentelemetry/exporter-metrics-otlp-http
• @opentelemetry/exporter-metrics-otlp-proto

I don't plan to cover these parts since the spec is not yet marked as Stable (if a dry run would be helpful, please let me know and I can do it):

• @opentelemetry/exporter-prometheus
• Exemplar in @opentelemetry/sdk-metrics
• Exponential Bucket Histogram in @opentelemetry/sdk-metrics

[legendecas]

API not mentioned in the spec but provided by the implementation:

• https://github.com/open-telemetry/opentelemetry-js/blob/d154066354863e8c682b4aa488d79c7e914bf91c/api/src/metrics/Meter.ts#L142
• https://github.com/open-telemetry/opentelemetry-js/blob/d154066354863e8c682b4aa488d79c7e914bf91c/api/src/metrics/Meter.ts#L156

This is defined in the spec, with an implementation specialized name: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/api.md#multiple-instrument-callbacks. Also, the spec requires the registered callbacks to be able to be unregistered: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/api.md#asynchronous-counter-operations.

[legendecas]

API spec requires a state object to be passed to the callback, I couldn't find how it is supported in opentelemetry-js, e.g. https://github.com/open-telemetry/opentelemetry-js/blob/d154066354863e8c682b4aa488d79c7e914bf91c/api/src/metrics/Meter.ts#L110.

The API SHOULD provide some way to pass state to the callback.

It is quite easy for JavaScript to capture variable in lambdas, so by the statement in the spec:

The API SHOULD provide some way to pass state to the callback. OpenTelemetry API authors MAY decide what is the idiomatic approach (e.g. it could be an additional parameter to the callback function, or captured by the lambda closure, or something else).

I believe the JavaScript API is sufficient to cover the requirements.

[legendecas]

Exemplar in the SDK spec is not yet Stable, need to be removed or hidden if the intention is to ship the initial stable JS package https://github.com/open-telemetry/opentelemetry-js/tree/d154066354863e8c682b4aa488d79c7e914bf91c/experimental/packages/opentelemetry-sdk-metrics/src/exemplar.

Exemplar is not exposed in the SDK as a public interface yet: https://github.com/open-telemetry/opentelemetry-js/blob/main/experimental/packages/opentelemetry-sdk-metrics/src/index.ts.

[pichlermarc]

MetricReader doesn't have ForceFlush according to the SDK spec.

open-telemetry/opentelemetry-js@d154066/experimental/packages/opentelemetry-sdk-metrics/src/export/MetricReader.ts#L171

Hmm, correct. 🤔
However, it is referenced in the spec for the MeterProvider.

ForceFlush MUST invoke ForceFlush on all registered MetricReader and Push Metric Exporter instances.

Both the Python and Java SDK implementations seem to have solved it in the same way. May be worth opening a spec issue over if one does not yet exist. 🤔

[dyladan]

I can create issues without the template so i'll move these into the JS repo

[dyladan]

Created issues so discussions can be moved to those. Thanks @reyang