Max Export Size for Metrics/Logs/Traces

[tsloughter]

What are you trying to achieve?

Reliable and consistent across languages exporting of metrics, logs and tracing over grpc.

I saw Python has added max_export_batch_size open-telemetry/opentelemetry-python#2809 to deal with the collector having a 4MB message size limit for grpc.

Since this is an issue across languages I thought it best to open an issue here, esp since a bit of googling and I see posts about increasing the size limit to 64MB (at last in Java), so I'm unsure what the expected limits going forward will be.

Before I would write up a PR I wanted to check how people thought it should be handled as well.

• Should max_export_batch_size be added to the spec for metrics?
  -Should this be a function of the exporter or reader? As in, the batch processor for traces has a size limit (maxExportBatchSize`) instead of being in the exporter -- I know these sizes (batch queue and proto message) are different but it should be smaller once serialized and compressed.
  • Needed for logs as well?
• Would be nice if the collector could tell the client what the size limit is?
• Partial success from the collector doesn't work here because the collector will be dropping the entire message, right?

[reyang]

Couple more things we need to think about:

1. What size limit do we need - transport layer (e.g. UDP MTU) or L7 (e.g. data before compression) or something else?
2. Do we need this across all targets (e.g. OTLP, Zipkin, Jaeger, whatever vendor specific exporters)?
3. What about other limits that are not directly tight to bytes (e.g. maximum number of points in a metric stream, maximum number of histogram buckets allowed in a single point, maximum number of attributes allowed, etc.)?

[AkselAllas]

I think increasing max export size makes sense at the metric exporter level.

[jmichalek132]

As discussed in this issue open-telemetry/opentelemetry-java#5394, the GRPC exporter can send bigger GRPC messages than what the limit is at the receiving end. In case you have control over the receiving end, the limit is something you could increase. But in case of sending directly to i.e. vendor you might not be able to.

So I think it would make sense to introduce an option that would limit the max size of grpc message and split the data into multiple messages in that case. Mentioning it here since this would require updating the spec.