2.0.0

Breaking

• instrument now declares hackney as an application dependency. Start
  it with application:ensure_all_started(instrument) (releases do this
  already); application:start(instrument) alone returns not_started.
• Labeled meter metrics now export under the instrument's own name, with the
  attributes as labels. A counter requests written with method and
  status used to export as requests_method_status_total; it now exports as
  requests_total{method="...",status="..."}. Dashboards and alerts that use
  the old names must be updated. See the 2.0 migration guide.
• Meter instruments no longer export a zero value before their first write.
  Prometheus-style metrics created with the simple API still appear as soon as
  they are created.
• The cardinality limit now applies per metric. Before, each combination of
  label keys under one instrument had its own limit, so one instrument could
  hold several times the configured number of series.
• instrument_registry:lookup/1 returns metric metadata instead of the
  internal metric record.
• Observable callbacks must not call the Prometheus formatter or the OTLP
  exporter: both now run the callbacks themselves, so doing so recurses.
• The flight recorder is now disabled by default. Set
  flight_recorder_auto_enable to true or call
  instrument_flight_recorder:enable/0 to turn it on.

Added

• One callback can now feed several observable instruments. When several
  metrics come from the same expensive source, such as one RPC returning all
  queue statistics, the source is read once per collection instead of once
  per instrument. Observable instruments can also be created with options
  (description, unit) and no callback of their own, to be fed this way.

Changed

• Require hackney 4.10 or newer (~> 4.10). Releases before 4.9.0 are
  affected by security advisories, which the previous ~> 4.4 range still
  allowed.
• Creating an instrument or writing a new label set no longer goes through the
  registry process and no longer triggers a garbage collection pass across the
  whole VM. Bursts of new label values used to queue every writer behind the
  registry and could bring the node down; they now complete quickly.
• Writes with labels are more than twice as fast (about 120 ns instead of
  325 ns for two attributes). Meter writes without labels take about 21 ns
  instead of 15 ns. See the benchmarks guide.
• The Prometheus formatter runs observable callbacks before each scrape, so
  Prometheus-only setups no longer serve stale observable values.
• OTLP labeled counters and histograms now carry their start time.
• Upgrading in place from 1.1.x cleans up the old internal state on first
  start.
• Setting OTEL_TRACES_EXPORTER (or the span_exporter application setting)
  now sends spans through the batch span processor, as other OpenTelemetry
  SDKs do. It used to go through the older exporter manager, which exported
  inside its own process and queued without limit while a collector was slow
  or down. OTEL_BSP_* settings apply without having to be set explicitly.
• The span exporter manager is only called by the tracer while it has an
  exporter registered, which makes span creation cheaper when it is idle and
  avoids exporting spans twice when a processor is also configured.

Fixed

• OTLP export works without starting hackney by hand: it was not listed as
  a dependency, so in a release it could be missing or not started and every
  export failed silently.
• OTLP collector responses are read up to 64 KB. Without a limit a broken or
  malicious collector could make the node hold up to 512 MB per request.
• Oversized traceparent, tracestate, baggage and b3 headers are
  ignored or cut at their limits before being parsed, instead of being split
  in full first.
• The text log format escapes control characters in messages, so a message
  carrying request data cannot fake additional log lines.
• Creating new label sets no longer slows down as a node accumulates series.
  Each new series used to add two persistent_term keys, and every key
  added copies the VM's whole key table; cells are now created ahead of time
  in chunks, so a metric uses a handful of keys whatever its cardinality.
• Removing label sets (remove_label, clear_labels) frees them for reuse,
  so label churn no longer grows memory and scrape time without bound.
• Removing and re-creating a metric while it is being written no longer
  sends later writes to a cell that is never exported.
• A flight recorder message nested deeply enough (a long left-nested iolist)
  could crash the whole VM when the payload limit was 64 KB or more; and
  tracing a process that registers a name with procs hung the node. Both
  are fixed, and the payload limit is capped at 1 MB.
• Prometheus output is always valid: dotted OpenTelemetry names and unusual
  label names are converted to the allowed characters (one such name used to
  make Prometheus reject the whole scrape, and a crafted label name could
  add lines), and one metric that cannot be rendered is skipped instead of
  failing the scrape.
• Label values of any type (IP address tuples, pids, non-Latin-1 strings)
  no longer crash the code writing the metric.
• Vector label values that print the same (floats differing past the sixth
  decimal, strings versus binaries) no longer grow memory without bound, and
  are removed with their series.
• Calling new_counter/2 and the like repeatedly for an existing metric no
  longer makes every scrape slower.
• Invalid histogram boundaries are rejected when the histogram is created,
  instead of failing every recording; non-numbers are rejected before any
  count changes; a gauge that overflows now saturates instead of becoming
  unreadable.
• One log call with non-Latin-1 text, a bad format, or a malformed trace id
  in its metadata no longer removes the log export handler for the rest of
  the node's life.
• Header extraction skips unreadable headers instead of crashing the request
  handler; an invalid tracestate no longer discards a valid traceparent;
  trace flags other than 00 and 01 are propagated instead of dropped.
• One invalid UTF-8 string in a span, log or metric no longer drops the whole
  OTLP or JSON export batch; boolean attributes export as booleans.
• Non-Latin-1 OTEL_SERVICE_NAME, OTEL_SERVICE_VERSION and resource
  attributes no longer crash startup or drop all resource attributes;
  OTEL_RESOURCE_ATTRIBUTES values are percent-decoded.
• Flight tracing leaves processes traced by another tool (dbg, recon_trace)
  untouched instead of switching their tracing off.
• The simple and tail-sampling processors drop spans once 2048 exports are
  waiting, and skip requests whose caller already gave up.
• Exporter configuration, including authorization headers, no longer appears
  in crash reports.
• Queued spans and logs are exported when the application stops, and export
  workers stop with it.
• Exception details recorded on spans are printed with a depth limit.
• OTEL_EXPORTER_OTLP_HEADERS (and its per-signal variants) is now applied
  to OTLP exporters configured from the environment. It was documented but
  ignored, so requests to authenticated collectors went out without
  credentials.
• A crash in any of instrument's processes no longer stops the whole
  application. Before, any single crash took it down, which in a release
  also stops the node. The crashed process is now restarted with the ones
  that depend on it, metrics and registered exporters and processors are
  kept, and only repeated crashes (ten in a minute) stop the application.
• Label key sets such as [a_b] and [a, b] no longer collide. The second
  one's writes used to be lost silently.
• The span and log exporter managers now drop spans and log records once 2048
  are waiting, instead of letting their mailbox grow without limit while an
  export is slow. Drops are reported by stats/0, and the limit is
  configurable. OTLP exports run outside the manager process so retries no
  longer block it.
• The batch span processor drops spans past its queue size before they reach
  its mailbox, sends at most max_export_batch_size spans per export, and
  reports drops through dropped_spans/0.
• force_flush/0 and shutdown/0 on the batch processor no longer time out
  on the caller while an export is still allowed to run.
• The simple and tail-sampling span processors no longer call their exporter
  from many processes at once or rewrite its state in persistent_term on
  every span; each exporter now runs in its own process.
• The flight recorder's memory is now bounded. It used to copy every traced
  message, at full size, into worker mailboxes with no limit, so a busy
  traced process could grow memory quickly. Each worker now holds at most
  flight_recorder_max_queue events (extra events are dropped and counted in
  stats/0), large payloads are stored as their size, and the buffer is
  trimmed as soon as it goes over its limit instead of once per second.
• A nested span ending no longer turns flight tracing off for its parent.
• A dropped (unsampled) span no longer leaves flight tracing on in its
  process after it ends.
• A flight recorder worker that crashes is restarted and keeps receiving
  events; it used to stay dead and its events were lost.
• Out-of-range flight recorder settings (pool size, buffer size) are ignored
  with a warning.
• OTEL_BSP_SCHEDULE_DELAY is now applied to the batch span processor. It
  was stored under the wrong name and silently ignored.
• OTEL_BSP_EXPORT_TIMEOUT and OTEL_METRIC_EXPORT_INTERVAL are now read.
• A zero, negative or non-integer metrics export interval or batch processor
  setting is ignored with a warning. A zero interval used to make the exporter
  loop without pause.
• Stopping the metrics exporter shuts exporters down with their latest state,
  so resources opened during the final export are released.
• Compressed log rotation works in small chunks instead of loading the whole
  file into memory.
• The file log exporter no longer drops the batch it receives right after
  reopening its file.

Removed

• instrument_registry:lookup_label, cache_label,
  create_vector_metric and get_or_create_overflow.
• instrument_lib:table and tables, along with the per-scheduler
  registry tables.
• instrument_vector:collect.