Release v1.0.0 · Neutrinic/flare

Flare v1.0.0 — Full-stack OpenTelemetry observability for Apache Spark

First stable release. Distributed tracing, metrics, and logs correlated across driver and executor JVMs — zero code changes required.

Highlights

Full span hierarchy — app → sql → job → stage → task spans across two JVM services with zero orphan spans
Real executor task spans — accurate wall-clock timing on the executor thread, not driver-side approximations
Per-stage context propagation — W3C traceparent injected per-stage via ByteBuddy hook on DAGScheduler.submitMissingTasks, so each task inherits its specific stage span as parent
OTEL metrics — task duration histograms with exemplar links, shuffle I/O counters, stage aggregates, records throughput
Trace-correlated logs — driver and executor logs linked to spans via OTLP, with MDC enrichment (trace ID + span ID)
Provisioned Grafana dashboard — task duration heatmaps, shuffle skew detection, executor comparison, stage summary, logs, and trace links out of the box
Published to Maven Central — --packages io.github.neutrinic:flare-spark-3-5_2.13:1.0.0

spark-submit \
  --packages io.github.neutrinic:flare-spark-3-5_2.13:1.0.0 \
  --conf "spark.plugins=io.flare.spark.plugin.FlareSparkPlugin" \
  ...

Artifacts for all supported Spark × Scala combinations:

Tracing

ByteBuddy SparkContextInstrumentationModule — auto-registers listener + executor plugin (#14)
ByteBuddy TaskRunnerInstrumentationModule — restores OTEL context on executor (#15)
Per-stage traceparent injection via DAGScheduler.submitMissingTasks (#26)
TracingSparkListener — application/job/stage span lifecycle on driver (#1)
FlareSparkPlugin — backward-compatible SparkPlugin fallback (#1)
Granularity control: FLARE_TRACE_GRANULARITY, FLARE_SLOW_TASK_MS, FLARE_RETRY_TASKS_ONLY (#3)
Executor shutdown flush for dynamic allocation / K8s SIGTERM (#7)

Metrics

Task duration histograms with exemplar links to traces (#11)
Shuffle read/write bytes per task and per stage (#8, #11)
Stage aggregates: executor run time, input/output bytes (#11)
Records throughput histogram (#11)
flare_* metric namespace to avoid collision with Spark's PrometheusServlet (#31)

Logs

MDC enrichment: trace_id and span_id injected into log context during task execution (#4)

Observability stack

Build & CI