Skip to content

v0.10.0 — Phase 9: Observability & Performance

Choose a tag to compare

@Abolfazlrwm Abolfazlrwm released this 23 Oct 13:42
· 62 commits to main since this release

v0.10.0 — Phase 9: Observability & Performance

One correlation id and one trace now follow a run from the HTTP request through
the outbox, the broker and the worker. Metrics are exposed per process, the
dashboards are code, and the first real benchmark is committed with its raw
output.

Added

Correlation ids. core.observability.context wraps structlog's contextvars.
The API binds the request id, the run stores it, the outbox row carries it, and
the worker rebinds it per message — clearing first, so one tenant's run cannot
leak into the next log line. Migration 0016 adds correlation_id and
traceparent to WorkflowRun and OutboxMessage.

Prometheus metrics, per process.

  • GET /api/v1/metrics/ on the web process; --metrics-port on the scheduler,
    publisher and worker (0 or empty disables it; compose sets 9101/9102/9103).
  • Process-local: flowforge_task_latency_seconds, flowforge_task_retries_total,
    flowforge_lease_expirations_total.
  • Durable-state, computed at scrape time so they cannot drift:
    flowforge_workflow_runs, flowforge_task_runs, flowforge_outbox_pending,
    flowforge_outbox_lag_seconds, flowforge_dead_tasks,
    flowforge_scheduled_triggers_due.

OpenTelemetry tracing. run.trigger → task.dispatch → task.execute as
one trace across three processes. The W3C traceparent is persisted on the run
and the outbox row because the outbox crosses a process boundary with no live
span. The provider is module-local so a test can install an in-memory exporter.

Grafana dashboards as code + a compose profile.
docker compose --profile observability up adds Prometheus, Grafana and an OTel
collector beside the unchanged application stack. Everything under
deploy/observability/ is code; the dashboard is provisioned read-only.

Load test and measured benchmark. loadtests/locustfile.py drives the whole
pipeline and reports trigger and run end-to-end separately;
manage.py loadtest_user creates the fixture account. A loadtest CI job runs a
fixed 10-user/60-second shape and uploads the raw CSVs.

Measured (GitHub ubuntu-latest, 4 vCPU, AMD EPYC 9V74, 15 GiB, one worker,
Django runserver):

5 s interval (default) 0.25 s interval
Runs completed 31 (124 task runs) 160 (640 task runs)
Failures 0 0
Throughput 0.36 runs/s 2.54 runs/s
POST /runs/ median / p95 62 / 150 ms 77 / 260 ms
Run end-to-end median 20 000 ms 3 500 ms

The 20-second figure is 4 dependency levels × 5 s — the poll cadence, not the
engine. Both configurations and the raw output are in
docs/development/BENCHMARKS.md and docs/development/benchmarks/.

Changed

Profiled bottleneck fixed. _record_task_run_event fetched the task run's
organization_id with a separate SELECT whenever the caller did not pass it,
and no caller ever did — although all five already held the row. On a 14-task
diamond that is 13 wasted round trips per run, every pass.

  • 5.00 → 4.00 queries per transition (−20%)
  • 1 672 → 1 542 queries over 140 scheduler passes (−7.8%)
  • Reproduce: scripts/profile_scheduler.py --micro

Configurable, with the shipped defaults unchanged and asserted by a test:
THROTTLE_AUTH_RATE, THROTTLE_TRIGGER_RATE, THROTTLE_READ_RATE,
SCHEDULER_INTERVAL.

Documentation

  • docs/deployment/observability.md — the correlation contract, the two kinds of
    metric and why they differ, the trace chain across the outbox, the profile, and
    a symptom-to-query table.
  • docs/development/load-testing.md — the reproducible command, what each number
    means, and what the test does not tell you.
  • docs/development/BENCHMARKS.md — the numbers with their hardware,
    configuration and limitations.

Verification

CI: https://github.com/Abolfazlrwm/flowforge/actions/runs/39962719864

  • 681 unit tests on PostgreSQL and 29 integration tests green.
  • lint (ruff, ruff format, mypy 73 files, bandit, pip-audit), schema, docs
    green.
  • docker green, including the observability profile: flowforge targets up: 4,
    scheduler/publisher/worker metrics reachable on 9101/9102/9103, the dashboard
    provisioned under flowforge-overview, and the collector accepting an
    OTLP/HTTP export.
  • loadtest green: the benchmark above.

Known limitations

  • No alerting rules ship; thresholds are an operator decision.
  • OTEL_EXPORTER_OTLP_ENDPOINT is empty by default, so traces are not exported
    unless set (an always-set endpoint would make the plain stack log a failed
    export every batch interval).
  • The collector exports to its debug sink; point it at a real backend to keep
    spans.
  • Histogram bucket boundaries are fixed in code.
  • The benchmark is CI hardware with the database, broker, workers and load
    generator on one VM, and runserver rather than gunicorn — the figures are a
    floor.
  • load_definition is still re-queried on every scheduler pass (~19% of
    schedule_once); caching it would remove a deliberate re-validation.
  • SCHEDULER_INTERVAL sets a floor on run latency:
    dependency_depth × interval.