Repository navigation
v0.10.0 — Phase 9: Observability & Performance
v0.10.0 — Phase 9: Observability & Performance
One correlation id and one trace now follow a run from the HTTP request through
the outbox, the broker and the worker. Metrics are exposed per process, the
dashboards are code, and the first real benchmark is committed with its raw
output.
Added
Correlation ids. core.observability.context wraps structlog's contextvars.
The API binds the request id, the run stores it, the outbox row carries it, and
the worker rebinds it per message — clearing first, so one tenant's run cannot
leak into the next log line. Migration 0016 adds correlation_id and
traceparent to WorkflowRun and OutboxMessage.
Prometheus metrics, per process.
GET /api/v1/metrics/on the web process;--metrics-porton the scheduler,
publisher and worker (0or empty disables it; compose sets 9101/9102/9103).- Process-local:
flowforge_task_latency_seconds,flowforge_task_retries_total,
flowforge_lease_expirations_total. - Durable-state, computed at scrape time so they cannot drift:
flowforge_workflow_runs,flowforge_task_runs,flowforge_outbox_pending,
flowforge_outbox_lag_seconds,flowforge_dead_tasks,
flowforge_scheduled_triggers_due.
OpenTelemetry tracing. run.trigger → task.dispatch → task.execute as
one trace across three processes. The W3C traceparent is persisted on the run
and the outbox row because the outbox crosses a process boundary with no live
span. The provider is module-local so a test can install an in-memory exporter.
Grafana dashboards as code + a compose profile.
docker compose --profile observability up adds Prometheus, Grafana and an OTel
collector beside the unchanged application stack. Everything under
deploy/observability/ is code; the dashboard is provisioned read-only.
Load test and measured benchmark. loadtests/locustfile.py drives the whole
pipeline and reports trigger and run end-to-end separately;
manage.py loadtest_user creates the fixture account. A loadtest CI job runs a
fixed 10-user/60-second shape and uploads the raw CSVs.
Measured (GitHub ubuntu-latest, 4 vCPU, AMD EPYC 9V74, 15 GiB, one worker,
Django runserver):
| 5 s interval (default) | 0.25 s interval | |
|---|---|---|
| Runs completed | 31 (124 task runs) | 160 (640 task runs) |
| Failures | 0 | 0 |
| Throughput | 0.36 runs/s | 2.54 runs/s |
POST /runs/ median / p95 |
62 / 150 ms | 77 / 260 ms |
| Run end-to-end median | 20 000 ms | 3 500 ms |
The 20-second figure is 4 dependency levels × 5 s — the poll cadence, not the
engine. Both configurations and the raw output are in
docs/development/BENCHMARKS.md and docs/development/benchmarks/.
Changed
Profiled bottleneck fixed. _record_task_run_event fetched the task run's
organization_id with a separate SELECT whenever the caller did not pass it,
and no caller ever did — although all five already held the row. On a 14-task
diamond that is 13 wasted round trips per run, every pass.
- 5.00 → 4.00 queries per transition (−20%)
- 1 672 → 1 542 queries over 140 scheduler passes (−7.8%)
- Reproduce:
scripts/profile_scheduler.py --micro
Configurable, with the shipped defaults unchanged and asserted by a test:
THROTTLE_AUTH_RATE, THROTTLE_TRIGGER_RATE, THROTTLE_READ_RATE,
SCHEDULER_INTERVAL.
Documentation
docs/deployment/observability.md— the correlation contract, the two kinds of
metric and why they differ, the trace chain across the outbox, the profile, and
a symptom-to-query table.docs/development/load-testing.md— the reproducible command, what each number
means, and what the test does not tell you.docs/development/BENCHMARKS.md— the numbers with their hardware,
configuration and limitations.
Verification
CI: https://github.com/Abolfazlrwm/flowforge/actions/runs/39962719864
- 681 unit tests on PostgreSQL and 29 integration tests green.
lint(ruff, ruff format, mypy 73 files, bandit, pip-audit),schema,docs
green.dockergreen, including the observability profile:flowforge targets up: 4,
scheduler/publisher/worker metrics reachable on 9101/9102/9103, the dashboard
provisioned underflowforge-overview, and the collector accepting an
OTLP/HTTP export.loadtestgreen: the benchmark above.
Known limitations
- No alerting rules ship; thresholds are an operator decision.
OTEL_EXPORTER_OTLP_ENDPOINTis empty by default, so traces are not exported
unless set (an always-set endpoint would make the plain stack log a failed
export every batch interval).- The collector exports to its debug sink; point it at a real backend to keep
spans. - Histogram bucket boundaries are fixed in code.
- The benchmark is CI hardware with the database, broker, workers and load
generator on one VM, andrunserverrather than gunicorn — the figures are a
floor. load_definitionis still re-queried on every scheduler pass (~19% of
schedule_once); caching it would remove a deliberate re-validation.SCHEDULER_INTERVALsets a floor on run latency:
dependency_depth × interval.