Skip to content

7392bb63 - Index the financial-log query columns - #4457

Merged
TaprootFreak merged 2 commits into
developfrom
fix/log-financial-query-index
Jul 29, 2026
Merged

7392bb63 - Index the financial-log query columns#4457
TaprootFreak merged 2 commits into
developfrom
fix/log-financial-query-index

Conversation

@TaprootFreak

@TaprootFreak TaprootFreak commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to #4446.

Goal

Every API call should complete in under 1 second. Measured against production right now, we are far from it:

Sample of 4'995 requests (30 min window)
  p50      63 ms      p90   1'216 ms      p95   2'627 ms
  p99   9'442 ms      max  79'284 ms
  → 11.3 % of all requests exceed 1 s

Even p90 is over the target. This PR is one step, not the answer.

What this changes

The log table is the largest in the database at 1353 MB / 527'339 rows and carried only its primary key. The minute-interval LedgerMarkService query filters on system, subsystem, severity, valid plus a created range and orders by created, id.

EXPLAIN (ANALYZE, BUFFERS) in production:

Limit
  ->  Gather Merge
        ->  Sort  (Sort Key: created, id)
              ->  Parallel Seq Scan on log
                    Rows Removed by Filter: 175629   (per worker)
                    Buffers: shared read=86270       (~674 MB from disk)

The composite index (system, subsystem, severity, valid, created, id) puts the four equality predicates first, then both ORDER BY columns in query order. id is included deliberately: an index ending at created would still require a sort whenever rows share a created value, which happens regularly at ~2 writes per minute.

What this does NOT fix

Disk I/O and the sort step. The query still returns up to 5'001 rows carrying an average 5.8 KB message column — roughly 47 MB transferred per call — which is the dominant remaining cost and keeps the Node event loop busy. Measured: the query executes server-side in 86 ms but sits at 7.3 s in pg_stat_activity; the difference is transfer and deserialisation, not execution.

Closing that gap means aggregating the needed prices in SQL instead of shipping raw JSON, which changes ledger logic and belongs in its own PR.

Migration notes

  • No CONCURRENTLY — migrations here run transactionally and boot-blocking (migrationsRun, gated by SQL_MIGRATE), and CREATE INDEX CONCURRENTLY is not permitted inside a transaction.
  • Lock behaviour, stated precisely: a plain CREATE INDEX holds a SHARE lock for the entire build, not briefly. Reads continue; writes to log block for the duration. SET LOCAL lock_timeout caps only the wait to acquire the lock, not how long it is held. The build duration has not been measured against production data, so no upper bound is claimed. Judged acceptable because log takes ~2'900 rows/day and those writes are retried by their jobs rather than lost.
  • No IF NOT EXISTS — the migration should fail loudly if the index unexpectedly exists rather than continue silently.

Review

One round, three findings, all addressed: id added as the sixth column (the original claim of eliminating the sort was not fully accurate), the misleading "short SHARE lock" wording corrected, and down() now schema-qualifies the index like every existing migration.

The log table is the largest in the database at 1353 MB (527339 rows) and
carried only its primary key. The minute-interval LedgerMarkService query
filters on system, subsystem, severity and valid with a created range, then
orders by created — which forced a Parallel Seq Scan plus an explicit Sort,
reading ~674 MB from disk per call and discarding 175629 rows per worker at
the filter.

The composite index puts the four equality predicates first and the range
column last, so Postgres can serve both the filter and the ordering from it.

This addresses disk load and the sort step only. The transferred payload —
up to 5001 rows carrying a 5.8 KB message column, roughly 47 MB per call —
is unchanged and remains the dominant cost on the Node side.
Review follow-ups.

The query orders by (created, id), so an index ending at created still needs
a sort step whenever rows share a created value. id is now the sixth column.

The lock comment claimed a short SHARE lock. It is held for the entire build,
and lock_timeout caps only the wait to acquire it, not the hold. The build
duration was not measured against production, so no upper bound is claimed.

down() now schema-qualifies the index like every existing migration does.
@TaprootFreak
TaprootFreak marked this pull request as ready for review July 29, 2026 15:06
@TaprootFreak

Copy link
Copy Markdown
Collaborator Author

Ready after one review round, three findings, all addressed:

  • MAJOR — the index ended at created while the query orders by created, id, so a sort would still have been required whenever rows share a timestamp. id is now the sixth column, and the PR description no longer claims the sort is eliminated outright.
  • MINOR — the "short SHARE lock" wording was misleading: the lock is held for the entire build, and lock_timeout caps only the wait to acquire it. Corrected, with an explicit note that the build duration was not measured against production, so no upper bound is claimed.
  • MINORdown() now schema-qualifies the index ("public"."IDX_…") like all four existing migrations do.

CI green on the final commit (13/13, including Migration immutability), both commits signed.

@TaprootFreak
TaprootFreak merged commit 274949f into develop Jul 29, 2026
13 checks passed
@TaprootFreak
TaprootFreak deleted the fix/log-financial-query-index branch July 29, 2026 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant