Read each rule's latest order through a LATERAL instead of aggregating every order

[davidleomay]

TradingJobService::processRules runs every minute and calls getCurrentTradingOrders, which asked PostgreSQL for MAX(id) GROUP BY tradingRuleId across the whole trading_order table. DFXServer/server#1223 measured it at roughly 0.5 s a run, 20 runs over 100 ms within 75 minutes.

Why the existing index did not fix it

The composite index on trading_order ("tradingRuleId", "id") (#4497, merged 30 July) is already in place and was live during that measurement. It does not help, because PostgreSQL has no skip scan: a per-group MAX reads the index end to end. The cost grows with the number of orders, which is unbounded, while the answer only ever has as many rows as there are rules.

The change

Driven from trading_rule instead, with a LATERAL that descends the same index once per rule and stops at the first row:

SELECT latest."id" AS "tradingOrderId"
FROM "trading_rule" rule
CROSS JOIN LATERAL (
  SELECT "order"."id" FROM "trading_order" "order"
  WHERE "order"."tradingRuleId" = rule."id"
  ORDER BY "order"."id" DESC LIMIT 1
) latest

Cost now grows with the number of rules — configuration, and small.

Still one statement, which is a correctness requirement rather than a preference. The previous comment records why: LogJobService writes the FinanceLog from this result, so every rule's latest order has to come from the same READ-COMMITTED snapshot. A per-rule loop would be faster to write and would quietly break that, mixing rows from different points in time. A LATERAL is a single statement under a single snapshot, so the property survives.

Set semantics are unchanged from the INNER JOIN: driving from trading_rule drops orders whose rule no longer exists, and a rule with no orders contributes nothing because its lateral subquery returns no row.

Raw SQL because LATERAL has no query-builder equivalent. There is precedent for raw reads in this codebase (24 call sites, e.g. bank-tx.service.ts:597).

The test engine had to move, and that is part of the change

The previous implementation comment said not to write this query without first solving pg-mem, and that was accurate — I checked rather than assuming. pg-mem cannot execute a LATERAL: it fails to resolve the subquery's reference to the outer row with column "rule.id" does not exist.

So trading-rule.service.pg.spec.ts becomes trading-rule.service.latest-orders.spec.ts, running against a real PostgreSQL behind MIGRATION_TEST_PG — the same gate the migration and projection specs use. All three original assertions are carried over unchanged: highest id per rule with empty rules and orphans excluded, empty table returning an empty array, and no null or undefined ids reaching findBy In(...).

Two assertions are added. One covers rules existing while none has an order — the case that separates CROSS JOIN LATERAL from LEFT JOIN LATERAL, where the latter would produce a null id and carry it into In(...). The other is an EXPLAIN assertion that the plan is a nested loop over the rules and contains no aggregate, so a future revert to MAX … GROUP BY fails there while every behavioural assertion still passes.

The cost is real and worth stating plainly: these assertions ran on every machine before and now skip without MIGRATION_TEST_PG. They run in CI, which is also the only place the plan assertion means anything.

What is not verified

That this is faster in production. Dev holds zero trading rules and zero orders, and I have no production access, so there is no before/after timing here — only the plan shape, asserted in CI. The argument for the change is structural: the old form's cost scales with a table that grows without bound, the new form's with one that does not.

No entity, column or DTO changes, so nothing else in the completeness list applies. No @DfxCron change: TradingJobService::processRules is untouched.