feat(dex): deterministic 0x Settler aggregator volume — supersede #9795 (CUR2-2844)

[thevaizman]

Stacked on #9798 (Phase 1). Rewrites the settler aggregator decode; requires a coordinated full-refresh (see deploy note). CI green; @tomfutago's Phase 1 regression passed and the Phase 2 blocker below is fixed.

What

Replaces the heuristic maker/taker leg-matching (and PR #9795's 5×-divergence band-aid) for 0x Settler trades in dex_aggregator.trades with a deterministic decode. Per settler call → one 0x API aggregator row = the user's net swap.

Method (verified across 3 on-chain R&D rounds)

• token_bought = AllowedSlippage.buyToken (calldata, deterministic).
• amount = the unique Transfer(buyToken, to=receiver) when resolvable, else the minAmountOut floor (verified tight — median ~1% under actual), guarded against "no minimum" sentinels (minAmountOut ≥ 2^128 → null, never the floor — see the 0xc01c2829… blocker fix below).
• receiver = tx-level sender (execute) / msgSender (executeMetaTxn) — verified to be the true buyToken recipient (reconciled on 0xf61374…: the calldata recipient is an intermediate hop; trace.from is the AllowanceHolder). Anchoring on a single calldata-named token to the verified user means it can't mis-bind to internal routing hops the way the heuristic did.
• CoW-routed fills (cow_rn set, ~0.5% of settler calls): the tx-level receiver is the CoW solver/settlement, not the user, so the receiver-pivot is unreliable amid the batch → use the deterministic minAmountOut floor + null sell leg (buyToken stays the calldata value). The deterministic analogue of zeroex_v2's cow_log_range window scoping.
• token_sold = best-effort (unique Transfer out of the user, token ≠ buyToken).
• volume_usd via either leg (add_amount_usd) → an unpriced buyToken still values off the sell leg.

Why it supersedes #9795

#9795 is a coarse 5×-threshold proxy that only patches volume_usd and leaves the wrong token columns. This computes the user's net swap deterministically. Example 0xf61374…: current $0.75 → deterministic ~$497.

Files

New macro zeroex_settler_agg; settler-txs staging extended with the AllowedSlippage fields (buy_token, min_amount_out, settler_msgsender); all 17 settler chains' zeroex_v2_<chain>_trades rewired to it. Restores the zid != 0xa00… sentinel exclusion, adds CoW floor-fallback, and preserves berachain/ink start dates (2025-02-15).

Validation

• ✅ Decode verified on-chain (3 R&D rounds); ✅ CI green (dex); ✅ @tomfutago's Phase 1 regression passed (key parity + exact set-diff + dex_info).
• ✅ Bugbot + reviewer comments resolved: sentinel zid, CoW scoping, berachain/ink start dates, and the minAmountOut = UINT256_MAX blocker (Base 0xc01c2829…, was ~1e71 USD) — now guarded.

⚠️ Deploy note — coordinated full-refresh, ORDER MATTERS

1. First, refresh / schema-sync all 17 zeroex_v2_<chain>_settler_txs staging tables — they exist in prod but lack this PR's new buy_token / min_amount_out / settler_msgsender columns, so a trades full-refresh would read null staging until they're synced (thanks @tomfutago for catching this dependency).
2. Then full-refresh zeroex_v2_<chain>_trades.
3. Then the large dex_aggregator_trades (merge can't delete the old mis-bound rows — the CUR2-2693 caution).

Stacked on #9798 (CUR2-2843). CUR2-2844

🤖 Generated with Claude Code

[cursor]

PR Summary

High Risk
Rewrites core DEX aggregator volume and token attribution for 0x Settler on every supported chain; large historical deltas and merge-only incremental runs cannot remove prior bad rows without a full refresh.

Overview
Replaces heuristic 0x Settler aggregator trade decoding (and the #9795 volume band-aid) with a deterministic path: one 0x-API row per settler call representing the user’s net swap.

zeroex_settler_txs_cte now surfaces AllowedSlippage from calldata (buy_token, min_amount_out, and settler_msgsender for executeMetaTxn). The new zeroex_settler_agg macro reads materialized settler txs, joins partition-pruned transactions and ERC20 Transfer logs, and sets token_bought from calldata, amount from a unique inbound transfer to the verified receiver (tx from or meta-txn msgSender) or minAmountOut, and token_sold from a best-effort outbound transfer; volume_usd uses add_amount_usd on either leg.

All 17 zeroex_v2_<chain>_trades models drop the zeroex_v2_trades / zeroex_v2_trades_detail stack and call zeroex_settler_agg only. Deploy expects a coordinated full-refresh so merge does not leave old mis-bound rows in downstream aggregator tables.

^{Reviewed by Cursor Bugbot for commit 9f732a5. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.

Bugbot Autofix prepared fixes for both issues found in the latest run.

• ✅ Fixed: Missing sentinel zid exclusion
  • Added the missing sentinel zid filter to the settler aggregator input so those staged rows cannot emit aggregator trades.
• ✅ Fixed: CoW settler log scoping removed
  • Restored CoW-specific transfer scoping by carrying cow_rn through the aggregator and limiting CoW leg matching to pre-settlement-window transfers involving the settler or GPv2 settlement contract.

Or push these changes by commenting:

@cursor push 795034933e

Preview (795034933e)

diff --git a/dbt_subprojects/dex/macros/models/_project/zeroex/zeroex_settler_agg.sql b/dbt_subprojects/dex/macros/models/_project/zeroex/zeroex_settler_agg.sql
--- a/dbt_subprojects/dex/macros/models/_project/zeroex/zeroex_settler_agg.sql
+++ b/dbt_subprojects/dex/macros/models/_project/zeroex/zeroex_settler_agg.sql
@@ -5,6 +5,9 @@
 {%- set weth = '0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2' -%}
 {%- set native_tokens = '(0x0000000000000000000000000000000000000000, 0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee)' -%}
 {%- set erc20_transfer_topic = '0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef' -%}
+{%- set cow_trade_topic = '0xed99827efb37016f2275f98c4bcf71c7551c75d59e9b450f79fa32e60be672c2' -%}
+{%- set cow_protocol_settlement = '0x9008d19f58aabd9ed0d60971565aa8510560ab41' -%}
+{%- set sentinel_zid = '0xa00000000000000000000000' -%}
 
 -- Deterministic 0x Settler aggregator decode (replaces the heuristic leg-matching + the PR #9795 band-aid).
 -- Per settler call, emits ONE 0x-API aggregator row = the user's net swap:
@@ -21,7 +24,7 @@
 
 WITH settler AS (
     SELECT
-        tx_hash, block_time, block_number, method_id, settler_address, zid, tag, rn,
+        tx_hash, block_time, block_number, method_id, settler_address, zid, tag, rn, cow_rn,
         buy_token, min_amount_out, settler_msgsender
     FROM {{ ref('zeroex_v2_' ~ blockchain ~ '_settler_txs') }}
     {% if is_incremental() %}
@@ -29,6 +32,7 @@
     {% else %}
     WHERE block_time >= DATE '{{ start_date }}'
     {% endif %}
+      AND zid != {{ sentinel_zid }}
 ),
 
 -- Distinct partition key of the settler txs, so the transactions/logs scans prune by partition
@@ -54,7 +58,7 @@
 
 calls AS (
     SELECT
-        s.tx_hash, s.block_time, s.block_number, s.settler_address, s.zid, s.tag, s.rn, s.min_amount_out,
+        s.tx_hash, s.block_time, s.block_number, s.settler_address, s.zid, s.tag, s.rn, s.cow_rn, s.min_amount_out,
         t.tx_from, t.tx_to,
         -- receiver (the user): execute -> tx-level sender; executeMetaTxn -> msgSender (relayer pays gas).
         CASE WHEN s.method_id = 0xfd3ad6d4 THEN s.settler_msgsender ELSE t.tx_from END AS receiver,
@@ -64,26 +68,30 @@
     LEFT JOIN txs t ON t.tx_hash = s.tx_hash
 ),
 
-transfers AS (
+tx_logs AS (
     SELECT
         logs.block_number,
         logs.tx_hash,
+        logs.index AS log_index,
         logs.contract_address AS token,
-        varbinary_substring(logs.topic1, 13, 20) AS transfer_from,
-        varbinary_substring(logs.topic2, 13, 20) AS transfer_to,
-        -- CASE-guard the conversion (Trino only evaluates the THEN branch when the WHEN holds): a non-standard
-        -- Transfer-topic log with >32-byte data would otherwise overflow bytearray_to_uint256. The WHERE filter
-        -- below is not sufficient on its own — Trino may evaluate this projection before applying it.
-        CASE WHEN varbinary_length(logs.data) = 32 THEN bytearray_to_uint256(logs.data) END AS amount
+        logs.topic0,
+        logs.topic1,
+        logs.topic2,
+        logs.data,
+        max(CASE
+                WHEN logs.topic0 = {{ cow_trade_topic }}
+                    AND varbinary_substring(logs.topic1, 13, 20) != 0xDef1C0ded9bec7F1a1670819833240f027b25EfF
+                    THEN logs.index
+            END) OVER (PARTITION BY logs.block_number, logs.tx_hash) AS cow_max_index
     FROM {{ source(blockchain, 'logs') }} AS logs
     JOIN settler_tx_keys k
         ON  k.block_time = logs.block_time
         AND k.block_number = logs.block_number
         AND k.tx_hash = logs.tx_hash
-    WHERE logs.topic0 = {{ erc20_transfer_topic }}
+    WHERE logs.topic0 IN ({{ erc20_transfer_topic }}, {{ cow_trade_topic }})
       -- standard ERC20 value transfers only (uint256 amount is exactly 32 bytes): drops NFT/ERC721 (0-byte
-      -- data) and non-standard >32-byte Transfer-topic logs. Row-reducer; the overflow guard is the CASE above.
-      AND varbinary_length(logs.data) = 32
+      -- data) and non-standard >32-byte Transfer-topic logs. Row-reducer; the overflow guard is the CASE below.
+      AND (logs.topic0 != {{ erc20_transfer_topic }} OR varbinary_length(logs.data) = 32)
     {% if is_incremental() %}
       AND {{ incremental_predicate('logs.block_time') }}
     {% else %}
@@ -91,6 +99,23 @@
     {% endif %}
 ),
 
+transfers AS (
+    SELECT
+        logs.block_number,
+        logs.tx_hash,
+        logs.log_index,
+        logs.token,
+        varbinary_substring(logs.topic1, 13, 20) AS transfer_from,
+        varbinary_substring(logs.topic2, 13, 20) AS transfer_to,
+        -- CASE-guard the conversion (Trino only evaluates the THEN branch when the WHEN holds): a non-standard
+        -- Transfer-topic log with >32-byte data would otherwise overflow bytearray_to_uint256. The WHERE filter
+        -- below is not sufficient on its own — Trino may evaluate this projection before applying it.
+        CASE WHEN varbinary_length(logs.data) = 32 THEN bytearray_to_uint256(logs.data) END AS amount,
+        logs.cow_max_index
+    FROM tx_logs AS logs
+    WHERE logs.topic0 = {{ erc20_transfer_topic }}
+),
+
 -- Single pass over the transfers: buy leg = unique Transfer(buyToken, to=receiver); sell leg (best-effort) =
 -- unique Transfer out of the user of a token other than buyToken.
 legs AS (
@@ -106,6 +131,16 @@
         AND t.block_number = c.block_number
         AND t.amount > UINT256 '0'
         AND (
+              c.cow_rn IS NULL
+           OR (
+                  t.log_index < t.cow_max_index
+              AND (
+                      t.transfer_from IN (c.settler_address, {{ cow_protocol_settlement }})
+                   OR t.transfer_to IN (c.settler_address, {{ cow_protocol_settlement }})
+              )
+           )
+        )
+        AND (
               (t.token = c.buy_token AND t.transfer_to = c.receiver)
            OR (t.transfer_from = c.receiver AND t.token <> c.buy_token)
         )

_{You can send follow-ups to the cloud agent here.}

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 9f732a5. Configure here.}

[tomfutago]

Regression found one blocker in the CI-built output.

For zeroex_v2_base_trades, CI emits this row:

tx_hash = 0xc01c2829a45c91f4a18e3bd3d1f28d6541b845ac581ad35a3e512b66dd250e75

It has:

maker_token = USDC
maker_token_amount_raw = 115792089237316195423570985008687907853269984665640564039457584007913129639935
volume_usd = 1.1577252037423508e+71

The corresponding CI staging row has min_amount_out = UINT256_MAX, and the new aggregator falls back to min_amount_out when it cannot resolve a unique buy transfer. So this looks like a sentinel/max-value minAmountOut case that needs to be filtered or guarded before fallback.

This is isolated to one Base row over $1B, but it makes the Base aggregate unusable. Excluding that row, Base CI volume is $629.5M vs bounded prod $324.2M; with it included, CI volume is ~1.16e71.

Uniqueness passed across the non-empty CI zeroex_v2_*_trades models, so this is not a duplicate-key issue.

[tomfutago]

Also, one deployment-path note before merge: the downstream refresh plan should include the zeroex_v2_<chain>_settler_txs staging tables, not just zeroex_v2_<chain>_trades and dex_aggregator_trades.

I checked prod via Dune: all 17 zeroex_v2_* .settler_txs tables already exist, but they currently only have the old shape:

tx_hash, block_time, block_number, method_id, contract_address, settler_address, zid, tag, rn, cow_rn, taker, block_month

They do not yet have the columns this PR’s zeroex_settler_agg requires:

buy_token, min_amount_out, settler_msgsender

So a historical full refresh of the trades models would read incomplete/null staging data unless the corresponding settler_txs models are refreshed/schema-synced first. Can we update the deploy note to explicitly include a coordinated refresh of all zeroex_v2_<chain>_settler_txs staging tables before or with the downstream trade refresh?

[tomfutago]

@thevaizman plz address regression blocker

[thevaizman]

@tomfutago thanks for the thorough regression — both items addressed in 7749f6669:

Blocker (minAmountOut = UINT256_MAX → ~1e71 USD): fixed. The agg now guards the floor fallback against "no minimum" sentinels — minAmountOut >= 2^128 is nulled (never used as the floor), so those rows yield null volume instead of an absurd amount. I measured the landscape first: 165 of 3.8M settler calls (ethereum+base, 30d) carry minAmountOut > 2^128 (1 exactly UINT256_MAX, 164 scattered across the high range), all far above any real token amount — even a $1B trade sits well under 2^128 — so a 2^128 ceiling catches every sentinel without dropping a legitimate floor. Confirmed on your exact example 0xc01c2829a45c91f4a18e3bd3d1f28d6541b845ac581ad35a3e512b66dd250e75: minAmountOut = 2^256-1, buyToken USDC (0x833589fcd6edb6e08f4c7c32d4f71b54bda02913) — now nulled, no blow-up. (Same commit also restored the zid != 0xa00… sentinel filter and added CoW-routed floor handling — see the resolved Bugbot threads.)

Deploy-path note (staging tables): great catch. Updated the PR description's deploy note to make the order explicit — refresh/schema-sync all 17 zeroex_v2_<chain>_settler_txs staging tables (they need the new buy_token / min_amount_out / settler_msgsender columns) first, then zeroex_v2_<chain>_trades, then dex_aggregator_trades.

Re-requesting your review 🙏

[thevaizman]

@tomfutago follow-up on the blocker — I investigated all 22 of these end-to-end (cast + dune), and the minAmountOut = UINT256_MAX was a symptom, not the disease. They're 0x Settler calls that reverted inside ERC-4337 UserOps: handleOps catches the inner revert so the outer tx succeeds, but the settler trace itself is success = false, emits no ERC-20 transfers, and the EntryPoint logs UserOperationRevertReason. All 22 share one 4337-settler contract and all reverted — genuine non-trades.

So I moved the fix to the root: the settler-txs staging now excludes reverted traces (AND success). That drops all 22, and more importantly cleans up 12.29% of Base settler calls (415,495 / 3,380,685 over 30d) that are reverts and were being recorded as trades at all. I kept the sentinel-minAmountOut guard as defense-in-depth (on Base there are 0 successful calls with a sentinel mao, so the success filter alone covers it). The RFQ path (Phase 1) was already robust — reverts have no maker-pivot transfers, so they're dropped — so Phase 1's output is unchanged.

Pushed in 78e059bd2. Deploy note unchanged (still need the zeroex_v2_<chain>_settler_txs staging schema-sync first).