feat(kalshi): add market_details and market_trades gold layer

[los-xyz]

Summary

Adds two new Kalshi prediction market spells, transforming API-sourced bronze tables into a clean gold layer:

• kalshi.market_details (TABLE): Market reference table joining markets_raw with event metadata from market_details_raw. Filtered to markets with >= 100 contracts traded.
• kalshi.market_trades (VIEW): Trade-level table enriched with market metadata via inner join to market_details.

Design choices (bronze → gold)

• >= 100 contracts filter: drops 85% of markets (dust/empty), keeps 99.7% of volume
• 12 columns dropped: universally null (functional_strike, mve_*, is_provisional, fee_waiver_expiration_time), constant (response_price_units, notional_value_dollars, price_ranges), always zero (liquidity_dollars), borderline sparse (rules_secondary, primary_participant_key, settlement_timer_seconds), internal (created_hour)
• INNER join on trades: ensures only trades for meaningful markets flow to gold layer
• Pricing snapshot kept: despite being latest-state-only, the orderbook/OI columns are well-populated on active markets and useful for current-state analysis

Test plan

• dbt compile passes for both models
• CI builds and tests pass
• Verify market_details unique on ticker
• Verify market_trades unique on trade_id
• Spot-check join completeness (NULL rates on enriched columns)

🤖 Generated with Claude Code

[jeff-dude]

pushed changes to make the models incremental, since larger tables.
would be good if you could test the output build in CI here. note that CI tables get dropped over the weekend, so you may need to rerun the actions to rebuild if looking later.
plz ensure these outputs still match what you expect

[cursor]

PR Summary

Medium Risk
Introduces new incremental/merge models with non-trivial refresh predicates, which can impact warehouse cost and data correctness (e.g., missed/duplicated rows) if predicates or keys are wrong; changes are isolated to new Kalshi datasets.

Overview
Adds a new Kalshi gold layer: kalshi.market_details (incremental merge on ticker) joins markets_raw with the latest per-event_ticker record from market_details_raw, filters to markets with volume_fp >= 100, and tracks source_updated_at to capture event-only metadata changes.

Adds kalshi.market_trades (incremental merge partitioned by block_month) that pulls from market_trades_raw, enriches trades via an inner join to kalshi_market_details, and reprocesses historical trades when market_details.source_updated_at changes to keep dimension columns current. Also registers the new Kalshi bronze sources (market_trades_raw, markets_raw, market_details_raw) and adds schema tests/docs for uniqueness and non-null keys.

^{Reviewed by Cursor Bugbot for commit 90f1a88. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Missing deduplication on event_details CTE causes row fan-out
  • Added QUALIFY with ROW_NUMBER() to event_details CTE to keep only the latest row per event_ticker, preventing join fan-out and duplicate ticker rows.

Or push these changes by commenting:

@cursor push 491ff46c63

Preview (491ff46c63)

diff --git a/dbt_subprojects/daily_spellbook/models/_projects/kalshi/kalshi_market_details.sql b/dbt_subprojects/daily_spellbook/models/_projects/kalshi/kalshi_market_details.sql
--- a/dbt_subprojects/daily_spellbook/models/_projects/kalshi/kalshi_market_details.sql
+++ b/dbt_subprojects/daily_spellbook/models/_projects/kalshi/kalshi_market_details.sql
@@ -54,6 +54,7 @@
 		strike_period,
 		last_updated_ts
 	from {{ source('kalshi', 'market_details_raw') }}
+	qualify row_number() over (partition by event_ticker order by last_updated_ts desc) = 1
 )
 
 select

_{You can send follow-ups to the cloud agent here.}

[jeff-dude]

@cursor review

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: amount_usd ignores taker side, wrong for No trades
  • Changed amount_usd calculation from always using yes_price_dollars to conditionally using yes_price_dollars or no_price_dollars based on taker_side.

Or push these changes by commenting:

@cursor push 8872215acd

Preview (8872215acd)

diff --git a/dbt_subprojects/daily_spellbook/models/_projects/kalshi/_schema.yml b/dbt_subprojects/daily_spellbook/models/_projects/kalshi/_schema.yml
--- a/dbt_subprojects/daily_spellbook/models/_projects/kalshi/_schema.yml
+++ b/dbt_subprojects/daily_spellbook/models/_projects/kalshi/_schema.yml
@@ -186,7 +186,7 @@
       - name: no_price_dollars
         description: "Price paid for the No side"
       - name: amount_usd
-        description: "USD notional value of the trade (yes_price_dollars * count_fp)"
+        description: "USD notional value of the trade (taker price * count_fp, where taker price depends on taker_side)"
       - name: event_ticker
         description: "Parent event identifier"
       - name: series_ticker

diff --git a/dbt_subprojects/daily_spellbook/models/_projects/kalshi/kalshi_market_trades.sql b/dbt_subprojects/daily_spellbook/models/_projects/kalshi/kalshi_market_trades.sql
--- a/dbt_subprojects/daily_spellbook/models/_projects/kalshi/kalshi_market_trades.sql
+++ b/dbt_subprojects/daily_spellbook/models/_projects/kalshi/kalshi_market_trades.sql
@@ -69,7 +69,7 @@
 	, t.count_fp
 	, t.yes_price_dollars
 	, t.no_price_dollars
-	, t.yes_price_dollars * t.count_fp as amount_usd
+	, case when t.taker_side = 'yes' then t.yes_price_dollars else t.no_price_dollars end * t.count_fp as amount_usd
 	, md.event_ticker
 	, md.series_ticker
 	, md.market_type

_{You can send follow-ups to the cloud agent here.}

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 90f1a88. Configure here.}