Skip to content

dev → master: ephemeral UTAs + IBKR-as-truth scope correction + FRED tools#171

Merged
luokerenx4 merged 15 commits intomasterfrom
dev
May 8, 2026
Merged

dev → master: ephemeral UTAs + IBKR-as-truth scope correction + FRED tools#171
luokerenx4 merged 15 commits intomasterfrom
dev

Conversation

@luokerenx4
Copy link
Copy Markdown
Contributor

@luokerenx4 luokerenx4 commented May 8, 2026

Summary

Two distinct streams landed on dev since the last cut to master:

  1. Trading subsystem maturation — ephemeral mock-UTA lifecycle for fixture testing, three real bug fixes surfaced via a hand-driven covered-call simulator session, and an architectural correction to the IBKR-as-truth refactor (Phase 3's canonical localSymbol erased CCXT's USDC-vs-USDT margin distinction; reverted in favor of broker-native uniqueness).
  2. Macro / economic data layer end-to-end — three independent failures along the FRED credential / executor / tool path, plus four more on the EIA path, all repaired and pushed through to the AI tool layer. AI agents can now read FRED + EIA macro data through one cohesive economy* toolset. UI provider-key form audited and pruned of dead links / unimplemented providers.

Per-session contributions

2026-05-08 — Macro data layer (FRED + EIA) end-to-end + UI cleanup

  • fix(market-data): repair FRED end-to-end + add market-data e2e harness (a86612c)
    • Three credential-chain misalignments fixed at once: webui/routes/config.ts referenced nonexistent fred provider (registry has federal_reserve); credential-map sent fred_api_key to both paths but in-process opentypebb auto-prefixes; etc. Adds a market-data e2e harness for future regressions.
  • feat(tool): expose FRED via three economy tools (search/series/regional) (0c7865e)
    • Wraps SDKEconomyClient (which had been exported but never instantiated by a factory). Three Vercel AI tools route through EconomyClientLike with provider: 'federal_reserve' pinned at the tool boundary. Schema-validation tests (economy.spec.ts) catch zod / passthrough / error-propagation regressions in CI.
  • fix(ui): MarketDataPage hints — drop unimplemented providers, refresh dead URLs (86233d6)
    • Audited the 8 entries in the API Keys form. Dropped nasdaq + tradingeconomics (no SDK provider registered — Test button always 500'd). Refreshed bls (registrationapps.bls.gov was retired; now points at data.bls.gov/registrationEngine/), econdb ("Optional" → "Required (free signup)" — anonymous API now 401), eia (direct register.php link), fred (the /apikeys deep link bounces unauthenticated users; standard fred.stlouisfed.org → My Account flow now). Widened intrinio and fmp descriptions to match what the providers actually return.
  • fix(eia): repair credential prefix + sort syntax + value coercion (1182dd9)
    • Same /test-provider button caught the next provider's failure layer. EIA had three independent bugs: (1) credentials: ['eia_api_key'] was already-prefixed and got double-prefixed by the Provider constructor to eia_eia_api_key, blocking credential propagation (mirror of the FRED bug, opposite direction); (2) both EIA fetchers built the sort query parameter as JSON.stringify(...), but EIA Open Data API v2 takes PHP-style brackets and silently 403s on JSON; (3) EIA serialises numeric observations as strings and the fetchers passed the string straight through to the Zod schema. All three fixed.
  • feat(tool): expose EIA energy outlook + petroleum status as economy tools (b50f054)
    • Two new AI tools — economyEnergyOutlook (STEO, with observed + forecast rows) and economyPetroleumStatus (weekly Petroleum Status Report). EIA endpoints route under /commodity/* upstream (OpenBB classifies oil/gas as commodity data) but conceptually they're macro, so the tool layer surfaces them in the economy namespace alongside FRED. createEconomyTools now takes both economyClient (FRED) and commodityClient (EIA); the AI / wire asymmetry is documented in the file header.

2026-05-07 — Trading: ephemeral UTAs, three bug fixes, Phase 3 revert

  • fix(mock): allow externalTrade SELL to open short positions (0e09127)
    • The over-strict guard rejected covered-call's short-call leg. externalTrade (user-on-exchange semantics) now allows SELL-to-open; placeOrder / _applyFill retain the strict guard for Alice-initiated orders.
  • feat(uta): ephemeral test UTA lifecycle (boot purge + manual destroy) (ea8e1d2)
    • New ephemeral?: boolean flag on utaConfigSchema, gated to the mock-simulator preset. Boot path purges all ephemeral entries (config row + data/trading/<id>/) before UTAManager init. DELETE /api/trading/config/uta/:id also wipes data dir for ephemeral UTAs. Closes the structural gap that let test residue pollute cost-basis across sessions.
  • fix(uta): wallet bootstrap respects broker-reported avgCost (ab51371)
    • _reconcileWalletPositions was hardcoding the synthesized reconcileBalance op's price to position.marketPrice, destroying broker-reported avgCost on first sight (Mock externalTrade, Alpaca, IBKR). Now prefers position.avgCost when non-zero, falls back to marketPrice only when the broker has nothing — CCXT spot path unchanged (broker fakes avgCost = mark anyway), Mock/Alpaca/IBKR newly respected.
  • revert(ccxt): drop canonical localSymbol; wire format is broker-native identity (afddd41)
    • Phase 3 of IBKR-as-truth (commit 4bcf2f4) inserted a normalization step that rewrote CCXT's wire symbol BTC/USDT:USDTBTC-PERP before storing it as Contract.localSymbol. That collapsed the :settle suffix and broke aliceId uniqueness within a single Bybit UTA (USDC-margined vs USDT-margined ETH perp became indistinguishable).
    • Architectural correction: IBKR-as-truth = taxonomy, not identity. IBKR's Contract type (secType union + per-secType fields) is canonical. Contract.localSymbol is broker-defined per broker — IBKR uses conId, CCXT uses unified wire symbol, Alpaca uses ticker. The system uniqueness primitive is aliceId = "{utaId}|{nativeKey}" where nativeKey comes from each broker's getNativeKey. We never normalize across brokers.
    • Phase 1 / 2 / 4 (position math, buildContract validation funnel, strict SecType union) are taxonomy work — kept. Phase 3 was identity work — reverted.

2026-05-07 — Workflow / CLAUDE.md

  • docs(claude): drop worktree convention; point parallel work at cloud (b6f7acc)
    • Local worktrees are heavyweight for OpenAlice (per-worktree pnpm install, data/ copying, port juggling); cloud Claude sessions are the right concurrency primitive for projects with stateful working trees.

TODO bookkeeping

  • docs(todo): cost-basis bootstrap overwrites broker avgCost (f294217)
  • docs(todo): drop cost-basis bootstrap entry — fixed in ab51371 (0239694)
  • docs(todo): IBKR getNativeKey may collide on option-chain fanout (15fdba8)
    • IBKR's symbol/localSymbol aren't reliably unique — conId is the actual primary key. Audit needed for IBKR + Alpaca + LeverUp + Bybit-direct getNativeKey implementations.

Full commit log

b50f054 feat(tool): expose EIA energy outlook + petroleum status as economy tools
1182dd9 fix(eia): repair credential prefix + sort syntax + value coercion
86233d6 fix(ui): MarketDataPage hints — drop unimplemented providers, refresh dead URLs
0c7865e feat(tool): expose FRED via three economy tools (search/series/regional)
15fdba8 docs(todo): IBKR getNativeKey may collide on option-chain fanout
afddd41 revert(ccxt): drop canonical localSymbol; wire format is broker-native identity
a86612c fix(market-data): repair FRED end-to-end + add market-data e2e harness
0239694 docs(todo): drop cost-basis bootstrap entry — fixed in ab51371
ab51371 fix(uta): wallet bootstrap respects broker-reported avgCost
f294217 docs(todo): cost-basis bootstrap overwrites broker avgCost
ea8e1d2 feat(uta): ephemeral test UTA lifecycle (boot purge + manual destroy)
0e09127 fix(mock): allow externalTrade SELL to open short positions
001c16f Merge remote-tracking branch 'origin/master' into dev
b6f7acc docs(claude): drop worktree convention; point parallel work at cloud
1df7060 docs(claude): branch convention for invasive refactors

Test plan

  • npx tsc --noEmit clean (root + ui workspaces)
  • pnpm vitest run — 75 files / 1463 tests passing (was 1454; +9 EIA tool spec cases)
  • pnpm vitest run --config vitest.e2e.config.ts src/domain/trading/__test__/e2e/ — 12 files / 64 trading-e2e tests passing, 23 skipped (live-broker preconditions)
  • pnpm vitest run --config vitest.e2e.config.ts src/domain/market-data/__test__/e2e/ — FRED + EIA both green against live APIs (5 FRED + 3 EIA cases)
  • After merge: smoke-test bybit-main / okx-test cost basis renders correctly. Heads-up: 16 commit.json aliceId entries on these UTAs from yesterday's Phase-3 window become orphan and re-bootstrap via reconcileBalance at current markPrice on next getPositions. Today's window was exploratory testing, no real cost-basis history lost.
  • After merge: hit FRED + EIA tools from an Agent SDK session — economyFredSearch / economyFredSeries / economyFredRegional / economyEnergyOutlook / economyPetroleumStatus should return real data without credential errors.

🤖 Generated with Claude Code

Ame and others added 15 commits May 7, 2026 10:51
Add a sibling subsection to the rolling dev → master rules covering
when to break from `dev` and run an isolated branch off `master`
instead. Default stays "everything → dev"; the isolation case is
narrow but recurring (refactors that touch shared types / interfaces
every broker implements, multi-day schema work, etc.).

Codifies what we just discussed on PR #168 close-out: branch from
freshly-merged master, rebase against master during the refactor,
PR straight to master skipping dev, and have dev absorb the merge
afterward via `git merge origin/master`. Adds the matching decision
rule for sessions starting *after* such a refactor lands so they
don't accidentally branch off stale dev.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The local worktree dance (per-worktree pnpm install, data/ copying,
port discipline) costs more than it saves on a project this size.
Cloud Claude sessions are the right concurrency primitive here —
each gets a clean sandbox and returns a PR without disturbing the
local dev server / data directory.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
externalTrade models the user manually trading on the exchange app
— opening a short (covered call's call leg, naked put, perp short)
is a legitimate user action there. The previous guard rejected any
SELL without an existing position, which blocked all short-open
flows in fixture-based testing.

Distinct from placeOrder / _applyFill — those still reject
SELL-without-position because Alice shouldn't silently flip a SELL
intent into a short open. The asymmetry is intentional: external
events let the user do whatever a real exchange allows; Alice's
own orders go through stricter intent verification.

Found while running a covered call scenario through the simulator
console — first concrete coverage gap surfaced by the manual fixture
testing loop.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mock simulator UTAs were registered through the same persistence path
as real broker accounts (Alpaca/IBKR/Bybit) — config row in
accounts.json + persistent commit history under data/trading/<id>/.
That meant test residue from one session polluted cost-basis
calculations in the next: 7 reconcileBalance ops on AAPL accumulated
over multiple QA sessions, weighted-average-cost the long-since-stale
$198 instead of the actual trade price.

Add ephemeral as a first-class UTA flag:

- utaConfigSchema: ephemeral?: boolean, gated by a refine check that
  rejects ephemeral: true on any non-mock-simulator preset (would
  silently destroy real broker history at next boot).

- Boot path (main.ts): purgeEphemeralUTAs runs before UTAManager
  initializes anything. For each ephemeral entry it wipes
  data/trading/<id>/ and removes the row from accounts.json, then
  the manager initializes only the survivors. Each session starts
  ephemeral mocks from a clean slate.

- POST /api/trading/config/uta: accepts ephemeral: true in the
  payload (passes through utaConfigSchema's refine).

- DELETE /api/trading/config/uta/:id: when the deleted UTA was
  ephemeral, also wipes data/trading/<id>/. For real brokers
  delete-from-config still preserves commit history (the audit
  trail outlives the connection); only ephemeral test UTAs are
  fully erased.

Closes the "fixture import/export and persistence" line item from
the simulator-console v2 backlog by establishing the test-account
lifecycle that fixture-based testing depends on.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surfaced via simulator covered-call test (see fix log: 0e09127 +
ea8e1d2 in same session). The reconcileBalance + git WAC pipeline
that solves CCXT spot's faked avgCost generalized over-aggressively
and now destroys correct avgCost reported by brokers that actually
know it (Mock, Alpaca, IBKR).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`_reconcileWalletPositions` was hardcoding the synthesized
`reconcileBalance` op's price to `position.marketPrice`, ignoring
whatever avgCost the broker had already supplied. The mark-bootstrap
behavior was designed for CCXT spot — where the broker fakes
`avgCost = markPrice` because it has no order history — but it
generalized over-aggressively and destroyed correct cost basis on
brokers that DO observe real fill prices.

Concrete case (covered call surfaced 2026-05-07): MockBroker
externalTrade records BUY 100 AAPL @ $148.50 → broker.avgCost is
$148.50. Mark moves to $152. UTA pipeline ran:

  recordReconcile({ markPrice: position.marketPrice })  // = $152

Result: avgCost overwritten to $152, unrealizedPnL = 0, the +$350
real gain disappeared from the UI.

Fix: prefer `position.avgCost` when non-zero, fall back to
`marketPrice` only when the broker has nothing. Behavior matrix:

  - 'broker' avgCostSource: filtered out before this loop, no-op
  - 'wallet' + broker.avgCost > 0 (Mock externalTrade, future
    CCXT-with-fetchMyTrades): bootstrap at broker.avgCost — fixes
    the cost-basis-destruction bug
  - 'wallet' + broker.avgCost == 0 or missing: bootstrap at
    markPrice — preserves the CCXT-spot fallback path
  - 'wallet' + CCXT spot today (avgCost == markPrice by
    construction): bootstrap at avgCost == bootstrap at markPrice,
    behavior unchanged

Verified end-to-end through the simulator covered-call scenario.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
FRED was completely broken on the SDK path. Three independent failures
in the credential chain had to misalign at the same time for the request
to fail without a clear error — and they all did:

- src/webui/routes/config.ts test-provider entry referenced a
  nonexistent `fred` provider; the registry has `federal_reserve`.
- credential-map's single mapping table sent `fred_api_key` to both
  paths, but the in-process opentypebb SDK auto-prefixes declared
  credentials with the provider name (Provider constructor at
  packages/opentypebb/src/core/provider/abstract/provider.ts:54-59),
  so it actually expects `federal_reserve_api_key`.
- getFredApiKey only looked for `fred_api_key` / `api_key`, missing
  the prefixed form even when the credential made it through.

On top of that, two data-quality bugs:
- fetchFredSeries defaulted to `sort_order=asc`, so any limit-only
  query returned 1946-era observations (GDP `limit=5` first row was
  1946-01-01, value `.`).
- fredRegionalApi had three independent errors in one helper: wrong
  base URL (`/fred/geofred/...` 404s; GeoFRED is at `/geofred/...`),
  wrong query param name (`series_group` should be `series_id`), and
  wrong response-parse path (`data.data` should be `data.meta.data`).

Fixes:

- src/webui/routes/config.ts: test-provider fred entry now uses
  `provider: 'federal_reserve'` + `credField: 'federal_reserve_api_key'`.
- src/domain/market-data/credential-map.ts: split into httpKeyMapping
  (legacy Python sidecar HTTP path, `fred -> fred_api_key` — the 5
  openbb-api clients still call buildCredentialsHeader and need this)
  and sdkKeyMapping (in-process opentypebb, `fred -> federal_reserve_api_key`).
  Public signatures unchanged.
- packages/opentypebb fred-helpers.ts: getFredApiKey prefers the
  prefixed form with legacy fallbacks; fetchFredSeries defaults to
  desc and reverses to ascending before returning so date-merge
  consumers keep their shape; new GEOFRED_BASE constant + optional
  base param on buildFredUrl; fredRegionalApi uses series_id and
  parses data.meta.data.
- packages/opentypebb fred-regional standard model: schema description
  reflects series_id semantics.

Tests — the structural piece. market-data had zero e2e specs while
trading has 14 (one per broker, real paper accounts). bbProvider specs
cover only the inner fetcher slice and miss the credential-mapping +
route-wiring layers where the community-reported "fred 配不下来"
failure modes actually live. Filling that gap now while the FRED
bug is fresh:

- src/domain/market-data/__tests__/credential-map.spec.ts (10 unit
  cases, CI main suite): pins the two-table contract so a future
  merge attempt fails loudly.
- src/domain/market-data/__test__/e2e/{setup,market-data.e2e.spec}.ts:
  first market-data e2e harness. Mirrors the trading e2e pattern
  (lazy-singleton TestApp + Hono app.request without listening port)
  but lighter (no isPaper guard — read-only by construction). Five
  FRED cases hit real webui routes, reproducing the actual
  user-facing failure path. Future fmp/yfinance/oecd describe blocks
  drop in cheaply.

Verified end-to-end against the live FRED API: GDP $31,856B (2026-01-01),
UNRATE 4.3%, California per-capita personal income $85,518.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…e identity

Phase 3 of the IBKR-as-truth refactor (commit 4bcf2f4) inserted a
normalization step that rewrote CCXT's wire symbol "BTC/USDT:USDT" into
"BTC-PERP" before storing it as Contract.localSymbol. That violated the
actual architecture:

  aliceId = "{utaId}|{nativeKey}"
  nativeKey = broker.getNativeKey(contract)

aliceId is the system-level uniqueness primitive. nativeKey is whatever
the broker considers its primary key — IBKR uses conId, CCXT uses
unified wire symbol, Alpaca uses ticker, etc. We never normalize across
brokers; each broker chose its identity scheme for product reasons that
shouldn't be erased.

The :settle suffix in CCXT's wire format encodes margin variant. Two
distinct perp products (USDC-margined vs USDT-margined ETH) collapsed
to the same canonical "ETH-PERP", which propagated through getNativeKey
and broke aliceId uniqueness within a single Bybit UTA.

What "IBKR-as-truth" actually means stays:
  - Phase 2: buildContract / assertContract — structural validation
  - Phase 4: strict SecType union — taxonomy
Each broker's localSymbol carrier remains broker-native.

Changes:
  - ccxt-contracts.ts: drop canonicalLocalSymbol + ccxtExpiryToCanonical;
    marketToContract emits localSymbol = market.symbol (wire format).
    KEEP this session's FUT/OPT field derivation (orthogonal to identity)
  - CcxtBroker.ts: drop canonical import; searchContracts indexes via
    direct markets[localSymbol] lookup; resolveNativeKey uses single
    direct lookup. KEEP this session's resolveContractSync defensive
    guards (orthogonal to identity)
  - contract-ext.ts: doc rewritten to describe per-broker uniqueness
    schemes (IBKR conId, CCXT wire, Alpaca symbol, Mock)
  - Tests: spec assertions and 5 e2e files updated to wire format. The
    `secType === 'CRYPTO_PERP'` heuristic added during this session
    stays — it's a more semantic check than localSymbol.includes()

Persisted-state impact: 16 commit.json aliceId entries on bybit-main +
okx-test (from today's Phase 3 window) become orphan and re-bootstrap
via reconcileBalance at current markPrice on next getPositions. Today's
activity was exploratory — no real cost-basis history lost.

Out of scope (filed in plan as follow-up): IBKR getNativeKey should
return String(conId) instead of localSymbol; non-CCXT brokers' nativeKey
schemes deserve a similar audit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surfaced during the Phase-3 revert (afddd41) when articulating
per-broker uniqueness schemes. IBKR's symbol/localSymbol aren't
reliably unique — conId is the actual primary key. Audit + fix
across IBKR / Alpaca / LeverUp / Bybit-direct brokers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The FRED HTTP route + executor path were repaired in a86612c, but
nothing in src/tool/ wraps them — the SDKEconomyClient class has been
exported all along while no factory ever instantiated it, so AI agents
have never been able to read FRED data. Closes that gap and pins the
provider name at the tool boundary so the AI never has to know about
the federal_reserve provider naming.

Three Vercel AI tools, all routed through EconomyClientLike with
provider: 'federal_reserve' baked in:

- economyFredSearch    — keyword → series_id discovery
- economyFredSeries    — observations for one or more series_ids,
                         comma-separated for date-merged multi-series
- economyFredRegional  — state-level cross-section for a regional series
                         (e.g. WIPCPI for per-capita income by state)

Wiring follows the equity/crypto pattern: branch on backend, both
SDKEconomyClient and OpenBBEconomyClient satisfy EconomyClientLike,
register once via toolCenter under the 'economy' namespace.

Tests at src/tool/economy.spec.ts mirror trading.spec.ts — mock the
client surface, verify the three things that silently break when a
domain function becomes an AI tool:

  1. Schema validation (zod): rejects missing required, rejects
     non-positive limit, rejects non-integer limit
  2. Passthrough: args land at the client unchanged + provider is
     pinned + optional fields omitted (not sent as undefined)
  3. Errors propagate (no swallowing)

These run in the CI main suite (`pnpm test`) — unlike the live e2e
specs which only run under `pnpm test:e2e`. So a future schema
mismatch (LLM passes string where number expected, etc.) fails fast
in CI, not in the field.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… dead URLs

Audited the 8 entries in ALL_PROVIDERS and the parallel test-provider
table at webui/routes/config.ts:156-165. Several were stale or pointing
at services we don't actually implement on the SDK backend.

Removed (no SDK provider; Test would 500 with "Provider not found"):

- nasdaq: Nasdaq Data Link is not registered in
  packages/opentypebb/src/providers/. The pre-existing comment at
  domain/market-data/equity/symbol-index.ts:13 already flagged it
  ("intrinio, nasdaq, tradier 暂未接入") but the form still offered
  it. Test button was always going to fail.
- tradingeconomics: same problem — no SDK provider. The TE Python
  upstream is also paid-only with no free tier worth advertising.

Refreshed (URL pointed at retired or marketing pages):

- bls hint: registrationapps.bls.gov/bls_registration → data.bls.gov/
  registrationEngine/. The old host is what the BLS site used to redirect
  through; OpenBB upstream's instructions field already points at the
  new path, we just hadn't followed.
- econdb hint: "Optional" → "Required (free signup)". Anonymous API
  calls now return 401 (verified live), the free-but-keyless era is
  over. Status quo would have users seeing inexplicable 401s for
  EconDB-routed economy queries.
- eia hint: eia.gov/opendata → eia.gov/opendata/register.php. Direct
  registration link instead of the index page.
- fred hint: fredaccount.stlouisfed.org/apikeys → the standard "fred.
  stlouisfed.org → My Account → API Keys" flow. The deep link works
  for already-signed-in users but bounces unauthenticated visitors;
  matches the OpenBB upstream onboarding text now.

Widened (descriptions undersold what the provider actually returns):

- intrinio: "Options snapshots, equity data." was misleading — Intrinio
  fetchers cover equity quotes/historical/info, financials (income/balance/
  cash flow), ratios, calendars, news, ETFs. Now: "Equities, ETFs,
  fundamentals, news, options snapshots."
- fmp: added "commodity" — FMP's CommoditySpotPrice path is wired and
  tested (fmp.bbProvider.spec.ts has gold/silver/brent), it was just
  missing from the user-visible blurb.

No behavior change for working providers; users with stale nasdaq /
tradingeconomics keys in their config still keep them on disk (the
schema entries in core/config.ts and the credential-map tables are
untouched), they just disappear from the form. Re-introduce when the
underlying SDK providers actually exist.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EIA went silently broken in three independent places. Surfaced when a
new EIA API key got plugged into the same /test-provider button that
caught the FRED bug, except this provider's failure mode was layered.

1. providers/eia/index.ts declared `credentials: ['eia_api_key']` —
   the already-prefixed form. The Provider constructor at
   abstract/provider.ts:54-59 unconditionally prepends the provider
   name to declared credentials, so the runtime credential field
   became `eia_eia_api_key` (double prefix). Symmetric to the FRED
   bug from a86612c, just from over-correcting instead of from a
   provider-name mismatch. Fix: declare `['api_key']` like every
   other provider in the package.

2. Both EIA fetcher models built the `sort` query parameter as
   JSON.stringify([{column, direction}]). EIA Open Data API v2 takes
   PHP-style bracket syntax (`sort[0][column]=period&sort[0][direction]=desc`)
   — the JSON form is silently rejected with 403 on every endpoint
   that uses sort. Verified live: same key + URL, swap the param
   shape, request goes from 403 to 200.

3. EIA serialises numeric observations as strings ("75.10") with null
   only for missing periods. Both fetchers passed `obs.value` straight
   through to the Zod schema, which expected number — so even after
   fixes 1+2 every row failed validation with "Expected number, received
   string". Fix: typeof check + parseFloat, NaN-skip the malformed ones.
   Updated the local response interface to admit string|number|null
   instead of lying about it being number.

Tests:

- src/domain/market-data/__test__/e2e/market-data.e2e.spec.ts: added
  an EIA describe block mirroring the FRED structure. Three live cases
  (test-provider button + STEO + petroleum status), each pinned to
  catch the specific bug it was originally hidden behind. Note: EIA
  endpoints sit under /commodity/* (OpenBB upstream classification),
  not /economy/*, and need explicit ?provider=eia because the commodity
  asset class default is yfinance.

Verified live with the user's key:
- STEO crude oil price: 120 monthly rows, latest 2027-12 = $68
  (forecast=true flag working — observed rows go through 2026-04ish,
  forecast extends ~2 years out)
- Crude oil stocks weekly: 260 rows, latest 2026-05-01 = 457,182
  thousand barrels (~457M, sane US commercial inventory level)
- Refinery utilization: 90.1%, typical seasonal high

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ools

Two new AI tools under the economy namespace, completing the FRED + EIA
macro surface for AI agents:

- economyEnergyOutlook   — EIA STEO monthly observations + forecasts
                           (oil/gas price, production, consumption)
- economyPetroleumStatus — EIA Weekly Petroleum Status Report
                           (crude/gasoline/distillate stocks, refinery
                           utilisation, US crude production)

EIA endpoints route under /commodity/* on the wire (OpenBB upstream
classifies oil/gas as commodity data) but conceptually they're macro,
so they belong with FRED in the economy tool namespace from the AI's
perspective. createEconomyTools now takes both economyClient (FRED) and
commodityClient (EIA) — the asymmetry between AI namespace and HTTP
route is documented in the file header.

Wiring:

- CommodityClientLike: added getPetroleumStatus / getEnergyOutlook
  declarations (both SDK + OpenBB-API client classes already implement
  them, just weren't on the interface). OpenBBCommodityClient gets the
  same `as unknown as CommodityClientLike` cast we use for the legacy
  HTTP path elsewhere — its untyped Promise<Record<string, unknown>[]>
  doesn't satisfy the typed contract, but the legacy path is being
  retired anyway.
- main.ts: createEconomyTools(economyClient, commodityClient).

Tests:

- economy.spec.ts: split mocks into makeMockEconomyClient +
  makeMockCommodityClient. Added 9 cases covering schema (zod enum
  rejects unknown category, missing category), passthrough (provider
  pinned to 'eia', date range forwarded), error propagation, and
  client-isolation (EIA tools must not touch economyClient and vice
  versa). The cross-client isolation tests exist because the file
  composes two clients — easy to wire the wrong one when adding a tool.

Verified live with the user's EIA key:
- economyEnergyOutlook crude_oil_price: 120 monthly rows, last
  observed 2026-05 = $112/Barrel, forecast tapers to $68 by 2027-12
- economyEnergyOutlook natural_gas_price: 120 rows, 2027-12 = $4.14/MMBtu
- economyPetroleumStatus crude_oil_stocks: 260 weekly rows, latest
  2026-05-01 = 457M barrels (typical commercial inventory level)
- economyPetroleumStatus refinery_utilization: 90.1%, gasoline
  stocks 219.8M barrels — both consistent with seasonal norms

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@luokerenx4 luokerenx4 merged commit 9a2d916 into master May 8, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant