Market Data Service

Market Data Service

The Market Data Service is the real-time quote ingestion, order book aggregation, and streaming distribution boundary of the Emporia Trading Platform. It owns source market data connections (Simulated, Alpaca IEX, FIX Simulators), cross-venue XOSR composite book aggregation, SSE streaming for the React dashboard, and high-frequency gRPC streaming interfaces.

---

📌 Service Specifications

• Directory: market-data-service
• HTTP/SSE Port: 8084
• gRPC Port: 50551 (marketdataservice.MarketDataService)
• Database Boundary: Stateless (Maintains in-memory quote caches & level-2 depth books).
• Primary Roles:
  • Ingest real-time market data from multiple source providers.
  • Construct composite Level-2 order books (XOSR) combining depth across lit venues.
  • Conflate continuous quote streams to protect browser and gRPC subscribers from queue backpressure.
  • Expose REST snapshot, SSE event stream, and gRPC streaming interfaces.

---

🏗️ Architecture & Component Flow

flowchart TD
    subgraph Sources ["Market Data Sources"]
        Sim["Simulated Market Data Provider (5-level synthetic)"]
        Alpaca["Alpaca IEX Provider (WebSocket + REST snapshots)"]
        FixSim["FIX Simulator Provider (gRPC Bidirectional Sources)"]
    end

    subgraph CoreMDS ["Market Data Service Core"]
        Sim --> ProviderDriver{"MARKET_DATA_PROVIDER"}
        Alpaca --> ProviderDriver
        FixSim --> ProviderDriver

        ProviderDriver --> Aggregator["QuoteAggregator (Composite XOSR Books)"]
        Aggregator --> StreamSvc["MarketDataStreamService (Conflated Streams)"]
    end

    subgraph OutboundInterfaces ["Delivery Interfaces"]
        StreamSvc -->|SSE /api/market-data/stream| ReactUI["React 19 Dashboard (:3001)"]
        StreamSvc -->|gRPC :50551 Connect/Subscribe| AlgoClients["Algorithmic Execution Clients"]
        Aggregator -->|REST /api/market-data/quotes| RestClients["REST Clients"]
    end

Loading

---

🔌 Delivery Interfaces

Interface	Path / Port	Purpose
REST Snapshot	GET :8084/api/market-data/quotes?listingIds=...	Batch top-of-book quotes for diagnostics
REST Depth	GET :8084/api/market-data/{listingId}/depth	Complete Level-2 order book snapshot for one listing
SSE Stream	GET :8084/api/market-data/stream?listingIds=...	Continuous, conflated Server-Sent Event quote stream for UI
gRPC Stream	:50551 marketdataservice.MarketDataService	Streaming gRPC Connect & Subscribe interface
Prometheus	GET :8084/actuator/prometheus	Real-time stream throughput and provider health metrics

---

⚙️ Provider Drivers

Set environment variable MARKET_DATA_PROVIDER to select the active ingestion driver:

1. simulated (Default): Generates deterministic 5-level order books with random walk price simulation for local development.
2. alpaca-iex: Connects to Alpaca's real-time IEX WebSocket stream and snapshot REST API for US equity quotes and trades.
3. fix-simulator: Establishes bidirectional gRPC connections to FIX protocol simulator replicas configured via FIX_SIMULATOR_CONNECTIONS (e.g. XNAS=nasdaq-a:50051|nasdaq-b:50051,XNYS=nyse:50051).

---

📊 Composite Order Books (XOSR)

Static data includes a special XOSR listing for each instrument symbol. When a client requests market data for an XOSR listing, the service resolves all venue listings for that symbol and combines their order books into a consolidated depth view:

• Bids: Sorted strictly from highest price to lowest price.
• Offers: Sorted strictly from lowest price to highest price.
• Venue Traceability: Each depth level retains its source exchangeMic, entryId, and listingId.
• Volume & Last Trade: Volume is aggregated across venues, and the most recent trade across any venue supplies the last price and last size.
• Interruption Resilience: If one venue stream disconnects, streamInterrupted=true is set on the payload without dropping remaining healthy venue feeds.

---

⏱️ Conflation & Backpressure Protection

• Conflation Window: Quotes are conflated at a default interval of 250ms (MARKET_DATA_PUBLISH_INTERVAL).
• Backpressure Protection: Each subscriber queue holds at most the latest pending quote per listing. Slow browser UI clients or gRPC consumers cannot cause unbounded memory growth.
• Heartbeats: A heartbeat payload is emitted every 5 seconds (MARKET_DATA_HEARTBEAT_INTERVAL) to prevent client timeouts during periods of unchanged prices.

---

📈 Observability Metrics

Prometheus metrics available at /actuator/prometheus:

• emporia_market_data_quotes_published_total
• emporia_market_data_quotes_conflated_total
• emporia_market_data_provider_failures_total
• emporia_market_data_subscribers