StreamForge

StreamForge is a planned configurable platform for ingesting real-time market data, normalizing it into a canonical event model, applying safe declarative transformations, and delivering it to multiple output formats and transports.

The Java 21 backend Maven reactor includes immutable market-data value types, STP v1 codecs, a deterministic tick simulator, a local TCP generator-to-parser path, streaming JSONL and CSV output sinks, and a local in-process pipeline runner for STP binary, JSONL, and CSV files. The React/Vite dashboard reads pipeline definitions and schema catalog entries from the versioned control-plane API, provides guided pipeline creation and safe field mapping, and shows bounded local run health and dead-letter summaries.

The local pipeline runner can also quarantine record-level failures to a staged JSONL dead-letter file with deterministic IDs and opt-in, bounded payload capture. Distributed dead-letter handling is not implemented.

The control plane is a separate Spring Boot service that persists validated, credential-free pipeline definitions and revisions in PostgreSQL, and runs finite local revisions with bounded live monitoring, safe dead-letter summaries, SSE updates, and managed output downloads. HTTP runs accept relative input and output paths only: inputs stay beneath a configured input root, while outputs and dead letters stay beneath a server-owned artifact root. It does not authenticate users. See docs/mvp-demo.md for the verified local monitoring walkthrough and backend/control-plane/README.md for service startup.

Run the self-checking MVP demo with Docker available:

./scripts/run-mvp-demo.sh

Verify the backend from the repository root:

./backend/mvnw -f backend/pom.xml verify

Generate a deterministic binary STP fixture on a POSIX shell:

./backend/mvnw -f backend/pom.xml -pl tick-simulator -am package
java -cp backend/tick-simulator/target/classes:backend/stp-protocol/target/classes:backend/common-model/target/classes \
  io.streamforge.ticksimulator.TickSimulatorCli \
  --seed 5 --symbols AAPL,MSFT --count 100 --output ticks.stp

Local TCP Demo

Build both local components first:

./backend/mvnw -f backend/pom.xml -pl tick-simulator,parser-engine -am package

Terminal 1 starts a server that exits after serving its first finite client:

java -cp backend/tick-simulator/target/classes:backend/stp-protocol/target/classes:backend/common-model/target/classes \
  io.streamforge.ticksimulator.TickTcpServerCli \
  --host 127.0.0.1 --port 9010 --seed 5 --symbols AAPL,MSFT --count 10 --rate 0

Terminal 2 connects, incrementally decodes the STP frames, and prints each parsed event:

java -cp backend/parser-engine/target/classes:backend/stp-protocol/target/classes:backend/common-model/target/classes \
  io.streamforge.parserengine.StpParserCli \
  --host 127.0.0.1 --port 9010 \
  --report-sequence-integrity --source demo-session

The classpath separators in these examples are for POSIX shells.

Install dashboard dependencies from the repository root:

npm --prefix web-dashboard ci

Run the dashboard development server:

npm --prefix web-dashboard run dev

The dashboard uses VITE_CONTROL_PLANE_API_URL, defaulting to /api/v1. During local Vite development, that relative path is proxied to http://localhost:8080; override the proxy target with VITE_CONTROL_PLANE_PROXY_TARGET when needed. See web-dashboard/.env.example.

Run dashboard checks:

npm --prefix web-dashboard run lint
npm --prefix web-dashboard run test
npm --prefix web-dashboard run build

Validation

Run every local quality check from the repository root:

make check

Run one area at a time:

make backend-check
make web-check

backend-check runs Maven Wrapper verification, including Java formatting enforcement. web-check runs npm ci, Prettier format checking, ESLint, Vitest in non-watch mode, and the Vite production build. GitHub Actions runs these same targets with Maven and npm caches.

Remove generated build outputs with:

make clean