PRD: CEX canonical Parquet、SHA 缓存与本地回放数据面

[proerror77]

Problem Statement

当前 CEX Collector 将可回放的原始 NDJSON/Zstd 段上传到 OSS；研究或回测再从 OSS 读取原始段或完整 NDJSON。该路径把归档证据层当作热查询层，重复回测会重复下载、解压、校验和解析相同的字节，且没有一个可验证的本地缓存契约。现有 lob-pit-materializer 只生成点时特征 JSONL，不能替代 L2 事件回放输入；把它直接接到 hft-backtest 会改变回测语义。

Solution

保留 OSS 原始 NDJSON/Zstd、collector manifest 与 _SUCCESS 作为不可变证据层。新增经过原始段验证的 canonical replay Parquet 分区和内容绑定 manifest；cache warmer 按指定分区校验 SHA 后原子落入 ACK 本地 ESSD/PVC；hft-backtest 只从本地 canonical Parquet 回放，并按时间/列读取。ClickHouse 仅接收已验证 Parquet 的物化数据用于共享分析和特征查询，不成为原始证据或顺序 LOB 回放的真相源。生产 Collector 上传/磁盘水位与 ACK/PVC/ClickHouse 实际切换均是独立 runtime 合同。

User Stories

1. As a quantitative researcher, I want immutable raw collector segments retained in OSS, so that every canonical partition can be traced to replay-safe source evidence.
2. As a quantitative researcher, I want a verified raw LOB segment selection materialized into a canonical Parquet replay partition, so that repeated backtests avoid raw NDJSON decoding.
3. As an auditor, I want the canonical manifest to bind its Parquet SHA-256, schema, source segment identities, source revision, event coverage, and sequence range, so that a partition cannot be silently substituted.
4. As a researcher, I want PIT feature artifacts kept distinct from replay event artifacts, so that a feature table is never mistaken for an executable order-book tape.
5. As a backtest worker, I want to read only local PVC/ESSD paths, so that ordinary replay is independent of OSS latency and request variability.
6. As a backtest worker, I want requested time windows and required columns projected from Parquet, so that work is proportional to the selected experiment rather than the whole raw archive.
7. As an operator, I want cache staging to verify manifests and bytes before publishing an atomic ready marker, so that partial or corrupted cache entries cannot be replayed.
8. As an operator, I want identical content-addressed cache requests to be idempotent, so that repeated ACK jobs do not redownload valid partitions.
9. As an auditor, I want cache hits to retain the canonical manifest SHA and source revision in backtest output evidence, so that results stay traceable after cache eviction.
10. As an analyst, I want ClickHouse to index canonical partitions and derived features, so that interactive analytics does not scan raw NDJSON.
11. As an analyst, I want ClickHouse excluded from sequential LOB replay truth, so that a query store cannot silently alter event ordering or evidence scope.
12. As a runtime owner, I want the Polymarket uploader/disk-watermark recovery handled under a separate runtime contract, so that data-plane code review does not authorize a production restart or cutover.
13. As a runtime owner, I want ACK PVC and ClickHouse provisioning controlled by a separately named controller and immutable target identities, so that infrastructure creation has explicit rollback and readback.
14. As a maintainer, I want each data-plane behavior delivered in a small independently reversible issue and PR, so that Parquet encoding, cache consumption, analytics materialization, and runtime adoption can be assessed separately.

Implementation Decisions

• Raw OSS .jsonl.zst segments, their collector manifests, and _SUCCESS markers remain immutable evidence and are never overwritten or replaced by Parquet.
• The first canonical replay format represents the exact event stream required by hft-backtest, not point-in-time feature rows. It has a versioned schema, explicit ordered event sequence, event time, and L2 payload needed for replay.
• A materializer verifies every selected raw segment against the collector contract before publishing a SHA-addressed Parquet object and a create-once canonical manifest.
• The canonical manifest is the cache and replay authority: it names the expected local Parquet artifact, SHA-256, source revision, source segment evidence, event-time bounds, sequence bounds, and schema version.
• The cache warmer may use OSS/object-store range reads to fill a cache, but the normal backtest hot path never reads an OSS bucket or FUSE-mounted bucket.
• Cache publication is atomic: download into a private temporary path, hash-check bytes, write the manifest, then create a ready marker. A mismatched existing path fails closed.
• hft-backtest accepts canonical Parquet only from a local path and validates the canonical manifest, Parquet schema, sequence/order and content SHA before replay. It need not re-read raw source segments on every cache hit.
• The existing lob-pit-materializer continues to own PIT feature materialization. Its eventual Parquet output is a separate canonical feature dataset and is not an hft-backtest input.
• ClickHouse materialization reads only manifest-verified canonical Parquet/derived feature partitions. It is optional for replay and must be idempotent by immutable partition identity.
• No object-store client is embedded in the ordinary backtest reader. Object-store access stays in the cache-fill/materialization boundary.
• Polymarket uploader repair, collector restart, disk cleanup, historical reconstruction, ACK PVC provisioning, ClickHouse provisioning, and any live mutation are separate runtime issues and require their own named controller, exact target, rollback, stop rules, and readback.

Testing Decisions

• The canonical-materializer CLI is tested at its public boundary with a replay-safe raw segment fixture, proving a Parquet partition and content-bound manifest are produced and corruption fails closed.
• The cache-warmer CLI is tested at its public boundary with a local source mirror, proving SHA mismatch leaves no ready cache entry and an existing valid entry is idempotent.
• The hft-backtest data seam is tested through config plus replay execution: a canonical local Parquet fixture succeeds without source raw files, while wrong hash, schema, ordering, or sequence fails.
• Existing raw collector manifest/_SUCCESS validation and backtest sequence tests remain prior art; tests assert observable artifacts and rejection behavior rather than private helper calls.
• ClickHouse writer tests use a local contract seam or generated SQL payload and prove immutable partition identity and duplicate handling; no cloud service is required for unit validation.
• Runtime issues require immutable candidate/readback evidence and do not treat CI or a healthy process as completion.

Out of Scope

• Mounting the whole OSS bucket as the backtest data volume.
• Direct OSS range reads as the normal replay path.
• Replacing raw evidence with Parquet, deleting raw segments, or backfilling unavailable history.
• Treating PIT feature rows or ClickHouse tables as sequential LOB replay data.
• Full L3 queue position, partial fills, market impact, venue capacity, or changes to live trading.
• Any Polymarket collector restart, uploader action, disk mutation, ACK/ClickHouse provisioning, or production cutover in this PRD.
• Cleaning unrelated branches, worktrees, or the dirty primary checkout.

Further Notes

• The first implementation target is a narrow Binance Spot/USD-M LOB event-tape vertical slice. Trade-dependent strategy inputs remain rejected until a separate modality contract exists.
• Canonical Parquet and cache artifacts are immutable evidence objects; cache residency itself is disposable.
• GitHub Issues are the authoritative plan. Child issues are published in dependency order and runtime issues remain non-executable without their required operational identities.