[RFC]OpenViking 多存储介质 V2版本:分区化元数据与异步备份架构/OpenViking Multi-Write V2: Partitioned Metadata and Async Backup Architecture

[baojun-zhang]

本 RFC 为双语版本（英文 + 中文）。中文版在前，英文版在后。
This RFC is bilingual (English + Chinese). Chinese version first, English version below.

---

中文版 / Chinese Version

🔍 待评审事项

#	评审项	所在章节	需要决策的问题
1	V2 元数据只存 primary	§2	是否接受多存储介质元数据可靠性等同 primary 可靠性？
2	固定异步多存储介质语义	§3	是否接受写入 ACK 不等待事件持久化或 backup 追平？
3	分区模型边界	§4	是否接受本期固定 vbucket 和物理分区数，动态扩容留到 P2？
4	旧同步能力废弃	§6	是否接受删除 sync-status / sync-retry，并兼容解析旧配置但不生效？

以下内容已在设计中收敛，本 Discussion 只保留整体架构和关键取舍：目录级 sidecar 淘汰、Checkpoint + Segment、provider 抽象、目录事件按状态通知处理。

---

Context

当前 OpenViking 多存储介质元数据使用“每目录一个 sidecar + 全局 next_seq”的模式。

这个模式在大规模文件场景下有明显上限：

问题	影响
每目录 sidecar	目录数越多，元数据文件越多。1 亿文件、每目录 10 个文件时，sidecar 可达到千万级。
目录级整文件读改写	热点目录下任一文件变化都会改整份目录 sidecar。
全局 next_seq	所有写入争用同一个发号点。
恢复按目录组织	retry、status、恢复都依赖目录模型，无法做到增量可控。

V2 的目标是把同步状态从“目录维度”改成“account 内部分区维度”。

V1: directory sidecar + global next_seq
V2: account partitions + Checkpoint + Segment

---

1. 目标与非目标

目标：

1. 支持 10^9 文件量级的多存储介质元数据管理。
2. 淘汰目录级同步 sidecar。
3. 去掉单全局日志和单全局 checkpoint。
4. 将同步状态和 retry 状态统一到分区化的 Checkpoint + Segment。
5. 让恢复和 backup catch-up 只处理增量，不扫描全量文件。
6. 保持普通 Python / HTTP / CLI 文件读写接口不变。

非目标：

1. 不提供跨 partition 的全局单调事件顺序。
2. 不向普通文件 API 暴露 partition 或 vbucket。
3. 不防御 primary 灾难性且不可恢复的丢失。
4. 不在本期实现动态扩容、停服扩容、缩容或 vbucket 重平衡。

---

2. 关键约束

V2 的第一条边界是：多存储介质元数据只存 primary。

这些路径不复制到 backup：

/local/_system/.multiwrite.json
/local/{account}/_system/partitions/**

这意味着：

1. 多存储介质元数据可靠性等同 primary 可靠性。
2. backup 只追平普通 account 数据和 mount 级系统数据。
3. primary 成功后，多存储介质只异步提交事件，不延长业务 PathLock。
4. 多存储介质失败只记录日志，不改变 primary 返回值。
5. 进程退出时，尚未持久化进 Segment 的内存事件允许丢失。

V2 追求的是：primary 主流程不被多存储介质拖慢，backup 最终收敛。

---

3. 架构概览

V2 把一个 account 的多存储介质元数据拆成固定物理分区。每个 partition 使用两类数据：

数据	作用
Segment	追加写入增量事件，如 Write、Remove、RemoveTree、MoveTree。
Checkpoint	保存该 partition 截至某个 seq 的当前文件状态快照。

核心目录形态：

/local/{account}/_system/partitions/
  manifest.json                    # vbucket -> partition 路由
  0/
    segments/manifest.json          # next_seq、segment 摘要、backend_states
    segments/*.ovsg                 # 增量事件流
    checkpoints/manifest.json       # latest_checkpoint 指针
    checkpoints/cp-*/chunk-*.ovcp   # 当前状态快照
  1/
    ...

运行时组件：

组件	职责
FlushWorker	异步消费 primary 成功后的事件，并写入 Segment。
CatchUpWorker	读取 Segment 或 Checkpoint，让 backup 按 primary 当前状态收敛。
Checkpoint Task	把旧 checkpoint 和 sealed segment 折叠成新的当前状态快照。
GC	删除已被 checkpoint 吸收且不再需要的旧 segment / checkpoint。

前台写入流程：

Client
  -> primary 写入成功
  -> 异步提交 PendingEvent
  -> 立即返回 ACK
  -> FlushWorker 持久化 Segment
  -> CatchUpWorker 追平 backup

ACK 不等待事件入队、Segment 持久化、backup 写入或 checkpoint 生成。

---

4. 分区与序列

每个文件使用完整的 mount-relative path 作为 hash 输入：

hash_key = mount_relative_path
vbucket = xxh3_64(hash_key) & (65536 - 1)
partition = routes[vbucket]

关键点：

1. VBUCKETS 固定为 65536，不是用户配置。
2. initial_partitions 只在 account 首次初始化时生效，默认 16。
3. 已有 account 始终使用落盘 routes，不因配置变化重新分区。
4. 动态扩容、缩容和 vbucket 重平衡留到 P2。

V2 不再使用全局裸 u64 序号。每个 partition 有自己的序列空间：

ScopeKey { account_id, partition_id, epoch }
ScopedSeq { scope, seq }

判断某个 backup 是否追平某个状态时，必须先比较 scope，再比较 seq。不同 scope 的 seq 禁止直接比较。

---

5. 目录语义与追平

V2 把事件视为“状态失效通知”，不是历史命令。

普通文件事件：

1. Write(path)：backup 读取 primary 当前内容并写入。
2. Remove(path)：如果 primary 当前不存在，则从 backup 删除。
3. 文件 rename 拆成 Write(new_path) + Remove(old_path)。

目录事件：

RemoveTree(source_path, op_id)
MoveTree(source_path, destination_path, op_id)

目录删除和目录移动不会展开子文件。V2 给每个 partition 写一条 marker，后台按 primary 当前前缀状态对齐 backup。

这样单次目录操作的元数据成本是 O(partition_count)，不是 O(descendant_files)。

CatchUpWorker 每轮按 partition 处理：

1. 读取 backup 的 synced_seq 和当前 tail_seq。
2. 如果后续 Segment 还在，则消费 Segment。
3. 如果 Segment 已被 checkpoint 吸收，则读取最新 Checkpoint。
4. 对普通路径读取 primary 当前状态并写入或删除 backup。
5. 对目录 marker 扫描 primary 当前前缀并对齐 backup。
6. 全部成功后推进该 partition 的 backend_states[backend_id].synced_seq。

---

6. Provider、配置与迁移

MultiWriteProvider 只负责单个 partition 的段存储：

归属	内容
provider	head 事件、next_seq、sealed 段摘要、backend_states
共享 filesystem	sealed .ovsg、Checkpoint、Partitions Manifest

本期 provider：

provider	适用场景	说明
filesystem	默认部署	用 PathLock Exact 锁保护 Segment Manifest 和 head segment。
cache	多实例事件聚合	用 Redis list + Lua 原子推进 head 与 Segment Manifest 状态；sealed segment 仍落共享 filesystem。

用户可感知配置：

配置项	含义	默认值
initial_partitions	单 account 首次初始化时创建的物理分区数	16
checkpoint_interval_secs	checkpoint 时间触发间隔	86400
provider	多存储介质事件持久化 provider	filesystem

V2 固定异步，废弃旧同步状态能力：

1. 删除 system_sync_status、system_sync_retry。
2. 删除 HTTP /api/v1/system/backend/sync-status、sync-retry、/api/v1/system/sync/{path} 等接口。
3. 删除 CLI ov system backend sync-status、ov system backend sync-retry。
4. 旧配置由 Python 层兼容接收，打印 WARN 后在进入 Rust 前删除。

数据迁移使用：

/local/_system/.multiwrite.json

启动行为：

状态	行为
文件不存在	创建 V2/migrating，启动全量导入。
V2/migrating	继续全量导入。
V2/stable	按 V2 正常恢复。

全量导入只把 primary 当前存在的文件按正常 Write 事件写入分区化 Segment。导入中断后重跑允许重复 Write，后续 checkpoint 会自然折叠。

---

7. 风险与边界

风险	边界
primary 不可恢复丢失	V2 多存储介质元数据也丢失；本方案不解决。
primary 成功后、事件持久化前崩溃	该事件可能丢失，需要重建对应 backup。
Redis 被误清	cache provider 的 head / backend_states 可能丢失；不承诺误清后的自动无损恢复。
checkpoint 扫描期间 primary 继续变化	后续 Segment 或 Checkpoint 会继续修正；中间态不保证一致。
目录操作并发	marker 不定义跨 partition 全局顺序，只按 primary 当前状态收敛。
动态扩容	P2 单独设计，修改 initial_partitions 不触发扩容。

---

8. 总结

V2 的核心取舍是：牺牲旧版同步写入和目录级状态接口，换取更简单的 primary 主路径和可扩展的元数据结构。

最终模型：

primary 成功即返回
  -> FlushWorker 写 partition Segment
  -> CatchUpWorker 按 primary 当前状态追平 backup
  -> Checkpoint 折叠当前状态
  -> GC 回收旧 Segment / Checkpoint

请重点评审：

1. 是否接受 primary-only 元数据可靠性边界。
2. 是否接受固定异步语义。
3. 是否接受本期不做动态扩容。
4. 是否接受旧同步接口与配置的废弃方式。

---

---

English Version / 英文版

🔍 Review Items

#	Item	Section	Decision Needed
1	V2 metadata lives only on primary	§2	Do we accept that multi-write metadata durability equals primary durability?
2	Fixed async multi-write semantics	§3	Do we accept that write ACK does not wait for event persistence or backup catch-up?
3	Partition model boundary	§4	Do we accept fixed vbuckets and physical partitions in this phase, leaving dynamic resharding to P2?
4	Legacy sync capability removal	§6	Do we accept removing sync-status / sync-retry while parsing old configs with WARN but ignoring them?

This Discussion only keeps the architecture and key tradeoffs: directory sidecars are removed, metadata becomes Checkpoint + Segment, providers are abstracted, and directory operations become state notifications.

---

Context

The current OpenViking multi-write metadata model is based on “one sidecar per directory + one global next_seq”.

This has clear limits at large file scale:

Problem	Impact
One sidecar per directory	Metadata file count grows with directory count. With 100M files and 10 files per directory, sidecars can reach around 10M.
Directory-level read-modify-write	Any file update in a hot directory rewrites the whole directory sidecar.
Global next_seq	All writes contend on one sequence allocator.
Directory-based recovery	Retry, status, and recovery depend on the directory model and are not bounded incremental streams.

V2 moves sync state from “directory scope” to “account-local partition scope”.

V1: directory sidecar + global next_seq
V2: account partitions + Checkpoint + Segment

---

1. Goals and Non-goals

Goals:

1. Support multi-write metadata management at 10^9 file scale.
2. Remove directory-level sync sidecars.
3. Remove the single global log and single global checkpoint.
4. Unify sync and retry states into partitioned Checkpoint + Segment.
5. Make recovery and backup catch-up incremental, without full filesystem scans.
6. Keep normal Python / HTTP / CLI file read/write APIs unchanged.

Non-goals:

1. No global monotonic event order across partitions.
2. No partition or vbucket details exposed to normal file APIs.
3. No protection against catastrophic, unrecoverable primary loss.
4. No dynamic expansion, offline expansion, shrinking, or vbucket rebalancing in this phase.

---

2. Key Constraint

The first V2 boundary is: multi-write metadata lives only on primary.

These paths are not replicated to backups:

/local/_system/.multiwrite.json
/local/{account}/_system/partitions/**

This means:

1. Multi-write metadata durability equals primary durability.
2. Backups only catch up normal account data and mount-level system data.
3. After primary succeeds, multi-write only submits events asynchronously.
4. Multi-write failures are logged but do not change the primary result.
5. In-memory events not yet persisted into Segment may be lost on process exit.

V2 optimizes for: primary path stays fast, backups eventually converge.

---

3. Architecture Overview

V2 splits each account’s multi-write metadata into fixed physical partitions. Each partition uses two data types:

Data	Purpose
Segment	Append-only incremental events, such as Write, Remove, RemoveTree, and MoveTree.
Checkpoint	Current file-state snapshot for this partition up to a seq boundary.

Core layout:

/local/{account}/_system/partitions/
  manifest.json                    # vbucket -> partition routes
  0/
    segments/manifest.json          # next_seq, segment descriptors, backend_states
    segments/*.ovsg                 # incremental event stream
    checkpoints/manifest.json       # latest_checkpoint pointer
    checkpoints/cp-*/chunk-*.ovcp   # current-state snapshot
  1/
    ...

Runtime components:

Component	Responsibility
FlushWorker	Asynchronously consumes events after primary success and writes Segment.
CatchUpWorker	Reads Segment or Checkpoint and converges backups to primary current state.
Checkpoint Task	Folds the previous checkpoint and sealed segments into a new current-state snapshot.
GC	Deletes old segments / checkpoints that are no longer needed.

Foreground write flow:

Client
  -> primary write succeeds
  -> submit PendingEvent asynchronously
  -> return ACK immediately
  -> FlushWorker persists Segment
  -> CatchUpWorker catches backups up

ACK does not wait for event enqueue, Segment persistence, backup write, or checkpoint generation.

---

4. Partitioning and Sequence

Each file uses the full mount-relative path as hash input:

hash_key = mount_relative_path
vbucket = xxh3_64(hash_key) & (65536 - 1)
partition = routes[vbucket]

Key points:

1. VBUCKETS is fixed to 65536; it is not user-configurable.
2. initial_partitions only applies when an account is first initialized. Default is 16.
3. Existing accounts always use persisted routes and are not repartitioned by config changes.
4. Dynamic expansion, shrinking, and vbucket rebalancing are left to P2.

V2 no longer uses one global bare u64 sequence. Each partition has its own sequence space:

ScopeKey { account_id, partition_id, epoch }
ScopedSeq { scope, seq }

To decide whether a backup has caught up with a state, compare scope first and seq second. Sequences from different scopes must not be compared directly.

---

5. Directory Semantics and Catch-up

V2 treats events as “state invalidation notifications”, not historical commands.

Normal file events:

1. Write(path): backup reads primary current content and writes it.
2. Remove(path): if primary current state does not exist, backup removes it.
3. File rename becomes Write(new_path) + Remove(old_path).

Directory events:

RemoveTree(source_path, op_id)
MoveTree(source_path, destination_path, op_id)

Directory delete and directory move do not expand descendant files. V2 writes one marker per partition, and background workers reconcile backup from primary current prefix state.

This makes metadata cost for one directory operation O(partition_count), not O(descendant_files).

Each CatchUpWorker round handles partitions independently:

1. Read this backup’s synced_seq and current tail_seq.
2. If later Segment data still exists, consume Segment.
3. If Segment was absorbed by checkpoint, read the latest Checkpoint.
4. For normal paths, read primary current state and write or remove backup.
5. For directory markers, scan primary current prefixes and reconcile backup.
6. After success, advance backend_states[backend_id].synced_seq for this partition.

---

6. Provider, Config, and Migration

MultiWriteProvider only owns per-partition segment storage:

Owner	Data
provider	Head events, next_seq, sealed segment descriptors, backend_states
shared filesystem	Sealed .ovsg, Checkpoints, Partitions Manifest

Providers in this phase:

Provider	Use Case	Description
filesystem	Default deployment	Uses PathLock Exact locks to protect Segment Manifest and head segment.
cache	Multi-instance event aggregation	Uses Redis list + Lua to atomically advance head and Segment Manifest state; sealed segments still live on shared filesystem.

User-visible configs:

Config	Meaning	Default
initial_partitions	Physical partition count when an account is first initialized	16
checkpoint_interval_secs	Time-based checkpoint interval	86400
provider	Multi-write event persistence provider	filesystem

V2 is always async and removes legacy sync status management:

1. Remove system_sync_status and system_sync_retry.
2. Remove HTTP /api/v1/system/backend/sync-status, sync-retry, /api/v1/system/sync/{path}, and related APIs.
3. Remove CLI ov system backend sync-status and ov system backend sync-retry.
4. Old configs are parsed by Python, logged with WARN, and removed before entering Rust.

Data migration uses:

/local/_system/.multiwrite.json

Startup behavior:

State	Behavior
Missing file	Create V2/migrating, then start full import.
V2/migrating	Continue full import.
V2/stable	Recover normally as V2.

Full import writes existing primary files into partitioned Segment as normal Write events. If import is interrupted and retried, duplicate Write events are allowed and later checkpoints fold them naturally.

---

7. Risks and Boundaries

Risk	Boundary
Primary is unrecoverably lost	V2 multi-write metadata is lost too; this proposal does not solve it.
Process crashes after primary success but before event persistence	That event may be lost; the corresponding backup may need rebuild.
Redis is manually cleared	For cache provider, head / backend_states may be lost; automatic lossless recovery is not promised.
Primary changes during checkpoint scan	Later Segment or Checkpoint will correct it; intermediate state is not guaranteed.
Concurrent directory operations	Markers do not define a global cross-partition order; workers converge from primary current state.
Dynamic resharding	P2 will be designed separately; changing initial_partitions does not trigger expansion.

---

8. Summary

V2 trades legacy synchronous writes and directory-level status APIs for a simpler primary path and scalable metadata layout.

Final model:

primary succeeds and returns
  -> FlushWorker writes partition Segment
  -> CatchUpWorker converges backups from primary current state
  -> Checkpoint folds current state
  -> GC removes old Segment / Checkpoint

Please review:

1. Whether the primary-only metadata durability boundary is acceptable.
2. Whether fixed async semantics are acceptable.
3. Whether leaving dynamic resharding to P2 is acceptable.
4. Whether the legacy sync APIs and config deprecation path is acceptable.