Why is "Local Mode" (Embedded Woodpecker) restricted in Milvus Cluster environments?

[tinswzy]

Answer:
The restriction on "Local Mode" (embedded Woodpecker) within a cluster environment (like a Milvus Cluster) is a deliberate design choice to ensure data consistency and system stability. Here’s the technical breakdown:

1. Compute Nodes vs. Local Storage
   By default, nodes in a cluster (e.g., StreamingNodes) are treated as compute nodes. Their local disks are typically ephemeral and intended for temporary data or caching, not for persistent, distributed state.

• In a cluster, multiple StreamingNode pods run concurrently.
• In Local Mode, Woodpecker attempts to write logs and state directly to the local disk of the specific pod it is currently running on.

2. The Data Fragmentation Problem during Rebalancing
   The core issue arises from the distributed nature of the cluster. Log streams in Milvus are balanced across different StreamingNodes.

• If StreamingNode A handles a log segment at $T_1$ and then a rebalance or failover occurs, StreamingNode B might take over that same stream at $T_2$.
• Woodpecker on Node B will look for the previous log segments on its own local disk.
• Since those logs only exist on Node A's physical disk, the log chain is broken. This leads to data fragmentation and failed log lookups.

3. The Shared File System Exception
   Technically, "Local Mode" could work in a cluster if, and only if, all StreamingNode pods mount the exact same shared storage (e.g., NFS, CephFS, or GlusterFS).

• Every pod would need access to a unified, shared directory to maintain a consistent state across the cluster.
• However, managing high-performance, low-latency shared file systems in a distributed environment is operationally complex and difficult for most users to maintain correctly.

4. Why we default to Panic
   To protect users from accidentally entering an unstable state where logs are silently lost or inaccessible during rebalancing, Woodpecker explicitly triggers a panic when Cluster Mode is detected with Local storage configured.

Conclusion:
If you are running in Cluster Mode, it is strongly recommended to use a distributed state provider (such as S3 or a dedicated database). "Local Mode" is reserved for single-node or local development environments where storage locality is guaranteed.