Woodpecker: segments with V6 footer (LAC) can't be read after downgrading Milvus 3.0.x -> 2.6.x (failed to decode footer record: buffer too short for record: expected N, got 45) — is downgrade supported?

[Siddeshvr]

Summary

After a cluster running Milvus 3.0.x (Woodpecker WAL, MixCoord + StreamingNode) was moved to a 2.6.22-based build (a downgrade),
StreamingNode WAL recovery fails on many channels with:

failed to read entry: failed to decode footer record: buffer too short for record: expected 55, got 45
failed to read entry: failed to decode footer record: buffer too short for record: expected 57, got 45
failed to read entry: failed to decode footer record: buffer too short for record: expected 65, got 45
failed to read entry: failed to decode footer record: CRC32 mismatch: expected <n>, got <hex>

Channels never get assigned (assignment not found), so CreateCollection blocks until the client times out.
Moving back to a 3.0.x build immediately fixes it (no data changes). We reproduced this on two independent clusters (EKS and AKS).

We believe this is a Woodpecker on-disk format incompatibility (format V6 footer read by a V5-only reader), and would like to confirm:

1. Is downgrading 3.0.x -> 2.6.x supported when Woodpecker is the WAL? (We understand it may not be, but the failure mode is a hang rather than a clear error.)
2. Can the reader detect the footer version instead of assuming a fixed footer size?
3. Is there a documented Woodpecker format/version compatibility matrix per Milvus release?
4. Is there a supported way to recover a cluster that was downgraded (other than rolling forward)?

Symptoms and logs

StreamingNode (2.6.22-line build), per channel:

[WARN] scanner consuming was interrpurted with error, start a backoff ... [channel=by-dev-rootcoord-dml_15:rw@445] [startMessageID=4818/53]
       [error="failed to read entry: failed to decode footer record: buffer too short for record: expected 65, got 45"]
[ERROR] wp readNext msg exception [error="failed to read entry: failed to decode footer record: buffer too short for record: expected 57, got 45"]
[WARN] failed to close wp scanner [error="... failed to decode footer record: CRC32 mismatch: expected <n>, got <hex>"]
[WARN] Failed to get segment reader for compaction ... [error="failed to decode footer record: CRC32 mismatch: expected 500, got 8cbf2f8e"]
[WARN] Failed to recover from storage, starting fresh ... MinioFileWriter
[WARN] auditor maintain the log segment failed ...

MixCoord:

[WARN] handler_client_impl.go ["assignment not found"]
[WARN] balancer_impl.go ["fail to assign channel"] [policy=vchannelFair] [error="... Unavailable, desc = subConn not exist"]

Proxy (client request):

CreateCollection received ... collection=<name> [len(schema)=52]
CreateCollection failed to WaitToFinish ... error="proxy TaskCondition context Done: context canceled"

Observed counts (2.6.22-line build): 39,725 decode errors / 5 min on 9 channels (AKS); 12,644 / 4 min on 13 channels (EKS).
Same data on the 3.0.1-line build: 0 decode errors, 0 assignment not found.

Each stuck channel's recovery scanner startMessageID points at one specific segment (segId/entry), e.g. dml_15 -> 4818/53, dml_13 -> 4881/46.

Evidence for the format mismatch

1. Woodpecker footer format versions differ

• Woodpecker v0.1.1 / commit fdf34b23 (what Milvus 2.6.22 pins): FormatVersion = 5, FooterRecordSize = 36
  (record on disk = 9-byte record header + 36 = 45 bytes).
• Woodpecker v0.1.14 ... v0.1.38 (Milvus 3.0.1 pins v0.1.38): FormatVersion = 6 ("Latest version with LAC support"),
  FooterRecordSizeV6 = 44 (+8-byte LAC; record on disk = 53 bytes).

Sources:

• https://github.com/zilliztech/woodpecker/blob/v0.1.1/server/storage/codec/format.go
• https://github.com/zilliztech/woodpecker/blob/fdf34b23e0d7/server/storage/codec/format.go
• https://github.com/zilliztech/woodpecker/blob/v0.1.14/server/storage/codec/format.go
• https://github.com/zilliztech/woodpecker/blob/v0.1.38/server/storage/codec/format.go
• https://github.com/milvus-io/milvus/blob/v2.6.22/go.mod
• https://github.com/milvus-io/milvus/blob/v3.0.1/go.mod
• Decoder emitting the error: https://github.com/zilliztech/woodpecker/blob/v0.1.1/server/storage/codec/codec.go (DecodeRecord: buffer too short for record: expected %d, got %d)

2. The error values fit "V5 reader on a V6 footer"

The failing reader always reports got 45 (= 9 + 36, the V5 footer record size), while expected varies (55, 57, 65, ...).
A V5 reader that slices the last 45 bytes of a 53-byte V6 footer record starts 8 bytes too late, so the length field it reads is
arbitrary; expected = 9 + garbage_length. When the garbage length is small enough, the slice is long enough and the CRC32 check fails instead
(CRC32 mismatch). Both variants appear in our logs.

3. The failing object in the bucket is a V6 footer

Hexdump tail of <root>/wp/1/4308/footer.blk (258 bytes), one of the segments the 2.6.22-line build could not read:

...  cd00 0000 0600 0000 4654 5201 0400 0000 0000 0000
                ^^^^ ^^^^ ^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^
              version=6 flags=0   FTR\x01     LAC = 4 (int64)

A healthy compacted segment from an earlier date ends with ... 0600 0100 4654 5201 3400 0000 0000 0000
(version 6, flags=compacted, magic, LAC=52). In other words, the footers we see are V6 with a trailing LAC.

4. Which segments fail

All segments failing on the 2.6.22-line build were finalized in a single 3-minute window on Oct 7 (one per channel, 16 total).
Older uncompacted segments with large footers (Oct 5-6, same log) never appear in errors. Already-compacted segments (footer 86/94 bytes)
are not read by recovery and never fail.

5. Could you please help me to understand this,

1. Is a 3.0.x -> 2.6.x downgrade supported for clusters using Woodpecker as the WAL? If not, can the docs state it clearly?
2. Is Woodpecker format V6 (LAC footer) readable by the Woodpecker version shipped in Milvus 2.6.22? If not, which 2.6.x release
   (if any) reads V6, or which min version is required before upgrading to 3.0?
3. Can the Woodpecker reader be made version-aware (read the footer version/flags, or fail with an explicit "unsupported format version N")
   instead of slicing a fixed-size buffer and reporting a CRC/length error? The current failure mode looks like corruption and makes
   CreateCollection hang instead of failing fast.
4. Is there a documented compatibility matrix (Milvus release -> Woodpecker release -> format version)?
5. Is there a supported recovery procedure for a cluster that was downgraded and wrote data under the older version in between
   (without restoring from backup)?

[xiaofan-luan]

3.0 -> 2.6.x downgrade is not supported.

We don't recommend any kind of downgrade. You can backup and restore to a older cluster if ncessary.

[Siddeshvr]

Ok @xiaofan-luan , Thank you