Milvus Message Queue Switch Failed — How Can I Roll Back?

[theeversun]

My Milvus version is 3.0. It currently uses Kafka as the message queue and a third-party object storage service. When I tried to switch the message queue from Kafka to Woodpecker, I encountered the following error:

AWS Error UNKNOWN (HTTP status 400) during CompleteMultipartUpload operation:
Unable to parse ExceptionName: EntityTooSmall
Message: Your proposed upload is smaller than the minimum allowed object size.
Each part must be at least 5MB in size, except the last part.

Apart from restoring from a backup, is there any way to roll back the change and avoid a prolonged service outage?

[yhmo]

This error means your S3-compatible storage rejected a multipart upload where a non-final part was under 5MB. By S3 spec, only non-final parts must be ≥5MB — the last part may be any size. MinIO's SDK (used by both Woodpecker and Milvus) always sizes non-final parts at ≥16MB, so this upload is spec-legal and works fine on real AWS and MinIO.

So this is a storage-compatibility issue with your third-party object storage, not a Milvus/Woodpecker bug: your provider enforces a stricter check than AWS (most likely rejecting a last part <5MB too).

to confirm: manually upload a 10MB object in two parts (8MB + 2MB) via AWS CLI/mc and complete the multipart upload:

• If it fails with the same error → your S3 provider is the problem; contact them or use MinIO.
• If it succeeds → share the Milvus/Woodpecker logs for the failing upload (object key + size) and we can dig further.

[theeversun]

My first question is: if an unexpected situation occurs, is there any way—other than restoring from a backup—to roll back to the previous normal state? Losing some data would be acceptable.

Below is my test code for OBS. The upload works properly without any issues. However, the Milvus logs do not contain any information about the data size. Do I need to enable debug mode to view this information?

bash-5.2# dd if=/dev/urandom of=/tmp/part1.bin bs=1M count=8
dd if=/dev/urandom of=/tmp/part2.bin bs=1M count=2
8+0 records in
8+0 records out
8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0308443 s, 272 MB/s
2+0 records in
2+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0082142 s, 255 MB/s
bash-5.2# cd /tmp/
bash-5.2# ls 
part1.bin  part2.bin
bash-5.2# ls -lh /tmp/part1.bin /tmp/part2.bin
-rw-r--r-- 1 root root 8.0M Aug 19 01:21 /tmp/part1.bin
-rw-r--r-- 1 root root 2.0M Aug 19 01:21 /tmp/part2.bin
bash-5.2# OBJECT_KEY="multipart-test-$(date +%s).bin"
echo "$OBJECT_KEY"
multipart-test-1787102538.bin
bash-5.2# UPLOAD_ID=$(
  aws s3api create-multipart-upload \
    --bucket "$S3_BUCKET" \
    --key "$OBJECT_KEY" \
    --endpoint-url "$S3_ENDPOINT" \
    --query UploadId \
    --output text
)
bash-5.2# echo "OBJECT_KEY=$OBJECT_KEY"
echo "UPLOAD_ID=$UPLOAD_ID" 
OBJECT_KEY=multipart-test-1787102538.bin
UPLOAD_ID=f4251af3-6699-4c8a-ba2c-1237a084cb2e
bash-5.2# ETAG1_RAW=$(
  aws s3api upload-part \
    --bucket "$S3_BUCKET" \
    --key "$OBJECT_KEY" \
    --part-number 1 \
    --upload-id "$UPLOAD_ID" \
    --body /tmp/part1.bin \
    --endpoint-url "$S3_ENDPOINT" \
    --query ETag \
    --output text
)
bash-5.2# ETAG1=$(printf '%s' "$ETAG1_RAW" | tr -d '"')
echo "ETAG1=$ETAG1"
ETAG1=69bd955c9aaa4a57d0ee5812500a2c44
bash-5.2# ETAG2_RAW=$(
  aws s3api upload-part \
    --bucket "$S3_BUCKET" \
    --key "$OBJECT_KEY" \
    --part-number 2 \
    --upload-id "$UPLOAD_ID" \
    --body /tmp/part2.bin \
    --endpoint-url "$S3_ENDPOINT" \
    --query ETag \
    --output text
)
bash-5.2# ETAG2=$(printf '%s' "$ETAG2_RAW" | tr -d '"')
echo "ETAG2=$ETAG2"
ETAG2=9be023d724d7fe673cb48339a60ee33b
bash-5.2# cat > /tmp/parts.json <<EOF
{
  "Parts": [
    {
      "ETag": "$ETAG1",
      "PartNumber": 1
    },
    {
      "ETag": "$ETAG2",
      "PartNumber": 2
    }
  ]
}
EOF
bash-5.2# aws s3api complete-multipart-upload \
  --bucket "$S3_BUCKET" \
  --key "$OBJECT_KEY" \
  --upload-id "$UPLOAD_ID" \
  --multipart-upload file:///tmp/parts.json \
  --endpoint-url "$S3_ENDPOINT"
{
    "Location": "http://****************************/milvus/multipart-test-1787102538.bin",
    "Bucket": "milvus",
    "Key": "multipart-test-1787102538.bin",
    "ETag": "\"c95b18314e481638893ce65aa30f6551-2\""
}
bash-5.2# aws s3api head-object \
  --bucket "$S3_BUCKET" \
  --key "$OBJECT_KEY" \
  --endpoint-url "$S3_ENDPOINT" \
  --query ContentLength \
  --output text
10485760
bash-5.2# 

My main concern is how we can quickly roll back if unexpected issues occur in the production environment. Alternatively, is the only option to create a new instance and restore the data from the original backup?

[yhmo]

Yes, you can roll back only if you did not enable any 3.0 features.

1. Storage V3 is OFF
   common.storage.useLoonFFI = false (default). If true, 2.6 cannot read your segments. The release notes for 3.0 mention this: https://milvus.io/docs/release_notes.md

2. No TEXT / Snapshot

3. Null values of StructArray

4. Index versions at defaults
   dataCoord.targetVecIndexVersion ≠ 10, dataCoord.targetScalarIndexVersion ≠ 4 (both default -1). Also mention in 3.0 release notes: https://milvus.io/docs/release_notes.md

[tinswzy]

A few things about this error:

It doesn't come from Woodpecker — Woodpecker never calls CompleteMultipartUpload. It writes through the MinIO Go SDK with explicit object sizes, and its objects are ~2MB by default, well below the 16MiB threshold where that SDK switches to multipart. What you hit is Milvus flushing data during the switch: that write goes through milvus-storage, which uploads with multipart (10MB parts), and your OBS-side S3 compatibility rejected it.

There is no way to roll back a switch in progress. The switch is driven by an AlterWAL message already durably written into the Kafka WAL plus state persisted in the channel checkpoint, so reverting mq.type doesn't undo it (a channel opens the WAL recorded in its checkpoint, not the one in config), and issuing the reverse alter needs a writable WAL — which is exactly what a stuck switch blocks. The upside is that nothing has been committed yet: checkpoints and metadata are only rewritten after the flush succeeds, so your cluster is still in its pre-switch state and resumes as soon as that upload goes through. Fixing the upload is both the fix and the way out.

Later on we may provide a birdwatcher tool that resets the checkpoint back to the AlterWAL position to recover a channel. That mainly targets the case where the downstream MQ isn't healthy — your case hasn't reached the point of opening the new MQ at all, it's failing while Milvus writes to object storage.

For validating your storage: the 8MB + 2MB upload doesn't reproduce what Milvus sends, so passing it doesn't clear OBS yet. Milvus only ever produces one part below 5MB — multipart is used only for files larger than 10MB, every part except the last is exactly 10MB, and only the last part is the remainder. So the last part is the only thing that can trip a minimum-size check, and your test already shows a 2MB last part is accepted.

What the OBS docs say is that a part must be within [100KB, 5GB] while the last part is [0, 5GB] — i.e. exempt — and that the size check happens at merge time (UploadPart), which is exactly where your error is thrown. Two cases worth uploading to check the implementation matches that:

• a 64KB object as a single-part multipart upload,
• 10MB + a 64KB tail,

If both succeed, part size isn't the reason and we need the failing object itself. Please export the full logs with
deployments/export-log/export-milvus-log.sh — the complete line carries the bucket and key, which your paste is cut just before.

[theeversun]

I have already tested the following:

A 64 KB object uploaded as a single-part multipart upload
A 10 MB object with a 64 KB tail
Both worked without any issues.

After switching the message objects from Woodpecker to Kafka through the recovery procedure, I am inclined to believe that this issue was caused by the S3 storage.

Next, I will briefly describe the steps I took:

1. Shut down all Milvus components.
2. Changed the message queue type back to Kafka in etcd.
3. Deleted the remaining broadcast tasks.
4. Cleared the StreamingNode recovery state and the DataCoord channel checkpoints.
5. Started the Milvus components.

After startup was complete, some DML operations panicked after fetching the channel.

Further investigation revealed that the connection was reset by the peer while the data was being sent, but the storage returned HTTP 200 and an ETag, which resulted in corrupted data.

After I removed the problematic data, Milvus returned to normal operation.