Safety of reducing PulsarV3 defaultRetentionTimeInMinutes from 7 days to 3 days in Milvus cluster

[kumarsah-deshaw]

Environment:

• Milvus version: v2.6.2 (distributed mode)
• Message queue: Apache Pulsar v3.0.7 (NetApp storage mounted as PVCs)
• Components: mixcoord (active-standby), proxy x 3, querynode x 88, datanode x 4, streamingnode x 2
• Storage: External MinIO (bare-metal disk storage)

Current Pulsar broker config:

  defaultRetentionTimeInMinutes: "10080"   # 7 days
  defaultRetentionSizeInMB: "-1"           # unlimited
  ttlDurationDefaultInSeconds: "259200"    # 3 days
  backlogQuotaDefaultLimitGB: "8"
  backlogQuotaDefaultRetentionPolicy: producer_exception
  subscriptionExpirationTimeMinutes: "3"

Problem statement:
We periodically run large-scale backfilling exercises (bulk-inserting historical data into multiple collections). During a recent backfill, the Pulsar BookKeeper ledger storage filled up, causing ingestion failures across the cluster. We have since added more storage, but we want a longer-term solution before the next backfill exercise.
We are considering reducing defaultRetentionTimeInMinutes from 10080 (7 days) to 4320 (3 days) - matching our existing ttlDurationDefaultInSeconds (3 days TTL). Our hypothesis is that since messages are already TTL'd at 3 days, the 7-day retention setting may be keeping acknowledged messages on disk unnecessarily.

Questions for Zilliz team:

1. Safety: Is it safe to reduce defaultRetentionTimeInMinutes from 7 days to 3 days with no data loss or corruption risk? Specifically - does Milvus ever need to replay Pulsar messages older than 3 days for recovery (e.g. after a DataNode or StreamingNode crash)?
2. TTL vs Retention interaction: Since ttlDurationDefaultInSeconds is already set to 259200 (3 days), are messages already being expired at 3 days regardless of the retention setting? If so, does the 7-day defaultRetentionTimeInMinutes have any practical effect on ledger disk usage in our setup?
3. Prerequisites - write operations: Do we need to pause all write operations (ingestion) and wait for DataNode to fully flush all growing segments to object storage before reducing retention? Or is it safe to change this while ingestion is active?
4. Apply mechanism: Can this be applied via helm upgrade with a rolling Pulsar broker restart, or does it require a full Milvus cluster restart? Will in-flight ingestion recover automatically after broker rolling restart?
5. Backfill recommendation: During future bulk-insert/backfill exercises, is there a recommended Pulsar configuration to prevent ledger overflow - e.g. temporarily reducing TTL further, or tuning
   backlogQuotaDefaultLimitGB?

Related error from last ingestion failures:
[BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/by-dev-rootcoord-dml_4] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6

Since this is a production cluster exercise, we can't afford any data loss or data corruption hence we are reaching out to you for advise.

[yhmo]

In Pulsar, defaultRetentionTimeInMinutes is for "topic retention", ttlDurationDefaultInSeconds is for "message ttl/expiry".

• defaultRetentionTimeInMinutes = 3 days --------------------- acked messages older than 3 days become eligible to be cleaned from pulsar storage, but it doesn't affect unacked messages(even if they are older than 3 days).
• ttlDurationDefaultInSeconds = 3 days ------------------- unacked messages older than 3 days become eligible for expiry, but it doesn't affect acked messages(even if they are older than 3 days).

If a datanode or streaming node crashes and restarts, it will read unacked messages from the consume checkpoint(we call recovery).

Some different concepts here:

• Pulsar topic retention: defaultRetentionTimeInMinutes
• Pulsar messages TTL: ttlDurationDefaultInSeconds
• Milvus collection TTL: expired data is filtered out by query/search requests; flush action can skip expired data so that only unexpired data can be persisted to milvus storage; compaction actions clean expired data for sealed segments

So, for your questions:

1. defaultRetentionTimeInMinutes from 7 days to 3 days doesn't affect unacked messages, so it is safe because milvus recovery only processes unacked messages. On the other hand, we don't recommend setting it to an extremely small value(one minute, for example).
2. ttlDurationDefaultInSeconds = 3 days, unacked messages older than 3 days become eligible for expiry, once a unacked message us expired, it is unavailable to be consumed. If milvus consume checkpoint is delay more than 3 days, there will be data loss, no matter what value is set for milvus collection TTL. 7-day defaultRetentionTimeInMinutes only affects acked messages, a smaller value of defaultRetentionTimeInMinutes can reduce disk usage for pulsar.
3. No need to do flush when you try to change defaultRetentionTimeInMinutes value. It only affects acked messages. Acked message already consumed by milvus. If you try to change ttlDurationDefaultInSeconds value, a smaller value will get a higher risk of data loss when milvus consume checkpoint is delayed. Assume we set ttlDurationDefaultInSeconds to 1200(20 minutes), it is recommended to enable the ttProtection in the milvus.yaml. Set the maxTimeTickDelay to less/equal to 1200 so that milvus can reject write actions when dml delay exceeds ttlDurationDefaultInSeconds.

quotaAndLimits:
  limitWriting:
    ttProtection:
      enabled: true
      # maxTimeTickDelay indicates the backpressure for DML Operations.
      # DML rates would be reduced according to the ratio of time tick delay to maxTimeTickDelay,
      # if time tick delay is greater than maxTimeTickDelay, all DML requests would be rejected.
      # seconds
      maxTimeTickDelay: 1200

4. Change defaultRetentionTimeInMinutes value via the configuration file; usually, only the relevant pulasr component(s) need rollout/restart.
5. I suppose your "bulk-insert" is: insert data batch by batch. I think the default configuration of ttlDurationDefaultInSeconds/defaultRetentionTimeInMinutes is ok. You can enable the ttProtection in milvus.yaml and set maxTimeTickDelay to less than ttlDurationDefaultInSeconds if you want to avoid unexpected data loss.

[BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/by-dev-rootcoord-dml_4] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6

This error indicates too many bookleeper bookies are down, pulsar could not safely write the message.