[Bug] ROW binlog is not garbage-collected after binlog.ttl_seconds expires

[yujun777]

Problem

For a UNIQUE KEY Merge-on-Write table with ROW binlog, binlog entries remain readable after they are older than the configured binlog.ttl_seconds. Please confirm whether this is a bug or an intentional limitation of ROW binlog TTL.

Related context: DORIS-27284

Environment

• Repository: apache/doris
• Commit: a4dae0b
• Deployment: local non-cloud FE/BE cluster

Reproduction

CREATE TABLE stream_ttl_probe.base (
  id BIGINT,
  v BIGINT
) UNIQUE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES (
  "replication_num" = "1",
  "enable_unique_key_merge_on_write" = "true",
  "binlog.enable" = "true",
  "binlog.format" = "ROW",
  "binlog.need_historical_value" = "true",
  "binlog.ttl_seconds" = "1",
  "binlog.max_history_nums" = "100000"
);

CREATE STREAM stream_ttl_probe.s ON TABLE stream_ttl_probe.base
PROPERTIES ("type" = "min_delta", "show_initial_rows" = "false");

INSERT INTO stream_ttl_probe.base VALUES (1, 10), (2, 20);

Waited about 45 seconds, which is well beyond the 1-second TTL and multiple FE GC intervals, then executed:

SET show_hidden_columns = true;
SELECT id, v, __DORIS_STREAM_CHANGE_TYPE_COL__, __DORIS_STREAM_SEQUENCE_COL__
FROM stream_ttl_probe.s ORDER BY id;

Observed result

The stream still returned both rows:

1  10  APPEND  467850668704530433
2  20  APPEND  467850668704530433

The stream remained non-stale (IS_STALE = 0). Filesystem inspection of the corresponding BE tablet also showed that the _row_binlog files remained after more than one minute. The same behavior was observed in an IVM reproduction after waiting about 74 seconds.

Expected result

If binlog.ttl_seconds applies to ROW binlog, entries older than the TTL should become eligible for physical GC. A stream whose required binlog range is no longer available should report a stale/binlog-broken condition, rather than continue reading expired entries.

If TTL is intentionally only supported for CCR/statement binlog and not ROW binlog, the documentation and property validation should make that limitation explicit.

Source analysis

The FE BinlogGcer path appears to process FE TableBinlog objects used by CCR and excludes ROW binlog. ROW binlog is stored under BE tablet _row_binlog; the current ROW binlog compaction/GC path does not appear to apply binlog.ttl_seconds. Tablet::binlog_ttl_ms() exists, but no effective ROW binlog TTL GC call path was found.

Please confirm the intended contract and, if this is a bug, add ROW binlog TTL GC and the corresponding stream/binlog-broken handling.

Related issues

• #65265
• #65418

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_2feae5dd26c0
This content is generated by AI for reference only.

Initial code-path check:

This should be treated as a real ROW-binlog retention gap unless maintainers decide that ROW binlog intentionally does not support TTL. The reported behavior is consistent with the current implementation: binlog.ttl_seconds is accepted together with binlog.format=ROW, but I do not find an effective ROW-binlog TTL GC path.

Evidence from the reported commit a4dae0b64e9e7c6db5983b566656442f1ae2bc5d:

• FE BinlogGcer calls BinlogManager.gc() and sends BinlogGcTask to BE based on DB/Table binlog tombstones. FE BinlogConfig.isEnableForCCR() explicitly excludes ROW, while isEnableForStreaming() is the ROW path, so the existing FE tombstone GC path does not cover ROW binlog.
• BE StorageEngine::gc_binlogs() -> Tablet::gc_binlogs(version) deletes ordinary _binlog metadata/data/files by FE-provided version. It does not traverse tablet_meta()->all_row_binlog_rs_metas() and does not delete tablet _row_binlog rowsets.
• ROW binlog files are tracked as rowsets in TabletMeta::_row_binlog_rs_metas and are read with capture_row_binlog=true. The observed _row_binlog files remaining is therefore expected with current code unless row-binlog compaction removes/replaces those rowsets. The row-binlog compaction policy is score/size/count/time-threshold driven; I did not find a TTL cutoff using binlog.ttl_seconds. Tablet::binlog_ttl_ms() exists, but I found no caller.
• IS_STALE is FE stream metadata (BaseTableStream.stale) reported by TableStreamManager. I found setters and readers for this flag, but no path that marks a stream stale when a required ROW-binlog range expires or is missing. If a row-binlog version path is absent, BE rowset capture returns a scan error, but that does not update the stream stale flag.

I also checked the same symbols on the upstream master head I fetched (a9816faa6652362b4fc8fbbf057d62652e016624) and did not find a different ROW-binlog TTL/stale path there.

Conclusion: current code does not make ROW binlog entries eligible for physical GC based on binlog.ttl_seconds, and it also does not convert ROW-binlog retention loss into an observable stream stale/binlog-broken state. This is best classified as a bug/missing implementation. If the intended contract is instead that TTL only applies to CCR/statement binlog, property validation and documentation should reject or clearly document binlog.ttl_seconds for ROW to avoid silent unlimited retention.

Suggested next steps:

1. Define the ROW-binlog retention contract. TTL does not need to mean deletion exactly at the boundary, but it should create a reachable eligibility cutoff if the property is accepted.
2. Add BE-side ROW-binlog GC over _row_binlog_rs_metas / _row_binlog files, coordinated with row-binlog compaction metadata and binlog delete-vector cleanup.
3. Add FE/Table Stream handling for retention loss: a stream or IVM whose required unconsumed range falls below the retained ROW-binlog low watermark should become observable as stale/binlog-broken, or fail with a clear diagnostic and documented recovery/fallback path.
4. Add regression coverage with a very small TTL and waits across FE GC plus BE compaction/GC intervals, verifying both physical cleanup and stream state/diagnostic behavior.

No critical information is missing to confirm the code gap. Useful additions for the eventual fix PR would be FE/BE logs around the wait window, information_schema.table_streams output before/after the wait, and BE tablet metadata showing _row_binlog_rs_metas before/after GC.