[Bug] [Bug] BE fails with [E-3110] parse date fail when __internal_schema.column_statistics contains '0000-00-00'

[1186683677]

Search before asking

• I had searched in the issues and found no similar issues.

Version

4.1.0

What's Wrong?

BE fails to initialize storage reader and continuously reports errors when __internal_schema.column_statistics contains illegal date strings like '0000-00-00 00:00:00'.

When a business table has invalid date strings (e.g. imported from legacy MySQL or skipping strict checks), the auto-analyze task collects '0000-00-00 00:00:00' as min/max values and writes them into __internal_schema.column_statistics.

When any query (or CBO optimizer) attempts to read statistics from __internal_schema.column_statistics, the BE Storage Reader fails to parse the string during initialization and cancels the entire query execution context with error:
[E-3110][INVALID_ARGUMENT] parse date or datetime fail, string: '0000-00-00 00:00:00' failed to initialize storage reader.

This creates a dead-lock where __internal_schema.column_statistics becomes completely unreadable (SELECT count(1) FROM __internal_schema.column_statistics fails), and CBO statistics loading for dependent queries gets broken.

What You Expected?

1. Sanitization on Write: Statistics collection (ANALYZE) should sanitize or convert invalid date/datetime strings (like '0000-00-00 00:00:00') to NULL or skip them before persisting into __internal_schema.column_statistics.
2. Graceful Degrade on Read: BE Storage Reader should handle or ignore unparsable date values during statistics deserialization gracefully, rather than throwing a fatal error [E-3110] and canceling the entire query.

How to Reproduce?

No response

Anything Else?

• Temporary Workarounds & Mitigations:

  1. Disabled global auto analyze to stop recurring failure:
     SET GLOBAL enable_auto_analyze = false;
  2. Dropped and truncated the corrupted internal statistics table to recover read ability:
     DROP STATS __internal_schema.column_statistics;
TRUNCATE TABLE __internal_schema.column_statistics;

• Impact:
  Because of this bug, we are forced to keep enable_auto_analyze = false, which prevents the cluster from automatically gathering CBO statistics for all other normal tables.

• Stack Trace snippet from BE:
  0# doris::BetaRowsetReader::_init_iterator()
  1# doris::BetaRowsetReader::_init_iterator_once()
  2# doris::Status doris::BetaRowsetReader::_next_batchdoris::Block(doris::Block*)
  3# doris::BetaRowsetReader::next_batch(doris::Block*)
  4# doris::VCollectIterator::Level0Iterator::refresh_current_row()
  5# doris::VCollectIterator::Level0Iterator::ensure_first_row_ref()
  6# doris::VCollectIterator::Level1Iterator::ensure_first_row_ref()
  7# doris::VCollectIterator::build_heap()
  8# doris::BlockReader::_init_collect_iter()
  9# doris::BlockReader::init()
  10# doris::OlapScanner::_open_impl()

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Initial triage: this looks like a 4.1.0 reader compatibility issue that is probably already fixed, while the proposed statistics write-sanitization issue is not yet demonstrated by the supplied evidence.

Verified from the 4.1.0 code:

• __internal_schema.column_statistics.min and max are VARCHAR(65533). The ANALYZE SQL explicitly casts MIN()/MAX() to STRING before persisting them. Therefore, a value such as 0000-00-00 00:00:00 in either field is stored as text; a plain scan of those two fields should not call the BE date SerDe.
• FE converts those strings back according to the source column type in ColumnStatistic.fromResultRow(). An AnalysisException while parsing a date min/max is caught and degraded to an unbounded min/max. This is an FE statistics-deserialization path, not the BE storage-reader initialization path shown in the report.
• In 4.1.0, date/datetime from_olap_string() failures are propagated through zone-map deserialization and can abort segment/rowset reader initialization with the reported parse date or datetime fail status. #63035, backported to branch 4.1 by #63073, changed DATE/DATETIME/DATEV2/DATETIMEV2 storage-string parse failures to a safe type default. Its code comment specifically covers invalid/zero-date placeholders in zone maps. The backport is absent from tag 4.1.0 and present in 4.1.1 and later 4.1 tags.
• #63010, also present starting with 4.1.1, prevents the optimizer from requesting column statistics for system tables such as tables in __internal_schema.

Consequently, the current stack is consistent with the known BE storage-string/zone-map issue, but it does not prove that the text in column_statistics.min/max itself makes that table unreadable. The failing storage field could instead be a DATE/DATETIME zone-map or default-value entry. The truncated stack is not sufficient to distinguish those paths.

Suggested next steps:

1. Clean the already affected statistics state, then reproduce on 4.1.1 or a newer 4.1 release. If the reader failure disappears, this can likely be closed as fixed by branch-4.1: [fix](parse) No longer throws exceptions when parse datetime failed in from_olap_string #63035 #63073 (with branch-4.1: [fix](fe) Return unknown stats for system tables #62913 #63010 also avoiding recursive/system-table CBO lookup).
2. If it still reproduces, please provide:
   • a minimal schema/load/ANALYZE/query SQL reproduction, including the exact failing SQL;
   • exact FE and BE build versions/commit hashes;
   • the complete BE error and stack, especially frames mentioning ZoneMap::from_proto, from_zonemap_string, from_fe_string, or DefaultValueColumnIterator;
   • the query ID plus tablet, rowset, and segment IDs from the BE log;
   • allow_zero_date from every BE and whether it changed after the data was written;
   • the business column type and load method/options that admitted the zero date;
   • evidence identifying whether 0000-00-00 00:00:00 is only in the VARCHAR min/max fields or also in a DATE/DATETIME storage zone map/default.
3. If a current 4.1 build reads the table successfully but still persists an unusable min/max statistic, treat that as a separate statistics-correctness issue. A regression should cover importing the value, running ANALYZE, querying column_statistics, and loading the resulting CBO statistic. Write-side sanitization should be based on the source column type; silently ignoring arbitrary parse failures in the general BE reader would be too broad.

For mitigation, prefer dropping statistics for only the affected business table once the internal table is readable. Truncating column_statistics removes statistics for every table and should remain a last-resort recovery step.

Breakwater-GitHub-Analysis-Slot: slot_a2bcc2b14fc1

[yizhimifeng]

Hello, I have also encountered this problem. This has been repeatedly deadlocked. Will it cause the statistical information collection of all tables to fail? Then will this collection fail to affect the query efficiency? Does it still affect the CPU and memory of the be node to a large extent? Thank you