4.0.7 Release Notes

[morningman]

4.0.7 Release Notes

4.0.6 Release Notes: #64065

Apache Doris 4.0.7 is a maintenance release in the 4.0 series. It focuses on query correctness, load stability, compute-storage decoupled deployments, File Cache reliability, and external data source compatibility. All 4.0.x users are advised to upgrade.

Highlights of this release include:

• Cloud enhancements: add table-level event-driven cache warmup, enable Packed File and empty Rowset optimizations by default, and fix several Recycler, Schema Change, Compaction, and File Cache issues.
• Object storage improvements: support S3 Storage Vault credential providers without a Role ARN, and allow deterministic S3 paths to be accessed without ListBucket permission.
• Better observability: add FE connection-limit metrics, per-job Routine Load metrics, File Cache queue metrics, and Compute Group information for MTMV refresh tasks.
• Many correctness and stability fixes covering TopN, joins, materialized view rewriting, Stream Load, Routine Load, external catalogs, File Cache crashes, and memory usage.

Before upgrading, read the Behavior Changes section below.

Behavior Changes

• INSERT OVERWRITE now permanently removes replaced partitions instead of placing them in the Recycle Bin. This reduces Recycle Bin pressure, but these partitions can no longer be recovered from it ([fix](iow) force drop partition in INSERT OVERWRITE #62510).
• Cloud mode now enables enable_packed_file and skip_writing_empty_rowset_metadata by default ([improvement](cloud) Enable packed file and empty rowset optimization by default #63475).
• The BE /api/file_cache?op=clear API no longer supports synchronous clearing. Requests with sync=true now run asynchronously and return a warning ([fix](filecache) Disable sync file cache clear in http api #64321).
• Object storage clients no longer retry S3 SlowDown or Azure HTTP 429 responses inside the SDK retry policy, avoiding additional delays before higher-level handling takes effect ([fix](s3) Avoid retrying object storage SlowDown errors #63776).
• COM_RESET_CONNECTION now resets connection-scoped state consistently with MySQL ([fix](connect) Align COM_RESET_CONNECTION behavior with MySQL #63884).
• In Cloud mode, SHOW PARTITIONS now reports StorageMedium as OBJECT_STORAGE and ReplicaAllocation as NULL ([fix](cloud) normalize SHOW PARTITIONS display for storage and replica #60871).

New Features & Improvements

Compute-Storage Decoupling

• Add table-level filtering through ON TABLES for event-driven cache warmup ([feature](cloud) Add table-level event-driven warm up #63832).
• Add a dynamic Recycler instance filter configuration ([chore](cloud) Support dynamic recycler instance filter config #63822).
• Cache the Compute Group ID per query and remove redundant locks from the Backend selection hot path ([opt](cloud) cache cluster id per query and drop redundant locks on getBackendId hot path #63636).
• Skip waiting for asynchronous Rowset warmup to reduce write latency (branch-4.0: [fix](cloud) Skip wait for async rowset warmup #63877).

Object Storage & External Data Sources

• Support using an S3 Storage Vault credential provider without specifying a Role ARN ([feat](s3) support S3 storage vault credentials provider without role ARN #64766).
• Use HEAD requests for deterministic S3 paths, allowing access with only s3:GetObject permission and without s3:ListBucket ([opt](s3) Skip S3 listing for deterministic file paths using HEAD requests #60414).
• Fill the Hive metadata cache while estimating table row counts to avoid duplicate HMS access during planning ([fix](hive table) Fill Hive meta cache when loading row count for queries #63470).

Observability & Operations

• Add doris_fe_connection_max and per-user doris_fe_user_connection_max metrics ([feature](fe) Add connection max metrics #64742).
• Add per-job Routine Load metrics and refresh Routine Load lag more promptly ([feat](job) add per-job routine load metrics #63576, [enhance](job) refresh routine load lag more timely #63654).
• Add partition-filter conditions to SQL Block Rules ([feature](fe) Add partition filter sql block rule #62196).
• Show the Compute Group used by MTMV refresh tasks ([feature](fe) Show compute group for MTMV refresh task #63206).
• Include scan-node IDs in Nereids physical plans ([opt](nerieds) PhysicalOlapScan and PhysicalFileScan should print its id in plan #62509).
• Record session variables set by per-query SET_VAR hints in Audit Logs ([fix](audit) record per-query SET_VAR hint session variables in audit log #64569).
• Improve password masking for CREATE USER and ALTER USER statements in Audit Logs ([enhance](nereids) improve masking of user's password for ALTER USER and CREATE USER commands in audit logs #62141).
• Add support for LDAP default roles ([improvement](fe) Support LDAP default roles #63411).

Memory & Performance

• Treat reclaimable inactive_file page cache as available memory in cgroup environments, reducing unnecessary query cancellation ([improvement](cgroup) inactive_file should be treated as available memory to avoid query be cancelled #64347).
• Add a BE-level cache for Tablet schemas used during load (branch-4.0: [opt](memory) add BE-level cache for load tablet schemas #64581).
• Reduce Row Store MemTable flush memory usage ([opt](memory) optimize row-store memtable flush memory in the row-store scenario #63342).
• Truncate Segment key bounds without retaining complete backing buffers ([opt](memory) truncate segment key bounds before storing segment stats #63469).
• Aggregate non-MOW Segment key bounds to reduce metadata size (branch-4.0: [improvement](rowset) Aggregate non-MOW segment key bounds #64305).
• Release Packed File writer buffers immediately after flush ([opt](memory) release packed file writer buffer after flush #63967).
• Use Segment footer raw_data_bytes to improve first-time Compaction batch-size estimation ([improve](compaction) Use segment footer raw_data_bytes for first-time batch size estimation #62263).
• Bound File Cache LRU replay queues and add queue backlog metrics ([improvement](filecache) limit file cache LRU replay queues #64381).

Important Bug Fixes

Query Result Correctness

• Fix incorrect Set Operation rewriting caused by using CTE-producer output in place of the regular child output ([fix](set operation) Use regular child output for set operation rules #64908).
• Remove an unsafe TopN-to-Max rewrite that could produce incorrect results ([fix](topn_to_max)Remove the topnToMax optimizer rewrite that converted TOPN into MAX. #63519).
• Fix incorrect row counts when merging nested TopN operators with an outer OFFSET ([fix](nereids) clamp the merged limit of MERGE_TOP_N by the parent offset #64306).
• Preserve deterministic ordering when merging TopN operators whose order keys have a prefix relationship ([fix](fe) Merge TopN with child prefix order keys #64685).
• Fix TopN Runtime Filter activation ([fix](be) Fix TopN runtime filter activation #63969).
• Reject TopN expressions in correlated scalar subqueries instead of generating an invalid plan ([fix](fe) Reject TopN in correlated scalar subquery #64251).
• Fix aggregate ORDER BY expression pushdown for window functions and scalar subqueries ([fix](analyzer) Normalize aggregate order by pushdown #64787).
• Fix Cast Project pushdown through UNION DISTINCT, which could change deduplication semantics ([fix](fe) Prevent cast project pushdown through union distinct #64080).
• Fix incorrect elimination of Null-Aware predicates in subqueries ([fix](subquery) eliminate null aware by mistake #64639).
• Fix pre-aggregation context leaking between Join branches ([fix](nereids) Fix pre-aggregation context leakage across join branches #63357).
• Fix Runtime Filter handling for Outer Joins ([fix](fe) Backport runtime filter outer join fix to 4.0 #64157).
• Align Legacy Planner literal comparison semantics with Nereids ([fix](planner) align legacy literal compareLiteral with Nereids ComparableLiteral semantics #63481).
• Fix missing query result packets during an FE Master switch, which could leave clients waiting until timeout ([fix](query) avoid missing result packet for query during master switch #62721).
• Fix MTMV rewriting for multi-hop Outer Joins and null-reject compensation ([fix](mtmv) Infer null-reject from INNER JoinEdge for multi-hop outer join MV rewrite #62492, [fix](fe) Add null reject compensation for join rewrite #63268).
• Fix mutation of excluded trigger tables while processing MTMV refresh triggers ([fix](mtmv) Avoid mutating excluded trigger tables #62984).
• Fix View queries failing after the schema of an underlying external table changes ([fix](nereids) Guard LogicalView.computeOutput() against schema drift (IndexOutOfBoundsException) #64007).

Functions & Types

• Fix convert_tz constant folding and partition pruning across Daylight Saving Time transitions ([fix](partition_prune) Avoid incorrect convert_tz partition pruning across DST transitions #63853, [fix](fe) Align convert_tz folding with BE DST handling #64029).
• Preserve the sign of negative sub-hour TIMESTAMPTZ offsets ([fix](function) preserve sign for negative sub-hour TIMESTAMPTZ offsets #62823).
• Fix datediff constant folding with zero dates ([fix](fe) Fix datediff folding for zero date #64084).
• Fix json_contains returning false for duplicate elements in candidate arrays ([fix](be) Fix json contains duplicate array candidates #63301).
• Support Boolean casts in array_first and array_last ([fix](array function) Support boolean cast for array first and last #64847).
• Validate the Lambda arity accepted by array_sort ([fix](function) Validate array_sort lambda arity #64825).
• Reject invalid multi-argument COUNT(DISTINCT ...) Window Functions during analysis ([fix](window) only allow 1 argument in count distinct in window function #64783).
• Limit retention() to its supported maximum of 32 arguments, preventing out-of-bounds memory access ([fix](retention) Limit param count to 32 to avoid BE heap overflow #64521).
• Reject COUNT(DISTINCT variant) with a clear unsupported-type error instead of a BE internal error ([fix](variant) Reject COUNT DISTINCT on variant arguments #63479).
• Reject Lucene-syntax search on columns without an Inverted Index ([branch-4.0][fix](search) reject Lucene-syntax search on columns without inverted index #63857).
• Reject invalid IPv4 default values during CREATE TABLE analysis ([fix](ddl) reject invalid IPv4 default value at CREATE TABLE time #62906).
• Fix parsing of ISNULL expressions by placing them under primaryExpression ([fix](parser) isNull should under primaryExpression #63619).

Load & Transactions

• Fix truncated input when inferring schemas from compressed files through http_stream (branch-4.0: [fix](stream-load) fix http_stream compressed file truncation and stabilize sql_cache P0 cases #64769).
• Fix Stream Load failures caused by integer overflow while comparing large Backend IDs ([fix](fe) Fix int overflow in BeIdComparator causing stream load failure #63565).
• Replace Tablet Writer close polling with event-based wakeup to reduce waiting and scheduling overhead ([fix](load) replace tablet writer close polling with event wakeup #64221).
• Fix a Broken Pipe risk when redirecting Stream Load requests without consuming the request body (branch-4.0: [fix](fe) Fix broken pipe risk on stream load redirect with unconsumed request body #64303).
• Fix load_to_single_tablet routing for Auto Partition tables ([fix](load) fix load_to_single_tablet routing for auto partition #64356).
• Fix COPY INTO ... SELECT failing to bind file-column placeholders ([fix](nereids) bind file column placeholders for copy into select #64395).
• Keep shared Delta Writer state independent of the Runtime State belonging to its original sink ([fix](delta writer) Fix shared delta writer state lifetime #64349).
• Fix enable_insert_strict incorrectly changing the semantics of enable_strict_cast ([fix](insert) enable_insert_strict should not affect semantic of enable_strict_cast #63794).
• Delay INSERT OVERWRITE partition routing until incremental open ([fix](insert overwrite) delay overwrite partition routing until incremental open #63209).
• Select the transaction INSERT Backend from the current Compute Group ([fix](transaction) select txn insert backend from current cluster #63634).
• Avoid quorum stalls when Delete Push tasks fail ([fix](delete) avoid quorum stalls on delete push failures #61647).
• Fix an NPE when an Auto Partition is concurrently dropped during creation ([branch-4.0][fix](auto-partition) avoid NPE in createPartition when a partition is concurrently dropped #65357).
• Keep load row-count metrics monotonic when Auto Partition creates new partitions ([fix](auto partition) keep load row metrics monotonic for auto partition #64109).
• Fix an NPE in force-finished Publish tasks ([fix](publish) Avoid NPE for force-finished publish task #63069).

Routine Load

• Serialize Routine Load task renewal when task submission fails ([fix](job) lock routine load task renew on submit failure #64731).
• Improve Kafka read_committed zero-row diagnostics and delay retries for zero-row batches ([enhance](job) add zero-row hint for Kafka read_committed load #63664, [opt](job) delay Kafka read committed zero-row retries #64046).

Compute-Storage Decoupling

• Refresh the base Tablet before retrying Cloud Schema Change V1 to avoid repeated Compaction conflicts caused by stale versions ([fix](cloud) Refresh base tablet before schema change V1 #64312).
• Fix races between Empty Cumulative Compaction and Base/Cumulative Compaction on the same Tablet ([fix](cloud-compaction) prevent EMPTY_CUMULATIVE / BASE-CUMU races on the same tablet #64619).
• Fix reading Packed Inverted Index files after a File Cache miss ([fix](cloud) Fix reading packed inverted index file on file cache miss #64383).
• Validate Recycle Rowset key state during commit_rowset to preserve idempotency and prevent invalid commits ([fix](cloud) Validate recycle rowset key state during commit rowset #63985).
• Recycle empty Rowsets without requiring a Resource ID ([fix](cloud) Recycle empty rowsets without resource id #64630).
• Prevent Tablet KV metadata leaks after a partially failed recycle operation ([fix](cloud) Prevent tablet KV leak after partial recycle failure #63377).
• Deduplicate pending one-shot Cache Warmup jobs ([fix](cloud) Deduplicate pending one-shot warm up jobs #62384).
• Fix Azure resource persistence across FE restart and replay ([fix](fe) fix Azure resource persistence #65052).
• Avoid false Tablet diagnosis alarms in Cloud mode ([fix](cloud) avoid false tablet diagnosis alarms in cloud mode #60805).

File Cache & Storage

• Fix a race while scanning Backend Tablet Rowset maps ([fix](be) Avoid rowset map race in backend tablets scanner #65288).
• Initialize Thread Context on asynchronous IO worker threads ([fix](be) Init thread context on AsyncIO worker threads #64846).
• Fix incorrect fallback reads after a partial File Cache hit, and move LRU updates off the query read path ([fix](filecache) add async lru update machanism and fix partial hit in cache reader #61083).
• Fix crashes when delayed File Cache holders reference an already-removed cache cell ([fix](filecache) avoid crash when late holder cleanup sees removed cache cell #62437).
• Fix crashes when a finalized scan cannot find its local File Cache writer ([fix](filecache) avoid BE crash when finalize misses local cache writer #62389).
• Fix File Cache percentage overflow when clearing the cache immediately after a BE restart ([fix](filecache) fix clear_file_cache right after reboot causing file cache size percent overflow #63410).
• Make synchronous internal File Cache removal safe ([fix](file-cache) Make sync clear use safe removal #64578).
• Fix duplicate File Cache metric accumulation in NewOlapScanner ([fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner #61072).
• Exclude Cache Warmup reads from File Cache hit-ratio metrics ([fix](filecache) exclude warmup reads from file cache hit ratio metrics #63394).
• Fix a finalized Pipeline task being submitted again and crashing BE ([fix](be) Avoid finalized pipeline task submit crash #64946).

Lakehouse & External Data Sources

• Fix Iceberg COUNT(*) pushdown throwing an NPE when snapshot summary counters are absent ([fix](iceberg) Fix NPE in COUNT(*) pushdown when snapshot summary omits total-* counters #64648).
• Fix unsupported mapping of Iceberg varint types ([bug](iceberg) mapping iceberg varint type to unsupported type #64331).
• Fix format detection for migrated Iceberg tables ([bug](iceberg) fix can't get migrated Iceberg tables format type #64134).
• Fix Parquet Reader failures or BE crashes when a previous Row Group used Dictionary Filtering (fix: block type reset after dict filter #63168).
• Normalize HDFS default paths and OSS bucket endpoint paths across external catalogs ([fix](fe) Normalize default HDFS paths in LocationPath #63476, branch-4.0: [fix](multi-catalog) OSS bucket endpoint path normalization #64943).
• Push SQL Server and Oracle Boolean predicates as 1 and 0 instead of unsupported TRUE and FALSE literals ([fix](jdbc) Push SQL Server/Oracle boolean predicates as 1/0 instead of TRUE/FALSE #64760).
• Correctly recognize Doris-compatible targets in MySQL JDBC Catalogs ([fix](fe) Detect Doris-compatible MySQL JDBC targets #64389).
• Fix a MaxCompute Scanner memory leak and improve large-field writes ([fix](mc) fix memory leak and optimize large data write for MaxCompute connector #61245).

Metadata, Protocol & Observability

• Fix remote Arrow Flight SQL result receiver initialization ([fix](fe) Fix remote Flight SQL result receiver initialization #63136).
• Fix SHOW PROCESSLIST FULL returning unexpected results ([fix](process list) return unexpected result in FULL mode #64631).
• Fix persisted-variable output in SHOW VARIABLES ([fix](show variables) Fix changed variable output in show variables #63734).
• Support $ in MySQL-compatible pattern matching (branch-4.0: [fix](fe) Support dollar sign in mysql pattern #64259).
• Preserve Histogram metric labels when exporting Prometheus metrics ([fix](metric) Preserve labels for histogram metrics to fix wrong metric name for prometheus #63485).
• Skip dropped columns during Follower FE statistics synchronization ([fix](fe) Skip dropped columns in follower stats sync #63882).

These notes focus on user-visible and operations-visible changes. Test-only commits, CI adjustments, release version updates, and purely internal changes without external symptoms are omitted. Refer to the complete 4.0.6-rc02 to 4.0.7-rc02 comparison for the full commit history.

Acknowledgments

Thanks to all contributors whose pull requests are included in this release:

@924060929 @airborne12 @BiteTheDDDDt @bobhan1 @CalvinKirs @csun5285 @dataroaring @deardeng @eldenmoon @englefly @feiniaofeiafei @felixwluo @foxtail463 @freemandealer @Gabriel39 @gavinchou @Hastyshell @HonestManXin @iaorekhov-1980 @jacktengg @Jungzhang @liaoxin01 @liutang123 @morningman @morrySnow @mrhhsg @Mryange @raghav-reglobe @seawinde @sollhui @starocean999 @suxiaogang223 @wyxxxcat @yiguolei @yoock @yujun777 @Yukang-Lian @zclllyybb @zhangrq5 @zhangstar333 @zhaorongsheng