[VL] Memory leak for version release 1.4.0

[j7nhai]

Backend

VL (Velox)

Bug description

E20250414 17:02:31.939500   222 VeloxMemoryManager.cc:392] Failed to release Velox memory manager after 43350ms as there are still outstanding memory resources.
E20250414 17:02:31.939989   222 Exceptions.h:66] Line: /data/deploy/gluten/ep/build-velox/build/velox_ep/velox/common/memory/Memory.cpp:149, Function:~MemoryManager, Expression:  pools_.size() != 0 (1 vs 0). There are unexpected alive memory pools allocated by user on memory manager destruction:
Memory Manager[capacity UNLIMITED alignment 64B usedBytes 0B number of pools 4
List of root pools:
__sys_root__ usage 0B reserved 0B peak 0B
    __sys_shared_leaf__0 usage 0B reserved 0B peak 0B
    __sys_tracing__ usage 0B reserved 0B peak 0B
    __sys_spilling__ usage 0B reserved 0B peak 0B
    __sys_caching__ usage 0B reserved 0B peak 0B
root usage 0B reserved 0B peak 3.73GB
    task.Gluten_Stage_1_TID_7976_VTID_56 usage 0B reserved 0B peak 3.73GB
        node.0 usage 0B reserved 0B peak 112.00MB
            op.0.0.0.TableScan usage 0B reserved 0B peak 104.66MB
	refcount 2
Memory Allocator[MALLOC capacity UNLIMITED allocated bytes 3145728 allocated pages 0 mapped pages 0]
ARBITRATOR[GLUTEN] CAPACITY 8388608.00TB numRequests 0 numRunning 0 numSucceded 0 numAborted 0 numFailures 0 numNonReclaimableAttempts 0 reclaimedFreeCapacity 0B reclaimedUsedCapacity 0B maxCapacity 0B freeCapacity 0B freeReservedCapacity 0B], Source: RUNTIME, ErrorCode: INVALID_STATE
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
  what():  Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: pools_.size() != 0 (1 vs 0). There are unexpected alive memory pools allocated by user on memory manager destruction:
Memory Manager[capacity UNLIMITED alignment 64B usedBytes 0B number of pools 4
List of root pools:
__sys_root__ usage 0B reserved 0B peak 0B
    __sys_shared_leaf__0 usage 0B reserved 0B peak 0B
    __sys_tracing__ usage 0B reserved 0B peak 0B
    __sys_spilling__ usage 0B reserved 0B peak 0B
    __sys_caching__ usage 0B reserved 0B peak 0B
root usage 0B reserved 0B peak 3.73GB
    task.Gluten_Stage_1_TID_7976_VTID_56 usage 0B reserved 0B peak 3.73GB
        node.0 usage 0B reserved 0B peak 112.00MB
            op.0.0.0.TableScan usage 0B reserved 0B peak 104.66MB
	refcount 2
Memory Allocator[MALLOC capacity UNLIMITED allocated bytes 3145728 allocated pages 0 mapped pages 0]
ARBITRATOR[GLUTEN] CAPACITY 8388608.00TB numRequests 0 numRunning 0 numSucceded 0 numAborted 0 numFailures 0 numNonReclaimableAttempts 0 reclaimedFreeCapacity 0B reclaimedUsedCapacity 0B maxCapacity 0B freeCapacity 0B freeReservedCapacity 0B]
Retriable: False
Function: ~MemoryManager
File: /data/deploy/gluten/ep/build-velox/build/velox_ep/velox/common/memory/Memory.cpp
Line: 149
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  0x0000000001aef77e
# 4  _ZN6gluten18VeloxMemoryManagerD1Ev
# 5  _ZN6gluten18VeloxMemoryManagerD0Ev
# 6  _ZN6gluten13MemoryManager7releaseEPS0_
# 7  Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_release
# 8  0x00007f1a157a6da7

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

[zhouyuan]

Hi @j7nhai thanks for reporting! would you please provide more details to help us to reproduce this issue? e.g. some sql codes.

thanks, -yuan

[nimesh1601]

@zhouyuan faced the same issue
Sample sql
WITH cdl_summary AS ( SELECT user_id, CASE WHEN <TRACKING_LABEL_COL> = 'store_front' THEN 'storefront' ELSE <TRACKING_LABEL_COL> END AS source, SUM(CASE WHEN name IN (<IMPRESSION_EVENTS>) AND datestr BETWEEN :START_7D AND :END_DATE THEN 1 ELSE 0 END) AS impression_count_7d, SUM(CASE WHEN name IN (<IMPRESSION_EVENTS>) AND datestr BETWEEN :START_14D AND :END_DATE THEN 1 ELSE 0 END) AS impression_count_14d, SUM(CASE WHEN name IN (<IMPRESSION_EVENTS>) AND datestr BETWEEN :START_28D AND :END_DATE THEN 1 ELSE 0 END) AS impression_count_28d, SUM(CASE WHEN name IN (<IMPRESSION_EVENTS>) AND datestr BETWEEN :START_56D AND :END_DATE THEN 1 ELSE 0 END) AS impression_count_56d, SUM(CASE WHEN name IN (<IMPRESSION_EVENTS>) AND datestr BETWEEN :START_112D AND :END_DATE THEN 1 ELSE 0 END) AS impression_count_112d, SUM(CASE WHEN name IN (<CLICK_EVENTS>) AND datestr BETWEEN :START_7D AND :END_DATE THEN 1 ELSE 0 END) AS click_count_7d, SUM(CASE WHEN name IN (<CLICK_EVENTS>) AND datestr BETWEEN :START_14D AND :END_DATE THEN 1 ELSE 0 END) AS click_count_14d, SUM(CASE WHEN name IN (<CLICK_EVENTS>) AND datestr BETWEEN :START_28D AND :END_DATE THEN 1 ELSE 0 END) AS click_count_28d, SUM(CASE WHEN name IN (<CLICK_EVENTS>) AND datestr BETWEEN :START_56D AND :END_DATE THEN 1 ELSE 0 END) AS click_count_56d, SUM(CASE WHEN name IN (<CLICK_EVENTS>) AND datestr BETWEEN :START_112D AND :END_DATE THEN 1 ELSE 0 END) AS click_count_112d, SUM(CASE WHEN name IN (<ORDER_EVENTS>) AND datestr BETWEEN :START_7D AND :END_DATE THEN 1 ELSE 0 END) AS order_count_7d, SUM(CASE WHEN name IN (<ORDER_EVENTS>) AND datestr BETWEEN :START_14D AND :END_DATE THEN 1 ELSE 0 END) AS order_count_14d, SUM(CASE WHEN name IN (<ORDER_EVENTS>) AND datestr BETWEEN :START_28D AND :END_DATE THEN 1 ELSE 0 END) AS order_count_28d, SUM(CASE WHEN name IN (<ORDER_EVENTS>) AND datestr BETWEEN :START_56D AND :END_DATE THEN 1 ELSE 0 END) AS order_count_56d, SUM(CASE WHEN name IN (<ORDER_EVENTS>) AND datestr BETWEEN :START_112D AND :END_DATE THEN 1 ELSE 0 END) AS order_count_112d, AVG(CASE WHEN name = 'marketplace_scrolled' AND datestr BETWEEN :START_7D AND :END_DATE THEN 0 WHEN name IN ('feed_item_card_scrolled','feed_item_dish_card_scrolled') AND datestr BETWEEN :START_7D AND :END_DATE THEN feed.display_item_position ELSE NULL END) AS avg_scroll_depth_7d, MAX(CASE WHEN name IN (<INTERACTION_EVENTS>) AND datestr BETWEEN :START_112D AND :END_DATE THEN epoch_ms ELSE NULL END) AS latest_interaction_time FROM <SCHEMA_CDL>.<TABLE_CDL> cdl WHERE TRUE AND datestr BETWEEN :START_112D AND :END_DATE AND name IN (<IMPRESSION_EVENTS>, <CLICK_EVENTS>, <ORDER_EVENTS>) AND COALESCE(<SESSION_ID_COL>, user_id) IS NOT NULL AND is_first_event = TRUE AND user_id IS NOT NULL AND user_id <> '' AND <FEED_CONTEXT_COL> IN ('home','vertical','allstores','all_stores') GROUP BY 1,2 ), xlb_summary AS ( SELECT user_id, CASE WHEN <TRACKING_LABEL_COL> = 'store_front' THEN 'storefront' ELSE <TRACKING_LABEL_COL> END AS source, MAX(CASE WHEN name IN (<INTERACTION_EVENTS>) AND datestr BETWEEN :START_112D AND :END_DATE THEN epoch_ms ELSE NULL END) AS latest_interaction_time FROM <SCHEMA_XLB>.<TABLE_XLB> xlb WHERE TRUE GROUP BY 1,2 ), data_summary AS ( SELECT COALESCE(cdl.user_id, xlb.user_id) AS user_id, COALESCE(cdl.source, xlb.source) AS source, COALESCE(cdl.impression_count_7d, 0) + COALESCE(xlb.impression_count_7d, 0) AS impression_count_7d, GREATEST(cdl.latest_interaction_time, xlb.latest_interaction_time) AS latest_interaction_time FROM cdl_summary cdl FULL OUTER JOIN xlb_summary xlb ON cdl.user_id = xlb.user_id AND cdl.source = xlb.source ), summary_agg AS ( SELECT user_id, SUM(impression_count_7d) AS total_impression_7d, SUM(order_count_112d) AS total_order_112d FROM data_summary GROUP BY 1 ) INSERT OVERWRITE TABLE <SCHEMA_TARGET>.<TABLE_TARGET> PARTITION (datestr) SELECT CONCAT(d.user_id, '|', d.source) AS uuid, d.user_id, d.source, d.impression_count_7d, agg.total_impression_7d, d.latest_interaction_time, :END_DATE AS datestr FROM data_summary d JOIN summary_agg agg ON d.user_id = agg.user_id WHERE d.source IS NOT NULL;

[atongpu]

@zhouyuan I also encountered a similar issue.

Memory Allocator[MALLOC capacity UNLIMITED allocated bytes 2097152 allocated pages 0 mapped pages 0]
ARBITRATOR[GLUTEN] CAPACITY 8388608.00TB numRequests 0 numRunning 0 numSucceded 0 numAborted 0 numFailures 0 numNonReclaimableAttempts 0 reclaimedFreeCapacity 0B reclaimedUsedCapacity 0B maxCapacity 0B freeCapacity 0B freeReservedCapacity 0B], Source: RUNTIME, ErrorCode: INVALID_STATE
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
what(): Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: pools_.size() != 0 (1 vs 0). There are unexpected alive memory pools allocated by user on memory manager destruction:
Memory Manager[capacity UNLIMITED alignment 64B usedBytes 0B number of pools 4
List of root pools:
sys_root usage 0B reserved 0B peak 0B
__sys_shared_leaf__0 usage 0B reserved 0B peak 0B
sys_tracing usage 0B reserved 0B peak 0B
sys_caching usage 0B reserved 0B peak 0B
sys_spilling usage 0B reserved 0B peak 0B
root usage 0B reserved 0B peak 280.00MB
task.Gluten_Stage_0_TID_38684_VTID_92 usage 0B reserved 0B peak 280.00MB
node.0 usage 0B reserved 0B peak 280.00MB
op.0.0.0.TableScan usage 0B reserved 0B peak 276.95MB
refcount 2
Memory Allocator[MALLOC capacity UNLIMITED allocated bytes 2097152 allocated pages 0 mapped pages 0]
ARBITRATOR[GLUTEN] CAPACITY 8388608.00TB numRequests 0 numRunning 0 numSucceded 0 numAborted 0 numFailures 0 numNonReclaimableAttempts 0 reclaimedFreeCapacity 0B reclaimedUsedCapacity 0B maxCapacity 0B freeCapacity 0B freeReservedCapacity 0B]
Retriable: False
Function: ~MemoryManager

[lifulong]

spark.gluten.sql.columnar.backend.velox.asyncTimeoutOnTaskStopping=120000
We also encountered the same issue. After adjusting the parameter mentioned above to 120 seconds, the problem was resolved.
our problem job stage has 12000+ task， and each task has multi input split file

[lifulong]

spark.gluten.sql.columnar.backend.velox.asyncTimeoutOnTaskStopping=120000 We also encountered the same issue. After adjusting the parameter mentioned above to 120 seconds, the problem was resolved. our problem job stage has 12000+ task， and each task has multi input split file

Looks like it related to prefetch and async data load, async load hold table scan memory pool, it's release time may be delayed, not memory leak

[JunhyungSong]

We also see this issue. Is there a plan to fix this? Does the latest version of Gluten not have this issue any more?

[philo-he]

FYI. This issue has been completely fixed in Gluten 1.7 by introducing GlutenDirectBufferedInput.h and registering the custom direct buffered input for Velox to use.