[Core] Data jobs able to trigger persistent mmap file leak in between job runs

[plv]

What happened + What you expected to happen

When I have a long-running Ray cluster that I’m running multiple jobs on (for iteration in development), eventually I will start getting OOMs during initial SQL read (i.e. SplitBlocks). I’m under the impression that sometimes Ray is “spilling” some data into tmpfs or shmem, but I don’t think it should persist in between jobs. Is there any way to clean up this junk in between runs so that I don’t have to restart my cluster?

I’m aware that spilling in Ray happens when object memory is high, but I have 500GiB object memory and my usage never reaches anywhere near that.

What's more is that the OOMs are always just slightly above Ray's 95% kill rate -- it's as if there is a rounding error happening somewhere.

// This is only tangentially related to datasets, I’m writing it here because my jobs only use the datasets library, and I’m wondering if this phenomenon is specific to the ray library. If this is the wrong place to post this, please let me know

Versions / Dependencies

• Ray 2.6.1 using this Docker imageray:2.6.1.7474f8-py38-cpu. You can pull down the docker image I have on top of it here, but I don't think it's relevant

• Python 3.8.13

• Running using Ray clusters (ie not in Kuberay)

• Running on a fleet of AWS m5.4xlarge and g5.2xlarge

Reproduction script

Run a script like this repeatedly. Eventually machines will OOM

import ray

ray.init()

@ray.remote()
def do_data():
   ds = ray.data.read_sql("<read huge SQL table here>") # we use this snowflake table: https://app.snowflake.com/marketplace/listing/GZTSZ290BUX1X/cybersyn-inc-cybersyn-llm-training-essentials
     .materialize().show()

ray.get(do_data.remote())

Issue Severity

Low: It annoys or frustrates me.

[z4y1b2]

I found a similar problem. Plasma object store in raylet process doesn't release mmaped memory of objects used in a finished job.

My environment is:

• MacBook Pro M2 Max, 64 GB RAM, 12 CPU Cores
• MacOS 13.5.1 (22G90)
• Python 3.11.4 arm64
• Ray version 2.6.3

My test script is:

import ray
import pyarrow as pa

schema = pa.schema([('v', pa.binary())])
def map(batch):
    lines = []
    for __ in range(len(batch['id'])):
        lines.append(b'*' * 99)
    return pa.Table.from_arrays([lines], schema=schema)

def test(n):
    ray.init()
    ray.data.DataContext.get_current().execution_options.verbose_progress = True
    ray.data.DataContext.get_current().use_push_based_shuffle = True
    parallelism = max(1, int(n * 100 / (128 * 1024 * 1024)))
    ds = ray.data.range(n, parallelism=parallelism)\
            .map_batches(map, zero_copy_batch=True)\
            .repartition(1)
    ds.materialize()

if __name__ == '__main__':
    test(700000000)

run following shell command:

ray start --head
python ./test.py

And you will find the disk usage increases, run the vmmap PID command and you will find a lot of mmaped files still exist in raylet process, and when you ray stop killing all the ray processes then the disk usage decreases.

[richardliaw]

cc @c21

[ericl]

Do you know if it's disk usage or object store memory itself that's leaked? It would be useful to post the graphs of physical disk as well as the logical object store memory usage from the Ray dashboard here

[z4y1b2]

I found a similar problem. Plasma object store in raylet process doesn't release mmaped memory of objects used in a finished job.

My environment is:

* MacBook Pro M2 Max, 64 GB RAM, 12 CPU Cores

* MacOS 13.5.1 (22G90)

* Python 3.11.4 arm64

* Ray version 2.6.3

My test script is:

import ray
import pyarrow as pa

schema = pa.schema([('v', pa.binary())])
def map(batch):
    lines = []
    for __ in range(len(batch['id'])):
        lines.append(b'*' * 99)
    return pa.Table.from_arrays([lines], schema=schema)

def test(n):
    ray.init()
    ray.data.DataContext.get_current().execution_options.verbose_progress = True
    ray.data.DataContext.get_current().use_push_based_shuffle = True
    parallelism = max(1, int(n * 100 / (128 * 1024 * 1024)))
    ds = ray.data.range(n, parallelism=parallelism)\
            .map_batches(map, zero_copy_batch=True)\
            .repartition(1)
    ds.materialize()

if __name__ == '__main__':
    test(700000000)

run following shell command:

ray start --head
python ./test.py

And you will find the disk usage increases, run the vmmap PID command and you will find a lot of mmaped files still exist in raylet process, and when you ray stop killing all the ray processes then the disk usage decreases.

I run the test script using Ray 2.7.0 (Python 3.11.5) in the same machine and the problem still exists.

Here are screenshots of Ray dashboard about the node.

Before running the script:

After running the script:

There's no visible files under '/tmp' because mmapped 'plasmaXXXXXX' files are unlinked right after its creation.

Here's the output of vmmap PID after running the script:

leak_vmmap.txt

[ericl]

Thanks for the screenshots. I think I can confirm the mmap file leaks from the following observations:

• Before: 0/2GB reported object store memory, 381GB used disk.
• After: 0/2GB reported object store memory, 516GB disk used.

That's 200GB of leaked plasma files, whereas normal object store allocation would at most create 2GB of files in /tmp (maybe additional 2GB for fallback allocation).

I'm going to tag this as P0 until further verification, since this seems like a possibly serious regression in a core component.

[anyscalesam]

@rynewang any update on this?

[rynewang]

I think Ray successfully released all object data in the object store; but then Ray did not release the mmap'd files. Further tracking down the release code...

[rynewang]

Update: I found the issue being the Plasma Client (core worker and raylet's client to the shared memory) never releases mmap'd files. This only happens on high memory pressure when the main memory is not enough and Ray is forced to allocate files and mmap them. After the memory pressure, we never munmap the files even though they are no longer used.

Actively working on PRs to tackle this. Preview: #40370 (not mature to merge yet)

[rynewang]

Tested that master branch Ray leaks on this repro script, and Ray with this PR no longer leaks. Next step: make tests fixed and @rkooo567 will merge it

[rkooo567]

The fix will be included in 2.8

[rkooo567]

we fixed this in the master, but the fix wasn't included in Ray 2.8 due to the risk of last-minute changes. The fix should be available in the master and will be available from Ray 2.9