[Ray Serve] Memory leak in 2.6

[akshay-anyscale]

What happened + What you expected to happen

See a memory leak in HTTPProxy on multiple serve clusters in Anyscale Workspaces/Services. This regression was likely introduced in 2.6

Versions / Dependencies

Ray 2.6.1

Reproduction script

Run a serve deployment and send requests repeatedly

Issue Severity

High: It blocks me from completing my task.

[zhe-thoughts]

When did this behavior start to happen? E.g. is it between 2.6.0 and 2.6.1?

[akshay-anyscale]

Between 2.5.1 and 2.6

[edoakes]

After much profiling and pain, we discovered that the root cause is a bug in the Ray core streaming object ref generator code causing the "end of stream" object to never be removed from the in memory object store.

Verified by:

1. Adding log statements for the number of objects on each request and seeing that it goes up monotonically by 1 each request.
2. Logging all objects being deleted from in memory store, the end of stream object is never deleted.
3. The destructor for the stream object is being called, so it's not an issue in the Python layer.

@rkooo567 is taking it from here

[edoakes]

Also verified the leak is not present with RAY_SERVE_ENABLE_EXPERIMENTAL_STREAMING=0, so we can recommend that to any users with problems.

[edoakes]

For posterity, some flamegraphs taken over time as the leak occurred for the leaking (flamegraph_{i}) and non-leaking (ffs_off_flamegraph_{i}) cases.

flamegraphs.zip

[rkooo567]

I am investigating it now. Btw, how did you guys find the memory leak? Do you run release tests with memory usage now?

[edoakes]

Re-opening until cherry picked

[edoakes]

NVM it's already cherry-picked!