[Bug Report] Isaac Overconsumes RAM, Significant Memory Bottleneck

[garylvov]

Describe the bug

Hi!

When using complex scenes with Isaac Lab (for example, the kitchen libero scenes from https://github.com/LightwheelAI/LW-BenchHub), Isaac Lab becomes very RAM hungry. My personal computer has 128 gigs of memory, and more than 256 parallel complex kitchen scenes completely exhaust this memory (even when applying mesh decimation, merging USD prims, etc). Meanwhile, my VRAM sits at a calm 8 gigs of utilization, so the RAM is the bottleneck for complex scenes, which I personally found very surprising; as why would each gig of VRAM need 16 gigs of corresponding CPU memory?

When I look into it, most of this memory is apparently left "cold", so it's allocated, and then as far as I can tell, no longer accessed. I've done some tests with a terabyte-sized virtual swap that works to some extent without significantly decreasing the simulation speed, which further suggests that this memory is left cold.

While a comedically large swap file might work locally, it's very hard to convince the admins of my university cluster to enable such a large swap file, so this makes it very challenging to run any large scale experiments.

I've found that I can unceremoniously evict some of the RAM usage after environment creation, and the simulator seems to continue to run fine. This is helpful as I can spawn environments, at one GPU per time, freeing up the allocated, but "cold" RAM after each GPU has it's environments spawn. However, this adds a complex barrier pattern, and significant overhead to each startup time. Furthermore, this strategy loses its effectiveness as the RAM is slowly filled up, and freeing ram post scene creation isn't feasible as the initial creation leads to an OOM of the RAM, prior to an opputunity where it can be freed.

It could also be possible that this unceremonious post-env creation RAM eviction leads to some physics solver fault that I haven't yet discovered (maybe some memory needs to lie dormant?). It could also be largely to do with the underlying Isaac sim, but frankly, it's hard for me to tell.

Anyways, although I'm not certain about root cause, or if parts of the allocated RAM are completely cold (although parts certainly seem to be!), this behavior is definitely sus, so I think it warrants this issue and further investigation. I'm also open to suggestions on how to optimize the scene USDs better, as even with highly aggressive mesh decimation and merging/removal of prims, I don't have very significant RAM savings. However, I think it should be possible for Isaac Lab to work well even with relatively complex scenes (like those in LW-benchhub) with a high degree of parallelization (after all, these complex scenes are a kitchen backdrop, table, and some items on the table)!

In my repro, I try to demonstrate that for some more complex scenes, nearly 40% of the RAM utilization could be apparently evicted.

Steps to reproduce

I made a standalone gist here, to show 46.2% of post-build RSS can be freed: https://gist.github.com/garylvov/9eec355c3071749f9c5f55587298233d

It can be run with

python isaac_ram_repro.py --headless --enable_cameras --num_envs 1024 --num_props 50

Obtaining the following info.

[INFO]: Completed setting up the environment...
[  8.25 GB  +    0 MB  T+  11.2s]   <- ManagerBasedRLEnv.__init__ (10.89s)
[  8.25 GB  +    0 MB  T+  11.2s] <- build_env
[  8.25 GB  +    0 MB  T+  11.2s] -> first env.reset
[  8.25 GB  +    0 MB  T+  11.2s]   -> ManagerBasedRLEnv.reset
[  8.29 GB  +   38 MB  T+  12.3s]   <- ManagerBasedRLEnv.reset (1.16s)
[  8.29 GB  +    0 MB  T+  12.3s] <- first env.reset
[  8.29 GB  +    0 MB  T+  12.4s] cleared refs (cold baseline)
[  8.29 GB  +    1 MB  T+  12.4s] -> ManagerBasedRLEnv.step
[  8.39 GB  +  101 MB  T+  12.6s] <- ManagerBasedRLEnv.step (0.18s)
[  8.39 GB  +    0 MB  T+  12.6s] -> ManagerBasedRLEnv.step
[  8.30 GB  -   96 MB  T+  12.8s] <- ManagerBasedRLEnv.step (0.17s)
[  8.30 GB  +    0 MB  T+  12.8s] -> ManagerBasedRLEnv.step
[  8.42 GB  +  128 MB  T+  12.9s] <- ManagerBasedRLEnv.step (0.17s)
[  8.42 GB  +    0 MB  T+  12.9s] -> ManagerBasedRLEnv.step
[  8.30 GB  -  128 MB  T+  13.1s] <- ManagerBasedRLEnv.step (0.18s)
[  8.30 GB  +    0 MB  T+  13.1s] -> ManagerBasedRLEnv.step
[  8.42 GB  +  128 MB  T+  13.3s] <- ManagerBasedRLEnv.step (0.17s)
[  8.42 GB  +    0 MB  T+  13.3s] -> ManagerBasedRLEnv.step
[  8.30 GB  -  128 MB  T+  13.5s] <- ManagerBasedRLEnv.step (0.17s)
[  8.30 GB  +    0 MB  T+  13.5s] -> ManagerBasedRLEnv.step
[  8.30 GB  +    0 MB  T+  13.6s] <- ManagerBasedRLEnv.step (0.17s)
[  8.30 GB  +    0 MB  T+  13.6s] -> ManagerBasedRLEnv.step
[  8.30 GB  +    0 MB  T+  13.8s] <- ManagerBasedRLEnv.step (0.17s)
[  8.30 GB  +    0 MB  T+  13.8s] -> ManagerBasedRLEnv.step
[  8.30 GB  +    0 MB  T+  14.0s] <- ManagerBasedRLEnv.step (0.17s)
[  8.30 GB  +    0 MB  T+  14.0s] -> ManagerBasedRLEnv.step
[  8.30 GB  +    0 MB  T+  14.1s] <- ManagerBasedRLEnv.step (0.18s)
[  8.30 GB  +    0 MB  T+  14.1s] after 10 warm steps (random actions)

Cold bytes: 3.81 GB across 1132 eligible VMAs
  0x610bb7284000-0x610c26922000  sz= 1782 MB  rss= 1520 MB  cold= 1237 MB
  0x7dd074fcf000-0x7dd079f6a000  sz=   79 MB  rss=   77 MB  cold=   77 MB
  0x7dd3187a0000-0x7dd31de22000  sz=   86 MB  rss=   69 MB  cold=   69 MB
  0x7dd474000000-0x7dd477e95000  sz=   62 MB  rss=   62 MB  cold=   61 MB
  0x7dd494000000-0x7dd497e3e000  sz=   62 MB  rss=   62 MB  cold=   61 MB
  0x7dd460000000-0x7dd463ffc000  sz=   63 MB  rss=   63 MB  cold=   61 MB
  0x7dc1fa541000-0x7dc1fe000000  sz=   58 MB  rss=   58 MB  cold=   58 MB
  0x7dd40c000000-0x7dd40fda8000  sz=   61 MB  rss=   61 MB  cold=   58 MB
  0x7dd478000000-0x7dd47bfc3000  sz=   63 MB  rss=   63 MB  cold=   56 MB
  0x7dd443fc2000-0x7dd447e6d000  sz=   62 MB  rss=   62 MB  cold=   55 MB

Single-VMA demo: evicting 0x7dd474000000-0x7dd477e95000 (62 MB, cold=62 MB)
[  8.24 GB  -   59 MB  T+  14.4s] after single-VMA eviction

Heap-carve: 0x610bb7284000-0x610c26922000 (1782 MB, rss=1520 MB, cold=1238 MB)
  scanning 27 x 64 MB chunks (sample 4 MB each)...
  content scan: 7.9s
  classifier: 18 safe chunks, cluster=9 chunks at bottom (576 MB skipped)
  [heap-carve   1/18] freed=63 MB  RSS=8.19 GB
  [heap-carve   2/18] freed=126 MB  RSS=8.13 GB
  [heap-carve   3/18] freed=190 MB  RSS=8.06 GB
  [heap-carve   4/18] freed=254 MB  RSS=8.00 GB
  [heap-carve   5/18] freed=305 MB  RSS=7.95 GB
  [heap-carve   6/18] freed=355 MB  RSS=7.91 GB
  [heap-carve   7/18] freed=396 MB  RSS=7.87 GB
  [heap-carve   8/18] freed=456 MB  RSS=7.81 GB
  [heap-carve   9/18] freed=507 MB  RSS=7.76 GB
  [heap-carve  10/18] freed=555 MB  RSS=7.71 GB
  [heap-carve  11/18] freed=606 MB  RSS=7.66 GB
  [heap-carve  12/18] freed=650 MB  RSS=7.62 GB
  [heap-carve  13/18] freed=696 MB  RSS=7.58 GB
  [heap-carve  14/18] freed=755 MB  RSS=7.52 GB
  [heap-carve  15/18] freed=817 MB  RSS=7.46 GB
  [heap-carve  16/18] freed=871 MB  RSS=7.41 GB
  [heap-carve  17/18] freed=929 MB  RSS=7.35 GB
  [heap-carve  18/18] freed=972 MB  RSS=7.31 GB
  heap-carve done in 0.6s: freed 956 MB  (0 errors)
[  7.31 GB  -  949 MB  T+  23.0s] after heap-carve eviction

Bulk-evicting 106 cold anon VMAs (total cold ~2414 MB)...
  [bulk    5/106] freed=   +332 MB  RSS=6.99 GB
  [bulk   10/106] freed=   +642 MB  RSS=6.69 GB
  [bulk   15/106] freed=   +912 MB  RSS=6.42 GB
  [bulk   20/106] freed=  +1178 MB  RSS=6.16 GB
  [bulk   25/106] freed=  +1402 MB  RSS=5.94 GB
  [bulk   30/106] freed=  +1645 MB  RSS=5.71 GB
  [bulk   35/106] freed=  +1858 MB  RSS=5.50 GB
  [bulk   40/106] freed=  +2036 MB  RSS=5.33 GB
  [bulk   45/106] freed=  +2151 MB  RSS=5.21 GB
  [bulk   50/106] freed=  +2233 MB  RSS=5.13 GB
  [bulk   55/106] freed=  +2297 MB  RSS=5.07 GB
  [bulk   60/106] freed=  +2358 MB  RSS=5.01 GB
  [bulk   65/106] freed=  +2390 MB  RSS=4.98 GB
  [bulk   70/106] freed=  +2423 MB  RSS=4.95 GB
  [bulk   75/106] freed=  +2456 MB  RSS=4.92 GB
  [bulk   80/106] freed=  +2488 MB  RSS=4.88 GB
  [bulk   85/106] freed=  +2519 MB  RSS=4.85 GB
  [bulk   90/106] freed=  +2547 MB  RSS=4.83 GB
  [bulk   95/106] freed=  +2576 MB  RSS=4.80 GB
  [bulk  100/106] freed=  +2598 MB  RSS=4.78 GB
  [bulk  105/106] freed=  +2618 MB  RSS=4.76 GB
  [bulk  106/106] freed=  +2622 MB  RSS=4.75 GB
[  4.75 GB  - 2622 MB  T+  24.1s] after bulk eviction
[  4.76 GB  +    2 MB  T+  24.1s] -> ManagerBasedRLEnv.step
[  5.67 GB  +  933 MB  T+  24.3s] <- ManagerBasedRLEnv.step (0.22s)
[  5.67 GB  +    0 MB  T+  24.3s] -> ManagerBasedRLEnv.step
[  5.66 GB  -   12 MB  T+  24.5s] <- ManagerBasedRLEnv.step (0.17s)
[  5.66 GB  +    0 MB  T+  24.5s] -> ManagerBasedRLEnv.step
[  5.66 GB  +    6 MB  T+  24.7s] <- ManagerBasedRLEnv.step (0.18s)
[  5.66 GB  +    0 MB  T+  24.7s] -> ManagerBasedRLEnv.step
[  5.66 GB  -    6 MB  T+  24.8s] <- ManagerBasedRLEnv.step (0.17s)
[  5.66 GB  +    0 MB  T+  24.8s] -> ManagerBasedRLEnv.step
[  5.65 GB  -    2 MB  T+  25.0s] <- ManagerBasedRLEnv.step (0.17s)
[  5.65 GB  +    0 MB  T+  25.0s] -> ManagerBasedRLEnv.step
[  5.67 GB  +   17 MB  T+  25.2s] <- ManagerBasedRLEnv.step (0.18s)
[  5.67 GB  +    0 MB  T+  25.2s] -> ManagerBasedRLEnv.step
[  5.67 GB  -    0 MB  T+  25.4s] <- ManagerBasedRLEnv.step (0.18s)
[  5.67 GB  +    0 MB  T+  25.4s] -> ManagerBasedRLEnv.step
[  5.70 GB  +   28 MB  T+  25.6s] <- ManagerBasedRLEnv.step (0.22s)
[  5.70 GB  +    0 MB  T+  25.6s] -> ManagerBasedRLEnv.step
[  5.70 GB  -    0 MB  T+  25.8s] <- ManagerBasedRLEnv.step (0.18s)
[  5.70 GB  +    0 MB  T+  25.8s] -> ManagerBasedRLEnv.step
[  5.69 GB  -    5 MB  T+  25.9s] <- ManagerBasedRLEnv.step (0.18s)
[  5.69 GB  +    1 MB  T+  25.9s] after 10 pre-reset steps
[  5.69 GB  +    0 MB  T+  25.9s] -> ManagerBasedRLEnv.reset
[  5.69 GB  +    4 MB  T+  26.0s] <- ManagerBasedRLEnv.reset (0.03s)
[  5.69 GB  +    0 MB  T+  26.0s] after env.reset()
[  5.69 GB  +    0 MB  T+  26.0s] -> ManagerBasedRLEnv.step
[  5.70 GB  +    8 MB  T+  26.2s] <- ManagerBasedRLEnv.step (0.19s)
[  5.70 GB  +    0 MB  T+  26.2s] -> ManagerBasedRLEnv.step
[  5.70 GB  -    3 MB  T+  26.3s] <- ManagerBasedRLEnv.step (0.18s)
[  5.70 GB  +    0 MB  T+  26.3s] -> ManagerBasedRLEnv.step
[  5.70 GB  -    2 MB  T+  26.5s] <- ManagerBasedRLEnv.step (0.17s)
[  5.70 GB  +    0 MB  T+  26.5s] -> ManagerBasedRLEnv.step
[  5.69 GB  -    5 MB  T+  26.7s] <- ManagerBasedRLEnv.step (0.17s)
[  5.69 GB  +    0 MB  T+  26.7s] -> ManagerBasedRLEnv.step
[  5.69 GB  -    3 MB  T+  26.9s] <- ManagerBasedRLEnv.step (0.17s)
[  5.69 GB  +    0 MB  T+  26.9s] -> ManagerBasedRLEnv.step
[  5.69 GB  +    0 MB  T+  27.0s] <- ManagerBasedRLEnv.step (0.17s)
[  5.69 GB  +    0 MB  T+  27.0s] -> ManagerBasedRLEnv.step
[  5.69 GB  +    0 MB  T+  27.2s] <- ManagerBasedRLEnv.step (0.17s)
[  5.69 GB  +    0 MB  T+  27.2s] -> ManagerBasedRLEnv.step
[  5.69 GB  -    2 MB  T+  27.4s] <- ManagerBasedRLEnv.step (0.17s)
[  5.69 GB  +    0 MB  T+  27.4s] -> ManagerBasedRLEnv.step
[  5.69 GB  -    2 MB  T+  27.5s] <- ManagerBasedRLEnv.step (0.17s)
[  5.69 GB  +    0 MB  T+  27.5s] -> ManagerBasedRLEnv.step
[  5.68 GB  -    2 MB  T+  27.7s] <- ManagerBasedRLEnv.step (0.17s)
[  5.68 GB  +    0 MB  T+  27.7s] after 10 post-reset steps

========================================================================
DIAGNOSTIC REPORT
========================================================================
env: 1024 envs | 50 props | 3x96px cams | NVIDIA GeForce RTX 3090 Ti | kernel 6.17.0-20-generic | py 3.11.0

Top 5 phases by RSS delta:
  <- SimulationContext.reset (7.17s)                    2.96 GB   11.1s
  <- ManagerBasedRLEnv.step (0.22s)                     0.91 GB   24.3s
  <- InteractiveScene.clone_environments (2.56s)        0.51 GB    4.0s
  <- ManagerBasedRLEnv.step (0.17s)                     0.13 GB   12.9s
  <- ManagerBasedRLEnv.step (0.17s)                     0.13 GB   13.3s

Post-build RSS:          8.25 GB
Cold anon-rw bytes:      3.81 GB  (1132 VMAs, 46.2 % of post-build RSS)

Single-VMA evict:        62.6 MB  (0.74 %)
Heap carve:              0.93 GB  (11.31 %)  18 chunks, 0 err
Bulk evict:              2.56 GB  (31.02 %)  106 anon VMAs, 0 err
Total freed:             3.56 GB  (43.07 %)  RSS 7.31 -> 4.75 GB

Post-eviction (10 steps -> env.reset() -> 10 steps):
  pre-reset steps:      +959.5 MB  PASS
  env.reset():            +3.8 MB  PASS
  post-reset steps:      -11.6 MB  PASS
========================================================================
garylvov@minerva:~/projects/gigastrap$ 

-->

System Info

Describe the characteristic of your environment:

• Commit: HEAD detached at 54cf64b
• Isaac Sim Version: 5.1
• OS: 22.04
• GPU: 2x 3090, 2x 3090 TI. I also have 128gb ECC RAM, and a threadripper 7955wx
• CUDA: 13.1
• GPU Driver: 590.48.01

Additional context

Add any other context about the problem here.

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

Isaac lab uses a reasonable amount of RAM per VRAM, I think 4:1 or better would be ideal for a GPU-first framework (for complex scenes, it's currently ~16:1).

[RandomOakForest]

Thank you for posting this, @garylvov. We will review the report with the team and discuss next steps.

[garylvov]

In my further testing, I believe that I've been able to isolate this issue as part of the IsaacRtxRenderer/PhysX/OVRTX rendering process (the vanilla rendering on main, not sure what the canonical name for this pipeline is), as there doesn't seem to have a RAM leak when cameras aren't enabled.

I have a private fork, where when using a custom integration with the Madrona render, the scenes look correct without the memory leak.

I suspect that this could be the root cause of #4178

@kellyguo11 could we please pass this along to the IsaacRtxRenderer/OVRTX team?

[kellyguo11]

Thanks for looking into this @garylvov ! we will investigate on our end.

[garylvov]

Thank you!! 😸

[adahdouh-nvidia]

Hey @garylvov ,

Thanks for the detailed issue....To make this more actionable/predictable, could we narrow down which scene features drive RSS growth?

Would it be possible to run the repro with --skip_eviction and provide a small scaling table with:

num_envs | num_props | num_cameras | camera_res | peak RSS | post-build RSS | after reset RSS | after 10 steps RSS | VRAM | cold anon-rw bytes | largest RSS phase

Suggested sweep:

• num_envs: 64, 128, 256, 512, 1024
• num_props: 0, 10, 20, 50
• num_cameras: 0, 1, 3
• camera_res: 96, 192

A few targeted comparisons would be especially helpful:

1. No cameras and no --enable_cameras
2. Cameras enabled but num_props=0
3. Props/contact sensors enabled but num_cameras=0
4. Default PhysX GPU buffer settings vs the enlarged buffer settings in the repro
5. clone_in_fabric=True / replicate_physics=True where supported
6. Longer post-eviction validation, e.g. 10k+ steps with resets, contact-heavy interactions, and camera reads

tl;dr: Can you provide a feature-sweep table showing RSS growth separately for env count, props/contact sensors, cameras, camera resolution, and PhysX buffer sizes?

[garylvov]

Hi,

Without cameras, there is no leak. I confirmed this by running identical scenes (without isaac cameras or --enable_cameras with my internal integration with the Madrona renderer (and I could view the madrona images correctly, training was stable). From what I remember, changing the PhysX buffer made no difference, and the more assets (and the more complex the assets are), that the scene has, the worse the leak gets. I never tested with fewer cameras. The RSS growth happens primarily at startup; from what I remember from watching htop, it's mostly directly after/during scene creation.

Unfortunately, I don't have any spare time to run a sweep. I have an internship starting on June 8th, and before then, I have to wrap up a paper submission for my RA-ship at Brown, which has been occupying all of my time.

The repro script I shared should run out of the box with the commit I pinned without any isaac lab modifications.

Can you please run it on your side? 😅

[adahdouh-nvidia]

Hey @garylvov. I didn't run your exact case here, but tried to reproduce it using Develop on Lab and Develop on Sim.

I also tried to create a sweep repo: https://github.com/adahdouh-nvidia/lab_lw_kitchen_oom_repro and used that to run on a test setup (L40).

TL;DR: We tested the real kitchen workload without memory eviction. The reported host-RAM OOM did not reproduce. Host RAM stayed around 19.4 GB at most. The failure we reproduced was GPU memory exhaustion from 640x480 camera observations. Lowering camera resolution to 320x240 or disabling camera observations allowed 256 environments to complete. At this point, we could not reproduce host-memory OOM .

Perhaps a separate follow-up for camera observation GPU memory scaling is the real underlying issue here.

Would you be able to perhaps try and rerun quickly with lower resolution cameras? The fact that you ran with no cameras and no issues is also telling.