[Performance] Streamline collector inner loop carrier update

[vmoens]

Stack from ghstack (oldest at bottom):

• [Test] Add tests and benchmarks for collector throughput optimizations #3567
• [Performance] Enable torch.compile(fullgraph=True) for step_and_maybe_reset #3566
• [Performance] Add fast path for step() and TransformedEnv._step() when _trust_step_output is set #3565
• -> [Performance] Streamline collector inner loop carrier update #3564
• [Performance] Add update_traj_ids flag to Collector to skip trajectory tracking #3563
• [Performance] Add _trust_step_output flag to skip step validation overhead #3562
• [Performance] Add out= parameter to _StepMDP for output buffer reuse #3561
• [Performance] Add _skip_maybe_reset flag to bypass auto-reset in step_and_maybe_reset #3560

Pre-computes _env_returns_carrier flag to avoid per-step identity check.
Skips collector data .copy() when update_traj_ids is False (no mutation
risk). Uses internal _get_str/_set_str for collector data transfer to
bypass public API validation overhead.

Made-with: Cursor

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3564

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 9 New Failures, 3 Cancelled Jobs

As of commit 2487bcb with merge base 0a1aea6 ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  ../../../../tmp/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_with_rb_cuda[200-img_shape1-large_batch]
• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh)
  test/test_collectors.py::TestDynamicEnvs::test_dynamic_multisync_collector[policy_device1]
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh)
  test/test_collectors.py::TestDynamicEnvs::test_dynamic_multisync_collector[policy_device1]
• Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh)
  test/test_collectors.py::TestDynamicEnvs::test_dynamic_multisync_collector[policy_device1]
• Unit-tests on Linux / tests-cpu (3.13) / linux-job (gh)
  test/test_collectors.py::TestDynamicEnvs::test_dynamic_multisync_collector[policy_device1]
• Unit-tests on Linux / tests-cpu (3.14) / linux-job (gh)
  test/test_collectors.py::TestDynamicEnvs::test_dynamic_multisync_collector[policy_device1]
• Unit-tests on Linux / tests-gpu (3.12, 13.0, 2) / linux-job (gh)
  test/test_collectors.py::TestCollectorDevices::test_no_synchronize[False-cuda:0-cuda:0]
• Unit-tests on Linux / tests-optdeps (3.12, 13.0) / linux-job (gh)
  test/test_collectors.py::TestCollectorDevices::test_no_synchronize[False-cuda:0-cuda:0]
• Unit-tests on Linux / tests-stable-gpu (3.12, 13.0, 2) / linux-job (gh)
  test/test_collectors.py::TestCollectorDevices::test_no_synchronize[False-cuda:0-cuda:0]

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• Continuous Benchmark (PR) / CPU Pytest benchmark (gh)
  ##[error]The operation was canceled.
• Unit-tests on Linux / tests-olddeps (3.10, 11.8) / linux-job (gh)
  ##[error]The operation was canceled.
• Unit-tests on Windows / unittests-cpu (3.10, windows.4xlarge, cpu) / windows-job (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.