[Feature] SAC Trainer

[vmoens]

Stack from ghstack (oldest at bottom):

• [Feature] Documentation #3192
• -> [Feature] SAC Trainer #3191
• [Feature] PPO Trainer Updates #3190
• [Feature] Trainer Algorithms - Configuration System #3189
• [Feature] Trainer Infrastructure - Timing and Utilities #3188
• [Feature] Collectors - Weight Sync Scheme Integration #3187
• [Feature] vLLM Weight Synchronization Schemes #3186
• [Feature] Weight Synchronization Schemes - Core Infrastructure #3185
• [Feature] Transform Module - ModuleTransform and Ray Service Refactor #3184
• [Feature] Storage Shared Initialization for Multiprocessing #3183

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3191

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 19 New Failures, 1 Cancelled Job, 1 Unrelated Failure

As of commit ec4e903 with merge base 2f4c919 ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / CPU Pytest benchmark (gh)
  Process completed with exit code 2.
• Generate documentation / build-docs (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 3f042688e18b1fb2800ece6d57124c27eff0a1ed5ce8883b94fea80f04a1974f /exec failed with exit code 1
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t da5698125a1221fa2e44c61e538bcaa4c4362e48365a925e0ea54f46b2d9a55e /exec failed with exit code 1
• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t bd6d2fb7e7713992d9fe392278681f0b4e22f29dc650c88a8d2823c2cdd363f9 /exec failed with exit code 1
• Libs Tests on Linux / unittests-sklearn (3.10, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 121be95354d3b9434fa09e4cb04b6496432753ee40de229c5fc9f13b38a9a951 /exec failed with exit code 2
• Lint / python-source-and-configs / linux-job (gh)
  RuntimeError: Command docker exec -t 9df80c77d083444d991e1ca822251a6e0c04ff225456e586d54bbe968189f6b0 /exec failed with exit code 1
• Push Binary Nightly / test-wheel-unix (linux, ubuntu-22.04, 3.9, cp39-cp39, cpu, cpu) (gh)
  TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
• Push Binary Nightly / test-wheel-unix (macos, macos-latest, 3.9, cp39-cp39, cpu, cpu) (gh)
  TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
• Push Binary Nightly / test-wheel-windows (3.9, 3.9) (gh)
  TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
• SOTA Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t c69b0574085ff6996066e43bc0cdffc8ed71423d5b29bac0b141a736940940ab /exec failed with exit code 1
• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-cpu (3.9) / linux-job (gh)
  RuntimeError: Command docker exec -t 11488f4ce80625f97462d8a4e4bcdad28d7c77de023a6a5d7bfb8c0291d2ce49 /exec failed with exit code 1
• Unit-tests on Linux / tests-gpu (3.11, 12.8) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-olddeps (3.9, 11.6) / linux-job (gh)
  RuntimeError: Command docker exec -t 3a312d4f7f95ff31419aef74d9f1073a759ca0e2e253b0f1529534928833ad51 /exec failed with exit code 1
• Unit-tests on Linux / tests-optdeps (3.11, 12.8) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Windows / unittests-cpu (3.10, windows.4xlarge, cpu) / windows-job (gh)
  Process completed with exit code 2.

CANCELLED JOB - The following job was cancelled. Please retry:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• LLM Tests on Linux / unittests (3.9, 12.8) / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.