[Feature] Weight Synchronization Schemes - Core Infrastructure

[vmoens]

Stack from ghstack (oldest at bottom):

• [Feature] Documentation #3192
• [Feature] SAC Trainer #3191
• [Feature] PPO Trainer Updates #3190
• [Feature] Trainer Algorithms - Configuration System #3189
• [Feature] Trainer Infrastructure - Timing and Utilities #3188
• [Feature] Collectors - Weight Sync Scheme Integration #3187
• [Feature] vLLM Weight Synchronization Schemes #3186
• -> [Feature] Weight Synchronization Schemes - Core Infrastructure #3185
• [Feature] Transform Module - ModuleTransform and Ray Service Refactor #3184
• [Feature] Storage Shared Initialization for Multiprocessing #3183

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3185

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 18 New Failures, 2 Cancelled Jobs, 1 Unrelated Failure

As of commit 9ccb2f3 with merge base 2f4c919 ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / CPU Pytest benchmark (gh)
  Process completed with exit code 1.
• Generate documentation / build-docs (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t c0b5daa2f7aa3266c02a9dd1681cac0243e4065739753c198660fcd0c57084dd /exec failed with exit code 1
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t b1900e418375abf66be8516bfc908b5ad0cfe431a9ff7fdfc1c5f4c63ee708fe /exec failed with exit code 1
• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t aa6e5ae3a0a77bcfe86d33a17048bcf919b2af5bf2f2996cfcff7a4930d81bbb /exec failed with exit code 1
• Libs Tests on Linux / unittests-sklearn (3.10, 12.8) / linux-job (gh)
  test/test_libs.py::TestOpenML::test_data[magic]
• Lint / python-source-and-configs / linux-job (gh)
  RuntimeError: Command docker exec -t 736a4ca8d769d11b091fad4096bfc02832eaa6b2f06bde9cc3447a8ba409c5eb /exec failed with exit code 1
• Push Binary Nightly / test-wheel-unix (linux, ubuntu-22.04, 3.9, cp39-cp39, cpu, cpu) (gh)
  TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
• Push Binary Nightly / test-wheel-unix (macos, macos-latest, 3.9, cp39-cp39, cpu, cpu) (gh)
  TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
• Push Binary Nightly / test-wheel-windows (3.9, 3.9) (gh)
  TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
• SOTA Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 8d55a40cc29e15150989bd718e6bfd4f9e90e1508d4e3699d04caa03e0e4d910 /exec failed with exit code 1
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-cpu (3.9) / linux-job (gh)
  RuntimeError: Command docker exec -t a17b9107d835a1ef608d8e5c436ccf752b0a1376ed08073bae9dbbb700ea1a24 /exec failed with exit code 1
• Unit-tests on Linux / tests-gpu (3.11, 12.8) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-olddeps (3.9, 11.6) / linux-job (gh)
  RuntimeError: Command docker exec -t dbfbaae8326edcaf806a1eb5fd5303480940ad66cc35bdbfbee0f1939150bca5 /exec failed with exit code 1
• Unit-tests on Linux / tests-optdeps (3.11, 12.8) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Windows / unittests-cpu (3.10, windows.4xlarge, cpu) / windows-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  ##[error]The operation was canceled.
• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh)
  ##[error]The operation was canceled.

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• LLM Tests on Linux / unittests (3.9, 12.8) / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.