[Refactor] Weight sync schemes refactor

[vmoens]

Stack from ghstack (oldest at bottom):

• [Doc] Huge doc refactoring #3231
• -> [Refactor] Weight sync schemes refactor #3230
• [BugFix] Fix schemes and refactor collectors to make them readable #3226
• [Refactor] Non-daemonic processes in PEnv #3225

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3230

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 19 New Failures

As of commit a044032 with merge base 9ca0e40 ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Process completed with exit code 139.
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 36e127421effaeff6449102022e6c539b1325c24386c31d7b9ad0b842e4ca35d /exec failed with exit code 1
• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh)
  RuntimeError: SumSegmentTreeFp32 is not available. See warning above.
• Lint / python-source-and-configs / linux-job (gh)
  RuntimeError: Command docker exec -t 55503634e8664c34d983217102a0e3ec07fe6b577db97f3cddecc23996596663 /exec failed with exit code 1
• Push Binary Nightly / build-wheel-unix (linux, ubuntu-22.04, 3.9, cp39-cp39, cpu, cpu) (gh)
  Process completed with exit code 1.
• Push Binary Nightly / build-wheel-unix (macos, macos-latest, 3.9, cp39-cp39, cpu, cpu) (gh)
  Process completed with exit code 1.
• Push Binary Nightly / build-wheel-windows (3.9, 3.9) (gh)
  Process completed with exit code 1.
• Push Binary Nightly / test-wheel-unix (linux, ubuntu-22.04, 3.9, cp39-cp39, cpu, cpu) (gh)
  Process completed with exit code 1.
• Push Binary Nightly / test-wheel-unix (macos, macos-latest, 3.9, cp39-cp39, cpu, cpu) (gh)
  Process completed with exit code 1.
• Push Binary Nightly / test-wheel-windows (3.9, 3.9) (gh)
  Process completed with exit code 1.
• SOTA Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 8bcb5f648528c4745e47d8b34c8b4eb7e4b730fbe897c4336fe04ccf5b1dfd23 /exec failed with exit code 1
• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh)
  RuntimeError: Command docker exec -t 8fcfef2cd5e303d96fd5a4eb40092edc9c2232e5035703885fb3e6e279e331b6 /exec failed with exit code 2
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh)
  RuntimeError: Command docker exec -t f6c763d7696a82444ab1c0345bd1cfa06a9e91f662fc5e754c66f535086347a7 /exec failed with exit code 2
• Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh)
  RuntimeError: Command docker exec -t 031363a385aba5509b6523628c66aefdc581a45236aee85177ef8947d5985d70 /exec failed with exit code 2
• Unit-tests on Linux / tests-cpu (3.9) / linux-job (gh)
  RuntimeError: Command docker exec -t b7dcb637cc0f52a32d181f62b32bc917e2b87fff6c4be6b9ff02e6ab6ae9e54c /exec failed with exit code 2
• Unit-tests on Linux / tests-gpu (3.11, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 0bb3b1c322b8a5bb41a8fc14dadbc05c9497f7b8e41f30eea482f2a2f530f18f /exec failed with exit code 2
• Unit-tests on Linux / tests-olddeps (3.9, 11.6) / linux-job (gh)
  RuntimeError: Command docker exec -t ac92e34ac76c5736918e97471675aa41a2cdc539421797b2e61a6afa2f4ecc0a /exec failed with exit code 2
• Unit-tests on Linux / tests-optdeps (3.11, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 458c0aa8480411b05931f8bc2e0686bd4a56e84019a6387d7c99cf28568af539 /exec failed with exit code 2
• Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 60018d048219c9c6887e175e608fff86861726c9bd117259f9666ba619a75a5e /exec failed with exit code 2

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 154. Improved: $\large\color{#35bf28}16$. Worsened: $\large\color{#d91a1a}12$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_tensor_to_bytestream_speed[pickle]	84.0476μs	83.0999μs	12.0337 KOps/s	11.7390 KOps/s	$\color{#35bf28}+2.51\%$
test_tensor_to_bytestream_speed[torch.save]	0.1421ms	0.1415ms	7.0692 KOps/s	6.9542 KOps/s	$\color{#35bf28}+1.65\%$
test_tensor_to_bytestream_speed[untyped_storage]	0.1232s	0.1230s	8.1316 Ops/s	8.3807 Ops/s	$\color{#d91a1a}-2.97\%$
test_tensor_to_bytestream_speed[numpy]	2.7987μs	2.7945μs	357.8452 KOps/s	351.3924 KOps/s	$\color{#35bf28}+1.84\%$
test_tensor_to_bytestream_speed[safetensors]	42.6226μs	42.3869μs	23.5922 KOps/s	22.9501 KOps/s	$\color{#35bf28}+2.80\%$
test_simple	0.6611s	0.5725s	1.7468 Ops/s	1.7516 Ops/s	$\color{#d91a1a}-0.27\%$
test_transformed	1.2338s	1.1520s	0.8681 Ops/s	0.8783 Ops/s	$\color{#d91a1a}-1.17\%$
test_serial	1.6758s	1.6681s	0.5995 Ops/s	0.5885 Ops/s	$\color{#35bf28}+1.87\%$
test_parallel	1.1814s	1.1361s	0.8802 Ops/s	0.9032 Ops/s	$\color{#d91a1a}-2.54\%$
test_step_mdp_speed[True-True-True-True-True]	0.2555ms	44.7494μs	22.3466 KOps/s	22.1700 KOps/s	$\color{#35bf28}+0.80\%$
test_step_mdp_speed[True-True-True-True-False]	63.5310μs	25.2665μs	39.5781 KOps/s	40.0694 KOps/s	$\color{#d91a1a}-1.23\%$
test_step_mdp_speed[True-True-True-False-True]	57.3010μs	25.3462μs	39.4536 KOps/s	39.7881 KOps/s	$\color{#d91a1a}-0.84\%$
test_step_mdp_speed[True-True-True-False-False]	58.8710μs	13.7469μs	72.7438 KOps/s	71.7709 KOps/s	$\color{#35bf28}+1.36\%$
test_step_mdp_speed[True-True-False-True-True]	0.1043ms	48.2778μs	20.7135 KOps/s	20.8662 KOps/s	$\color{#d91a1a}-0.73\%$
test_step_mdp_speed[True-True-False-True-False]	59.2010μs	28.1142μs	35.5692 KOps/s	35.5804 KOps/s	$\color{#d91a1a}-0.03\%$
test_step_mdp_speed[True-True-False-False-True]	57.1810μs	28.4025μs	35.2081 KOps/s	35.8114 KOps/s	$\color{#d91a1a}-1.68\%$
test_step_mdp_speed[True-True-False-False-False]	61.6110μs	16.5499μs	60.4233 KOps/s	59.5935 KOps/s	$\color{#35bf28}+1.39\%$
test_step_mdp_speed[True-False-True-True-True]	0.1047ms	50.6998μs	19.7240 KOps/s	19.6148 KOps/s	$\color{#35bf28}+0.56\%$
test_step_mdp_speed[True-False-True-True-False]	61.4910μs	30.8879μs	32.3752 KOps/s	32.4031 KOps/s	$\color{#d91a1a}-0.09\%$
test_step_mdp_speed[True-False-True-False-True]	62.7910μs	28.1501μs	35.5238 KOps/s	35.7236 KOps/s	$\color{#d91a1a}-0.56\%$
test_step_mdp_speed[True-False-True-False-False]	44.9910μs	16.4730μs	60.7054 KOps/s	59.8449 KOps/s	$\color{#35bf28}+1.44\%$
test_step_mdp_speed[True-False-False-True-True]	87.1210μs	53.4392μs	18.7129 KOps/s	18.7895 KOps/s	$\color{#d91a1a}-0.41\%$
test_step_mdp_speed[True-False-False-True-False]	99.3220μs	33.6020μs	29.7601 KOps/s	29.5470 KOps/s	$\color{#35bf28}+0.72\%$
test_step_mdp_speed[True-False-False-False-True]	60.6310μs	30.2719μs	33.0339 KOps/s	32.5898 KOps/s	$\color{#35bf28}+1.36\%$
test_step_mdp_speed[True-False-False-False-False]	87.8910μs	19.1227μs	52.2938 KOps/s	50.8665 KOps/s	$\color{#35bf28}+2.81\%$
test_step_mdp_speed[False-True-True-True-True]	85.4910μs	50.8198μs	19.6774 KOps/s	19.6707 KOps/s	$\color{#35bf28}+0.03\%$
test_step_mdp_speed[False-True-True-True-False]	56.5400μs	30.9005μs	32.3619 KOps/s	31.6685 KOps/s	$\color{#35bf28}+2.19\%$
test_step_mdp_speed[False-True-True-False-True]	2.3482ms	32.2014μs	31.0546 KOps/s	31.2553 KOps/s	$\color{#d91a1a}-0.64\%$
test_step_mdp_speed[False-True-True-False-False]	47.9810μs	17.9960μs	55.5680 KOps/s	53.7836 KOps/s	$\color{#35bf28}+3.32\%$
test_step_mdp_speed[False-True-False-True-True]	0.1012ms	53.2786μs	18.7693 KOps/s	18.5113 KOps/s	$\color{#35bf28}+1.39\%$
test_step_mdp_speed[False-True-False-True-False]	66.6710μs	33.2256μs	30.0972 KOps/s	29.6044 KOps/s	$\color{#35bf28}+1.66\%$
test_step_mdp_speed[False-True-False-False-True]	76.3210μs	34.1337μs	29.2966 KOps/s	29.3188 KOps/s	$\color{#d91a1a}-0.08\%$
test_step_mdp_speed[False-True-False-False-False]	54.1110μs	20.8275μs	48.0134 KOps/s	46.9339 KOps/s	$\color{#35bf28}+2.30\%$
test_step_mdp_speed[False-False-True-True-True]	0.1531ms	55.9082μs	17.8864 KOps/s	17.6713 KOps/s	$\color{#35bf28}+1.22\%$
test_step_mdp_speed[False-False-True-True-False]	66.0710μs	36.5169μs	27.3846 KOps/s	27.3818 KOps/s	$\color{#35bf28}+0.01\%$
test_step_mdp_speed[False-False-True-False-True]	67.0010μs	34.3777μs	29.0886 KOps/s	29.0850 KOps/s	$\color{#35bf28}+0.01\%$
test_step_mdp_speed[False-False-True-False-False]	59.9010μs	20.8012μs	48.0743 KOps/s	47.3175 KOps/s	$\color{#35bf28}+1.60\%$
test_step_mdp_speed[False-False-False-True-True]	89.0610μs	58.4006μs	17.1231 KOps/s	17.1864 KOps/s	$\color{#d91a1a}-0.37\%$
test_step_mdp_speed[False-False-False-True-False]	83.9510μs	38.9317μs	25.6860 KOps/s	25.4877 KOps/s	$\color{#35bf28}+0.78\%$
test_step_mdp_speed[False-False-False-False-True]	91.0620μs	36.2497μs	27.5864 KOps/s	27.3909 KOps/s	$\color{#35bf28}+0.71\%$
test_step_mdp_speed[False-False-False-False-False]	49.3710μs	23.4679μs	42.6115 KOps/s	42.4917 KOps/s	$\color{#35bf28}+0.28\%$
test_values[generalized_advantage_estimate-True-True]	10.7991ms	9.9478ms	100.5243 Ops/s	98.5849 Ops/s	$\color{#35bf28}+1.97\%$
test_values[vec_generalized_advantage_estimate-True-True]	19.5348ms	17.6520ms	56.6508 Ops/s	56.9360 Ops/s	$\color{#d91a1a}-0.50\%$
test_values[td0_return_estimate-False-False]	0.2339ms	0.1329ms	7.5218 KOps/s	7.7448 KOps/s	$\color{#d91a1a}-2.88\%$
test_values[td1_return_estimate-False-False]	27.8162ms	27.5061ms	36.3555 Ops/s	35.7613 Ops/s	$\color{#35bf28}+1.66\%$
test_values[vec_td1_return_estimate-False-False]	18.0813ms	17.7251ms	56.4171 Ops/s	56.5225 Ops/s	$\color{#d91a1a}-0.19\%$
test_values[td_lambda_return_estimate-True-False]	41.5920ms	40.9645ms	24.4114 Ops/s	24.0621 Ops/s	$\color{#35bf28}+1.45\%$
test_values[vec_td_lambda_return_estimate-True-False]	18.5554ms	17.7336ms	56.3900 Ops/s	56.5220 Ops/s	$\color{#d91a1a}-0.23\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	8.6289ms	8.5131ms	117.4660 Ops/s	113.9418 Ops/s	$\color{#35bf28}+3.09\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	1.7722ms	1.5507ms	644.8622 Ops/s	662.3349 Ops/s	$\color{#d91a1a}-2.64\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.4766ms	0.4091ms	2.4441 KOps/s	2.4342 KOps/s	$\color{#35bf28}+0.41\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	34.9039ms	34.2949ms	29.1589 Ops/s	31.2388 Ops/s	$\textbf{\color{#d91a1a}-6.66\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	1.9491ms	1.7450ms	573.0655 Ops/s	582.9284 Ops/s	$\color{#d91a1a}-1.69\%$
test_dqn_speed[False-None]	6.3144ms	1.4306ms	698.9932 Ops/s	695.1669 Ops/s	$\color{#35bf28}+0.55\%$
test_dqn_speed[False-backward]	1.9953ms	1.9340ms	517.0533 Ops/s	516.8500 Ops/s	$\color{#35bf28}+0.04\%$
test_dqn_speed[True-None]	0.7239ms	0.5213ms	1.9181 KOps/s	1.8531 KOps/s	$\color{#35bf28}+3.51\%$
test_dqn_speed[True-backward]	1.0296ms	0.9834ms	1.0168 KOps/s	892.1052 Ops/s	$\textbf{\color{#35bf28}+13.98\%}$
test_dqn_speed[reduce-overhead-None]	0.5956ms	0.5222ms	1.9150 KOps/s	1.8227 KOps/s	$\textbf{\color{#35bf28}+5.06\%}$
test_dqn_speed[reduce-overhead-backward]	1.0603ms	0.9869ms	1.0133 KOps/s	1.0246 KOps/s	$\color{#d91a1a}-1.10\%$
test_ddpg_speed[False-None]	3.2777ms	2.9305ms	341.2423 Ops/s	343.8010 Ops/s	$\color{#d91a1a}-0.74\%$
test_ddpg_speed[False-backward]	4.2463ms	4.1561ms	240.6094 Ops/s	241.3895 Ops/s	$\color{#d91a1a}-0.32\%$
test_ddpg_speed[True-None]	1.5715ms	1.4221ms	703.1894 Ops/s	702.8211 Ops/s	$\color{#35bf28}+0.05\%$
test_ddpg_speed[True-backward]	2.4965ms	2.4252ms	412.3402 Ops/s	366.3583 Ops/s	$\textbf{\color{#35bf28}+12.55\%}$
test_ddpg_speed[reduce-overhead-None]	1.5390ms	1.3754ms	727.0819 Ops/s	680.0815 Ops/s	$\textbf{\color{#35bf28}+6.91\%}$
test_ddpg_speed[reduce-overhead-backward]	2.4392ms	2.3863ms	419.0555 Ops/s	407.2914 Ops/s	$\color{#35bf28}+2.89\%$
test_sac_speed[False-None]	8.3868ms	7.9671ms	125.5157 Ops/s	124.2640 Ops/s	$\color{#35bf28}+1.01\%$
test_sac_speed[False-backward]	13.0876ms	11.3280ms	88.2766 Ops/s	88.0068 Ops/s	$\color{#35bf28}+0.31\%$
test_sac_speed[True-None]	2.4087ms	2.1443ms	466.3580 Ops/s	442.7720 Ops/s	$\textbf{\color{#35bf28}+5.33\%}$
test_sac_speed[True-backward]	4.1830ms	4.0827ms	244.9352 Ops/s	241.7678 Ops/s	$\color{#35bf28}+1.31\%$
test_sac_speed[reduce-overhead-None]	2.2215ms	2.0902ms	478.4262 Ops/s	460.8513 Ops/s	$\color{#35bf28}+3.81\%$
test_sac_speed[reduce-overhead-backward]	4.1048ms	3.9787ms	251.3386 Ops/s	241.2174 Ops/s	$\color{#35bf28}+4.20\%$
test_redq_speed[False-None]	15.2207ms	10.4447ms	95.7427 Ops/s	92.7828 Ops/s	$\color{#35bf28}+3.19\%$
test_redq_speed[False-backward]	18.8139ms	17.7293ms	56.4037 Ops/s	53.5813 Ops/s	$\textbf{\color{#35bf28}+5.27\%}$
test_redq_speed[True-None]	4.7481ms	4.3501ms	229.8820 Ops/s	227.9720 Ops/s	$\color{#35bf28}+0.84\%$
test_redq_speed[True-backward]	10.2641ms	9.7527ms	102.5362 Ops/s	100.1109 Ops/s	$\color{#35bf28}+2.42\%$
test_redq_speed[reduce-overhead-None]	4.5315ms	4.3319ms	230.8478 Ops/s	215.7851 Ops/s	$\textbf{\color{#35bf28}+6.98\%}$
test_redq_speed[reduce-overhead-backward]	10.4804ms	9.9641ms	100.3606 Ops/s	96.8509 Ops/s	$\color{#35bf28}+3.62\%$
test_redq_deprec_speed[False-None]	11.4323ms	11.0493ms	90.5038 Ops/s	93.1736 Ops/s	$\color{#d91a1a}-2.87\%$
test_redq_deprec_speed[False-backward]	16.6240ms	16.0083ms	62.4675 Ops/s	65.2579 Ops/s	$\color{#d91a1a}-4.28\%$
test_redq_deprec_speed[True-None]	3.9400ms	3.7532ms	266.4394 Ops/s	284.9387 Ops/s	$\textbf{\color{#d91a1a}-6.49\%}$
test_redq_deprec_speed[True-backward]	9.1643ms	7.7878ms	128.4056 Ops/s	131.7276 Ops/s	$\color{#d91a1a}-2.52\%$
test_redq_deprec_speed[reduce-overhead-None]	4.4366ms	3.7129ms	269.3330 Ops/s	284.7037 Ops/s	$\textbf{\color{#d91a1a}-5.40\%}$
test_redq_deprec_speed[reduce-overhead-backward]	8.3709ms	7.8738ms	127.0036 Ops/s	125.1639 Ops/s	$\color{#35bf28}+1.47\%$
test_td3_speed[False-None]	9.6614ms	8.0525ms	124.1855 Ops/s	125.3761 Ops/s	$\color{#d91a1a}-0.95\%$
test_td3_speed[False-backward]	11.3578ms	10.9333ms	91.4637 Ops/s	92.3183 Ops/s	$\color{#d91a1a}-0.93\%$
test_td3_speed[True-None]	1.8609ms	1.8269ms	547.3708 Ops/s	541.9265 Ops/s	$\color{#35bf28}+1.00\%$
test_td3_speed[True-backward]	3.7010ms	3.5426ms	282.2770 Ops/s	256.5985 Ops/s	$\textbf{\color{#35bf28}+10.01\%}$
test_td3_speed[reduce-overhead-None]	1.7708ms	1.7320ms	577.3686 Ops/s	574.6963 Ops/s	$\color{#35bf28}+0.46\%$
test_td3_speed[reduce-overhead-backward]	3.6877ms	3.5796ms	279.3646 Ops/s	278.8715 Ops/s	$\color{#35bf28}+0.18\%$
test_cql_speed[False-None]	30.0368ms	26.1865ms	38.1876 Ops/s	39.3192 Ops/s	$\color{#d91a1a}-2.88\%$
test_cql_speed[False-backward]	39.6877ms	35.4168ms	28.2352 Ops/s	28.7007 Ops/s	$\color{#d91a1a}-1.62\%$
test_cql_speed[True-None]	12.6527ms	12.2373ms	81.7177 Ops/s	81.9150 Ops/s	$\color{#d91a1a}-0.24\%$
test_cql_speed[True-backward]	18.7401ms	18.1703ms	55.0348 Ops/s	56.5176 Ops/s	$\color{#d91a1a}-2.62\%$
test_cql_speed[reduce-overhead-None]	12.6354ms	12.2005ms	81.9640 Ops/s	81.9283 Ops/s	$\color{#35bf28}+0.04\%$
test_cql_speed[reduce-overhead-backward]	18.4858ms	18.2043ms	54.9321 Ops/s	55.2016 Ops/s	$\color{#d91a1a}-0.49\%$
test_a2c_speed[False-None]	5.5699ms	5.2866ms	189.1568 Ops/s	182.6143 Ops/s	$\color{#35bf28}+3.58\%$
test_a2c_speed[False-backward]	11.8927ms	11.6679ms	85.7055 Ops/s	84.7191 Ops/s	$\color{#35bf28}+1.16\%$
test_a2c_speed[True-None]	3.9062ms	3.6887ms	271.0969 Ops/s	274.1507 Ops/s	$\color{#d91a1a}-1.11\%$
test_a2c_speed[True-backward]	8.7808ms	8.5760ms	116.6044 Ops/s	116.5227 Ops/s	$\color{#35bf28}+0.07\%$
test_a2c_speed[reduce-overhead-None]	3.9107ms	3.6641ms	272.9151 Ops/s	274.6509 Ops/s	$\color{#d91a1a}-0.63\%$
test_a2c_speed[reduce-overhead-backward]	8.9033ms	8.7255ms	114.6066 Ops/s	106.0071 Ops/s	$\textbf{\color{#35bf28}+8.11\%}$
test_ppo_speed[False-None]	6.1005ms	5.8669ms	170.4492 Ops/s	170.9924 Ops/s	$\color{#d91a1a}-0.32\%$
test_ppo_speed[False-backward]	12.8092ms	12.4550ms	80.2890 Ops/s	81.0998 Ops/s	$\color{#d91a1a}-1.00\%$
test_ppo_speed[True-None]	3.7866ms	3.5971ms	278.0005 Ops/s	276.7933 Ops/s	$\color{#35bf28}+0.44\%$
test_ppo_speed[True-backward]	8.5490ms	8.3702ms	119.4714 Ops/s	120.0271 Ops/s	$\color{#d91a1a}-0.46\%$
test_ppo_speed[reduce-overhead-None]	3.7848ms	3.5939ms	278.2474 Ops/s	276.6683 Ops/s	$\color{#35bf28}+0.57\%$
test_ppo_speed[reduce-overhead-backward]	8.9625ms	8.6336ms	115.8266 Ops/s	108.0039 Ops/s	$\textbf{\color{#35bf28}+7.24\%}$
test_reinforce_speed[False-None]	4.7271ms	4.5238ms	221.0537 Ops/s	222.1888 Ops/s	$\color{#d91a1a}-0.51\%$
test_reinforce_speed[False-backward]	7.4997ms	7.2916ms	137.1439 Ops/s	136.0892 Ops/s	$\color{#35bf28}+0.78\%$
test_reinforce_speed[True-None]	2.9824ms	2.8280ms	353.6026 Ops/s	351.9827 Ops/s	$\color{#35bf28}+0.46\%$
test_reinforce_speed[True-backward]	7.9828ms	7.7253ms	129.4455 Ops/s	132.1259 Ops/s	$\color{#d91a1a}-2.03\%$
test_reinforce_speed[reduce-overhead-None]	2.9565ms	2.8059ms	356.3873 Ops/s	346.5807 Ops/s	$\color{#35bf28}+2.83\%$
test_reinforce_speed[reduce-overhead-backward]	7.9231ms	7.7080ms	129.7345 Ops/s	126.8395 Ops/s	$\color{#35bf28}+2.28\%$
test_iql_speed[False-None]	20.6200ms	19.7663ms	50.5913 Ops/s	49.8622 Ops/s	$\color{#35bf28}+1.46\%$
test_iql_speed[False-backward]	30.8268ms	30.2309ms	33.0787 Ops/s	33.1221 Ops/s	$\color{#d91a1a}-0.13\%$
test_iql_speed[True-None]	8.5977ms	8.3570ms	119.6595 Ops/s	117.9330 Ops/s	$\color{#35bf28}+1.46\%$
test_iql_speed[True-backward]	16.8341ms	16.4963ms	60.6196 Ops/s	58.1262 Ops/s	$\color{#35bf28}+4.29\%$
test_iql_speed[reduce-overhead-None]	8.6616ms	8.4521ms	118.3139 Ops/s	112.7681 Ops/s	$\color{#35bf28}+4.92\%$
test_iql_speed[reduce-overhead-backward]	17.6832ms	17.2386ms	58.0093 Ops/s	57.0293 Ops/s	$\color{#35bf28}+1.72\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	7.6173ms	6.0430ms	165.4818 Ops/s	166.0311 Ops/s	$\color{#d91a1a}-0.33\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.5652ms	0.3252ms	3.0751 KOps/s	3.5174 KOps/s	$\textbf{\color{#d91a1a}-12.58\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.5874ms	0.3109ms	3.2168 KOps/s	3.3680 KOps/s	$\color{#d91a1a}-4.49\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	6.0330ms	5.7635ms	173.5052 Ops/s	173.7143 Ops/s	$\color{#d91a1a}-0.12\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	2.1503ms	0.2971ms	3.3660 KOps/s	3.4018 KOps/s	$\color{#d91a1a}-1.05\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.6307ms	0.2966ms	3.3718 KOps/s	3.5570 KOps/s	$\textbf{\color{#d91a1a}-5.21\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	1.6271ms	1.3889ms	719.9690 Ops/s	796.6278 Ops/s	$\textbf{\color{#d91a1a}-9.62\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	1.5206ms	1.2940ms	772.8011 Ops/s	850.8926 Ops/s	$\textbf{\color{#d91a1a}-9.18\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	6.1259ms	5.9150ms	169.0614 Ops/s	168.3080 Ops/s	$\color{#35bf28}+0.45\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.1961ms	0.4645ms	2.1529 KOps/s	2.2795 KOps/s	$\textbf{\color{#d91a1a}-5.55\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.7402ms	0.4331ms	2.3088 KOps/s	2.1607 KOps/s	$\textbf{\color{#35bf28}+6.86\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	5.8762ms	5.7382ms	174.2709 Ops/s	171.7339 Ops/s	$\color{#35bf28}+1.48\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.7885ms	0.3762ms	2.6582 KOps/s	2.7480 KOps/s	$\color{#d91a1a}-3.27\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6558ms	0.3601ms	2.7767 KOps/s	2.9299 KOps/s	$\textbf{\color{#d91a1a}-5.23\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.9709ms	5.7018ms	175.3822 Ops/s	173.9692 Ops/s	$\color{#35bf28}+0.81\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.8675ms	0.3424ms	2.9205 KOps/s	3.5240 KOps/s	$\textbf{\color{#d91a1a}-17.12\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.4705ms	0.2612ms	3.8286 KOps/s	3.0221 KOps/s	$\textbf{\color{#35bf28}+26.69\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	6.0522ms	5.8794ms	170.0865 Ops/s	168.1598 Ops/s	$\color{#35bf28}+1.15\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.1443ms	0.4543ms	2.2012 KOps/s	2.0441 KOps/s	$\textbf{\color{#35bf28}+7.69\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.6903ms	0.4611ms	2.1686 KOps/s	2.1360 KOps/s	$\color{#35bf28}+1.53\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.4485s	13.9025ms	71.9294 Ops/s	56.6696 Ops/s	$\textbf{\color{#35bf28}+26.93\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	3.9369ms	1.7400ms	574.7179 Ops/s	486.5785 Ops/s	$\textbf{\color{#35bf28}+18.11\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	27.1031ms	1.7561ms	569.4585 Ops/s	852.2747 Ops/s	$\textbf{\color{#d91a1a}-33.18\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	7.0926ms	5.0602ms	197.6203 Ops/s	194.5218 Ops/s	$\color{#35bf28}+1.59\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	8.7073ms	2.0414ms	489.8716 Ops/s	482.5451 Ops/s	$\color{#35bf28}+1.52\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	7.2864ms	1.1949ms	836.8654 Ops/s	834.7412 Ops/s	$\color{#35bf28}+0.25\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.4094s	13.3605ms	74.8475 Ops/s	60.7492 Ops/s	$\textbf{\color{#35bf28}+23.21\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	12.6618ms	2.0734ms	482.2950 Ops/s	513.0290 Ops/s	$\textbf{\color{#d91a1a}-5.99\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	1.2170ms	1.0176ms	982.7429 Ops/s	979.4538 Ops/s	$\color{#35bf28}+0.34\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True]	34.9183ms	32.8927ms	30.4019 Ops/s	30.2836 Ops/s	$\color{#35bf28}+0.39\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False]	19.0106ms	17.5237ms	57.0657 Ops/s	57.9918 Ops/s	$\color{#d91a1a}-1.60\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True]	35.7838ms	33.6643ms	29.7051 Ops/s	29.2650 Ops/s	$\color{#35bf28}+1.50\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False]	19.4210ms	17.6280ms	56.7279 Ops/s	57.0714 Ops/s	$\color{#d91a1a}-0.60\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True]	40.6929ms	35.5230ms	28.1508 Ops/s	27.6055 Ops/s	$\color{#35bf28}+1.98\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False]	20.2934ms	19.0201ms	52.5761 Ops/s	53.3625 Ops/s	$\color{#d91a1a}-1.47\%$