[CI] Fix benchmarks for LLMs

[vmoens]

Stack from ghstack (oldest at bottom):

• [Quality] Fix flaky test #3211
• [Feature] Documentation #3192
• [Feature] SAC Trainer #3191
• [Feature] PPO Trainer Updates #3190
• [Feature] Trainer Algorithms - Configuration System #3189
• [Feature] Trainer Infrastructure - Timing and Utilities #3188
• [BugFix] Fix GRPO tests and runs #3213
• -> [CI] Fix benchmarks for LLMs #3212

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3212

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• [ROCm][CI] Machines under the label linux.rocm.gpu.2 are undergoing maintenance.

❌ 6 New Failures, 10 Unrelated Failures

As of commit 070d407 with merge base a7707ca ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Process completed with exit code 139.
• Generate documentation / build-docs (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t d4a223699567dc4f20e1af1e656947360946343730e3bd76047539d9189919f0 /exec failed with exit code 2
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 572f5bd3bb814c8f0c9a9060b81bd93ac0da4528a5ae922dd65fd0af0af52acf /exec failed with exit code 1
• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 0a7988314f1bba411457f0204652cf8df73d2b0c2d0c0b3570b9b66f51bb802a /exec failed with exit code 1
• Libs Tests on Linux / unittests-sklearn (3.10, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 483adf0e1e796b3c3f7694ea8f16e8b8ed0a90c7497d8e5a9bc639f984285387 /exec failed with exit code 2
• SOTA Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 85aae376e4448aef571329c56f6c5fe81a3e0c168aaa033e77eecfadd9e34e39 /exec failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh) (similar failure)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-cpu (3.9) / linux-job (gh) (similar failure)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• LLM Tests on Linux / unittests (3.9, 12.8) / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh) (trunk failure)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh) (trunk failure)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-gpu (3.11, 12.8) / linux-job (gh) (trunk failure)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-olddeps (3.9, 11.6) / linux-job (gh) (trunk failure)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-optdeps (3.11, 12.8) / linux-job (gh) (trunk failure)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh) (trunk failure)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Windows / unittests-cpu (3.10, windows.4xlarge, cpu) / windows-job (gh) (trunk failure)
  Process completed with exit code 2.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 154. Improved: $\large\color{#35bf28}8$. Worsened: $\large\color{#d91a1a}12$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_tensor_to_bytestream_speed[pickle]	84.0285μs	82.1667μs	12.1704 KOps/s	12.2513 KOps/s	$\color{#d91a1a}-0.66\%$
test_tensor_to_bytestream_speed[torch.save]	0.1403ms	0.1397ms	7.1561 KOps/s	7.1781 KOps/s	$\color{#d91a1a}-0.31\%$
test_tensor_to_bytestream_speed[untyped_storage]	0.1253s	0.1251s	7.9953 Ops/s	8.0861 Ops/s	$\color{#d91a1a}-1.12\%$
test_tensor_to_bytestream_speed[numpy]	2.9152μs	2.9123μs	343.3665 KOps/s	346.0077 KOps/s	$\color{#d91a1a}-0.76\%$
test_tensor_to_bytestream_speed[safetensors]	43.0905μs	42.9402μs	23.2882 KOps/s	23.6859 KOps/s	$\color{#d91a1a}-1.68\%$
test_simple	0.5631s	0.5523s	1.8105 Ops/s	1.7460 Ops/s	$\color{#35bf28}+3.69\%$
test_transformed	1.1107s	1.1092s	0.9016 Ops/s	0.8781 Ops/s	$\color{#35bf28}+2.67\%$
test_serial	1.6809s	1.6792s	0.5955 Ops/s	0.5917 Ops/s	$\color{#35bf28}+0.64\%$
test_parallel	1.1678s	1.1078s	0.9027 Ops/s	0.9024 Ops/s	$\color{#35bf28}+0.02\%$
test_step_mdp_speed[True-True-True-True-True]	0.2502ms	46.2396μs	21.6265 KOps/s	21.9158 KOps/s	$\color{#d91a1a}-1.32\%$
test_step_mdp_speed[True-True-True-True-False]	56.8720μs	25.2990μs	39.5273 KOps/s	39.2657 KOps/s	$\color{#35bf28}+0.67\%$
test_step_mdp_speed[True-True-True-False-True]	69.5610μs	24.9653μs	40.0555 KOps/s	39.3522 KOps/s	$\color{#35bf28}+1.79\%$
test_step_mdp_speed[True-True-True-False-False]	40.3810μs	13.9464μs	71.7032 KOps/s	70.3370 KOps/s	$\color{#35bf28}+1.94\%$
test_step_mdp_speed[True-True-False-True-True]	0.1285ms	47.9102μs	20.8724 KOps/s	20.6207 KOps/s	$\color{#35bf28}+1.22\%$
test_step_mdp_speed[True-True-False-True-False]	70.1420μs	27.9308μs	35.8028 KOps/s	35.0522 KOps/s	$\color{#35bf28}+2.14\%$
test_step_mdp_speed[True-True-False-False-True]	90.0020μs	27.8452μs	35.9129 KOps/s	35.2372 KOps/s	$\color{#35bf28}+1.92\%$
test_step_mdp_speed[True-True-False-False-False]	40.0810μs	16.8009μs	59.5207 KOps/s	59.0890 KOps/s	$\color{#35bf28}+0.73\%$
test_step_mdp_speed[True-False-True-True-True]	87.0010μs	50.8787μs	19.6546 KOps/s	19.3837 KOps/s	$\color{#35bf28}+1.40\%$
test_step_mdp_speed[True-False-True-True-False]	57.2410μs	30.6673μs	32.6080 KOps/s	32.0691 KOps/s	$\color{#35bf28}+1.68\%$
test_step_mdp_speed[True-False-True-False-True]	57.9420μs	27.9882μs	35.7294 KOps/s	34.9351 KOps/s	$\color{#35bf28}+2.27\%$
test_step_mdp_speed[True-False-True-False-False]	41.9710μs	16.8914μs	59.2018 KOps/s	58.8298 KOps/s	$\color{#35bf28}+0.63\%$
test_step_mdp_speed[True-False-False-True-True]	80.9920μs	54.4382μs	18.3694 KOps/s	18.5426 KOps/s	$\color{#d91a1a}-0.93\%$
test_step_mdp_speed[True-False-False-True-False]	67.7120μs	33.5983μs	29.7634 KOps/s	30.1868 KOps/s	$\color{#d91a1a}-1.40\%$
test_step_mdp_speed[True-False-False-False-True]	65.0010μs	30.8398μs	32.4256 KOps/s	32.3188 KOps/s	$\color{#35bf28}+0.33\%$
test_step_mdp_speed[True-False-False-False-False]	50.7610μs	19.5578μs	51.1305 KOps/s	51.2627 KOps/s	$\color{#d91a1a}-0.26\%$
test_step_mdp_speed[False-True-True-True-True]	81.4920μs	50.8359μs	19.6711 KOps/s	19.3541 KOps/s	$\color{#35bf28}+1.64\%$
test_step_mdp_speed[False-True-True-True-False]	55.4710μs	30.6686μs	32.6067 KOps/s	32.2040 KOps/s	$\color{#35bf28}+1.25\%$
test_step_mdp_speed[False-True-True-False-True]	2.4546ms	32.6229μs	30.6533 KOps/s	30.6319 KOps/s	$\color{#35bf28}+0.07\%$
test_step_mdp_speed[False-True-True-False-False]	95.1310μs	18.4365μs	54.2404 KOps/s	52.8930 KOps/s	$\color{#35bf28}+2.55\%$
test_step_mdp_speed[False-True-False-True-True]	89.1620μs	54.2907μs	18.4194 KOps/s	18.4792 KOps/s	$\color{#d91a1a}-0.32\%$
test_step_mdp_speed[False-True-False-True-False]	63.6120μs	33.3006μs	30.0295 KOps/s	29.4729 KOps/s	$\color{#35bf28}+1.89\%$
test_step_mdp_speed[False-True-False-False-True]	74.5310μs	34.8062μs	28.7305 KOps/s	28.5055 KOps/s	$\color{#35bf28}+0.79\%$
test_step_mdp_speed[False-True-False-False-False]	50.6610μs	21.3260μs	46.8911 KOps/s	46.2099 KOps/s	$\color{#35bf28}+1.47\%$
test_step_mdp_speed[False-False-True-True-True]	0.1013ms	56.7675μs	17.6157 KOps/s	17.6221 KOps/s	$\color{#d91a1a}-0.04\%$
test_step_mdp_speed[False-False-True-True-False]	66.2220μs	36.8965μs	27.1028 KOps/s	27.5483 KOps/s	$\color{#d91a1a}-1.62\%$
test_step_mdp_speed[False-False-True-False-True]	61.9210μs	35.2234μs	28.3902 KOps/s	29.0258 KOps/s	$\color{#d91a1a}-2.19\%$
test_step_mdp_speed[False-False-True-False-False]	45.7810μs	21.7403μs	45.9975 KOps/s	46.6036 KOps/s	$\color{#d91a1a}-1.30\%$
test_step_mdp_speed[False-False-False-True-True]	89.7410μs	58.1690μs	17.1913 KOps/s	17.2933 KOps/s	$\color{#d91a1a}-0.59\%$
test_step_mdp_speed[False-False-False-True-False]	0.1271ms	39.2634μs	25.4690 KOps/s	25.8474 KOps/s	$\color{#d91a1a}-1.46\%$
test_step_mdp_speed[False-False-False-False-True]	66.9120μs	37.0560μs	26.9862 KOps/s	26.4476 KOps/s	$\color{#35bf28}+2.04\%$
test_step_mdp_speed[False-False-False-False-False]	57.8510μs	23.9451μs	41.7622 KOps/s	41.7939 KOps/s	$\color{#d91a1a}-0.08\%$
test_values[generalized_advantage_estimate-True-True]	10.6780ms	10.1487ms	98.5353 Ops/s	100.8887 Ops/s	$\color{#d91a1a}-2.33\%$
test_values[vec_generalized_advantage_estimate-True-True]	19.9287ms	17.5955ms	56.8327 Ops/s	57.3450 Ops/s	$\color{#d91a1a}-0.89\%$
test_values[td0_return_estimate-False-False]	0.2251ms	0.1313ms	7.6145 KOps/s	4.9528 KOps/s	$\textbf{\color{#35bf28}+53.74\%}$
test_values[td1_return_estimate-False-False]	28.6908ms	27.3641ms	36.5442 Ops/s	36.7066 Ops/s	$\color{#d91a1a}-0.44\%$
test_values[vec_td1_return_estimate-False-False]	18.0087ms	17.6849ms	56.5453 Ops/s	57.0057 Ops/s	$\color{#d91a1a}-0.81\%$
test_values[td_lambda_return_estimate-True-False]	42.3360ms	40.6983ms	24.5710 Ops/s	24.3878 Ops/s	$\color{#35bf28}+0.75\%$
test_values[vec_td_lambda_return_estimate-True-False]	17.9221ms	17.6326ms	56.7132 Ops/s	56.3101 Ops/s	$\color{#35bf28}+0.72\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	8.6835ms	8.5466ms	117.0050 Ops/s	117.7568 Ops/s	$\color{#d91a1a}-0.64\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	1.7256ms	1.5305ms	653.4018 Ops/s	683.5987 Ops/s	$\color{#d91a1a}-4.42\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.6968ms	0.4325ms	2.3123 KOps/s	2.4406 KOps/s	$\textbf{\color{#d91a1a}-5.26\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	34.8020ms	34.2347ms	29.2101 Ops/s	28.8693 Ops/s	$\color{#35bf28}+1.18\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	1.8781ms	1.7123ms	583.9991 Ops/s	588.4511 Ops/s	$\color{#d91a1a}-0.76\%$
test_dqn_speed[False-None]	6.2920ms	1.4360ms	696.3603 Ops/s	701.5400 Ops/s	$\color{#d91a1a}-0.74\%$
test_dqn_speed[False-backward]	2.2530ms	1.9676ms	508.2343 Ops/s	517.2491 Ops/s	$\color{#d91a1a}-1.74\%$
test_dqn_speed[True-None]	0.8338ms	0.5194ms	1.9252 KOps/s	1.9215 KOps/s	$\color{#35bf28}+0.19\%$
test_dqn_speed[True-backward]	1.0488ms	0.9796ms	1.0208 KOps/s	918.9788 Ops/s	$\textbf{\color{#35bf28}+11.08\%}$
test_dqn_speed[reduce-overhead-None]	0.8517ms	0.5143ms	1.9446 KOps/s	1.8880 KOps/s	$\color{#35bf28}+3.00\%$
test_dqn_speed[reduce-overhead-backward]	0.9920ms	0.9639ms	1.0374 KOps/s	1.0189 KOps/s	$\color{#35bf28}+1.81\%$
test_ddpg_speed[False-None]	3.2310ms	2.9103ms	343.6058 Ops/s	345.9145 Ops/s	$\color{#d91a1a}-0.67\%$
test_ddpg_speed[False-backward]	4.2590ms	4.1321ms	242.0062 Ops/s	242.3604 Ops/s	$\color{#d91a1a}-0.15\%$
test_ddpg_speed[True-None]	1.5912ms	1.4142ms	707.0934 Ops/s	694.2258 Ops/s	$\color{#35bf28}+1.85\%$
test_ddpg_speed[True-backward]	2.4882ms	2.4111ms	414.7470 Ops/s	375.5618 Ops/s	$\textbf{\color{#35bf28}+10.43\%}$
test_ddpg_speed[reduce-overhead-None]	1.5375ms	1.4009ms	713.8295 Ops/s	688.5043 Ops/s	$\color{#35bf28}+3.68\%$
test_ddpg_speed[reduce-overhead-backward]	2.8124ms	2.4349ms	410.6942 Ops/s	414.0170 Ops/s	$\color{#d91a1a}-0.80\%$
test_sac_speed[False-None]	8.4348ms	7.9102ms	126.4192 Ops/s	124.6662 Ops/s	$\color{#35bf28}+1.41\%$
test_sac_speed[False-backward]	11.5690ms	11.1933ms	89.3390 Ops/s	87.2894 Ops/s	$\color{#35bf28}+2.35\%$
test_sac_speed[True-None]	2.4868ms	2.1340ms	468.6066 Ops/s	460.4470 Ops/s	$\color{#35bf28}+1.77\%$
test_sac_speed[True-backward]	4.2264ms	4.0804ms	245.0725 Ops/s	236.7599 Ops/s	$\color{#35bf28}+3.51\%$
test_sac_speed[reduce-overhead-None]	2.4940ms	2.1200ms	471.7080 Ops/s	462.1082 Ops/s	$\color{#35bf28}+2.08\%$
test_sac_speed[reduce-overhead-backward]	4.2225ms	4.0554ms	246.5825 Ops/s	208.0432 Ops/s	$\textbf{\color{#35bf28}+18.52\%}$
test_redq_speed[False-None]	10.9248ms	10.3675ms	96.4555 Ops/s	95.2284 Ops/s	$\color{#35bf28}+1.29\%$
test_redq_speed[False-backward]	18.4485ms	18.0095ms	55.5263 Ops/s	54.4134 Ops/s	$\color{#35bf28}+2.05\%$
test_redq_speed[True-None]	4.6801ms	4.3873ms	227.9306 Ops/s	221.4394 Ops/s	$\color{#35bf28}+2.93\%$
test_redq_speed[True-backward]	10.3263ms	10.0554ms	99.4486 Ops/s	101.4134 Ops/s	$\color{#d91a1a}-1.94\%$
test_redq_speed[reduce-overhead-None]	4.9103ms	4.3842ms	228.0938 Ops/s	220.5950 Ops/s	$\color{#35bf28}+3.40\%$
test_redq_speed[reduce-overhead-backward]	10.3510ms	10.0845ms	99.1624 Ops/s	97.9968 Ops/s	$\color{#35bf28}+1.19\%$
test_redq_deprec_speed[False-None]	11.4524ms	11.0683ms	90.3480 Ops/s	90.3585 Ops/s	$\color{#d91a1a}-0.01\%$
test_redq_deprec_speed[False-backward]	16.5587ms	16.0502ms	62.3044 Ops/s	62.5971 Ops/s	$\color{#d91a1a}-0.47\%$
test_redq_deprec_speed[True-None]	3.8511ms	3.6566ms	273.4797 Ops/s	269.1162 Ops/s	$\color{#35bf28}+1.62\%$
test_redq_deprec_speed[True-backward]	7.8918ms	7.6967ms	129.9264 Ops/s	133.2908 Ops/s	$\color{#d91a1a}-2.52\%$
test_redq_deprec_speed[reduce-overhead-None]	3.8340ms	3.6530ms	273.7447 Ops/s	278.3518 Ops/s	$\color{#d91a1a}-1.66\%$
test_redq_deprec_speed[reduce-overhead-backward]	7.7626ms	7.5981ms	131.6126 Ops/s	116.2291 Ops/s	$\textbf{\color{#35bf28}+13.24\%}$
test_td3_speed[False-None]	8.1795ms	7.9962ms	125.0591 Ops/s	127.0243 Ops/s	$\color{#d91a1a}-1.55\%$
test_td3_speed[False-backward]	11.4677ms	10.9246ms	91.5363 Ops/s	93.4837 Ops/s	$\color{#d91a1a}-2.08\%$
test_td3_speed[True-None]	1.8890ms	1.8381ms	544.0290 Ops/s	531.2051 Ops/s	$\color{#35bf28}+2.41\%$
test_td3_speed[True-backward]	4.0012ms	3.6714ms	272.3756 Ops/s	269.7132 Ops/s	$\color{#35bf28}+0.99\%$
test_td3_speed[reduce-overhead-None]	1.8464ms	1.7970ms	556.4906 Ops/s	544.2724 Ops/s	$\color{#35bf28}+2.24\%$
test_td3_speed[reduce-overhead-backward]	3.8554ms	3.7078ms	269.7041 Ops/s	272.0146 Ops/s	$\color{#d91a1a}-0.85\%$
test_cql_speed[False-None]	29.4923ms	26.2733ms	38.0614 Ops/s	38.6873 Ops/s	$\color{#d91a1a}-1.62\%$
test_cql_speed[False-backward]	39.6370ms	35.8528ms	27.8918 Ops/s	28.5397 Ops/s	$\color{#d91a1a}-2.27\%$
test_cql_speed[True-None]	12.8177ms	12.4976ms	80.0156 Ops/s	80.2315 Ops/s	$\color{#d91a1a}-0.27\%$
test_cql_speed[True-backward]	19.0089ms	18.6868ms	53.5136 Ops/s	55.9452 Ops/s	$\color{#d91a1a}-4.35\%$
test_cql_speed[reduce-overhead-None]	13.0260ms	12.6391ms	79.1198 Ops/s	77.4783 Ops/s	$\color{#35bf28}+2.12\%$
test_cql_speed[reduce-overhead-backward]	19.0080ms	18.5870ms	53.8010 Ops/s	54.6193 Ops/s	$\color{#d91a1a}-1.50\%$
test_a2c_speed[False-None]	5.8399ms	5.4686ms	182.8611 Ops/s	179.0223 Ops/s	$\color{#35bf28}+2.14\%$
test_a2c_speed[False-backward]	12.4148ms	11.8319ms	84.5173 Ops/s	81.8321 Ops/s	$\color{#35bf28}+3.28\%$
test_a2c_speed[True-None]	3.8847ms	3.7150ms	269.1777 Ops/s	261.3985 Ops/s	$\color{#35bf28}+2.98\%$
test_a2c_speed[True-backward]	8.9263ms	8.7460ms	114.3376 Ops/s	114.9828 Ops/s	$\color{#d91a1a}-0.56\%$
test_a2c_speed[reduce-overhead-None]	4.0143ms	3.7460ms	266.9529 Ops/s	262.3471 Ops/s	$\color{#35bf28}+1.76\%$
test_a2c_speed[reduce-overhead-backward]	9.0555ms	8.8516ms	112.9736 Ops/s	104.2359 Ops/s	$\textbf{\color{#35bf28}+8.38\%}$
test_ppo_speed[False-None]	6.3445ms	5.9845ms	167.0995 Ops/s	167.4487 Ops/s	$\color{#d91a1a}-0.21\%$
test_ppo_speed[False-backward]	13.0228ms	12.6434ms	79.0927 Ops/s	78.5192 Ops/s	$\color{#35bf28}+0.73\%$
test_ppo_speed[True-None]	4.0278ms	3.6797ms	271.7611 Ops/s	259.7418 Ops/s	$\color{#35bf28}+4.63\%$
test_ppo_speed[True-backward]	8.7878ms	8.5019ms	117.6212 Ops/s	115.9810 Ops/s	$\color{#35bf28}+1.41\%$
test_ppo_speed[reduce-overhead-None]	4.3821ms	3.6686ms	272.5823 Ops/s	266.1811 Ops/s	$\color{#35bf28}+2.40\%$
test_ppo_speed[reduce-overhead-backward]	9.1153ms	8.7435ms	114.3706 Ops/s	111.3779 Ops/s	$\color{#35bf28}+2.69\%$
test_reinforce_speed[False-None]	5.3574ms	4.6651ms	214.3578 Ops/s	213.4407 Ops/s	$\color{#35bf28}+0.43\%$
test_reinforce_speed[False-backward]	7.6292ms	7.4747ms	133.7854 Ops/s	133.0957 Ops/s	$\color{#35bf28}+0.52\%$
test_reinforce_speed[True-None]	3.3134ms	2.9191ms	342.5719 Ops/s	335.1324 Ops/s	$\color{#35bf28}+2.22\%$
test_reinforce_speed[True-backward]	8.0185ms	7.8134ms	127.9851 Ops/s	128.3031 Ops/s	$\color{#d91a1a}-0.25\%$
test_reinforce_speed[reduce-overhead-None]	3.2502ms	2.8962ms	345.2835 Ops/s	338.5451 Ops/s	$\color{#35bf28}+1.99\%$
test_reinforce_speed[reduce-overhead-backward]	8.3133ms	8.0698ms	123.9187 Ops/s	119.7360 Ops/s	$\color{#35bf28}+3.49\%$
test_iql_speed[False-None]	24.3199ms	20.3607ms	49.1142 Ops/s	50.7308 Ops/s	$\color{#d91a1a}-3.19\%$
test_iql_speed[False-backward]	37.4637ms	31.3764ms	31.8710 Ops/s	33.2783 Ops/s	$\color{#d91a1a}-4.23\%$
test_iql_speed[True-None]	8.9389ms	8.5981ms	116.3049 Ops/s	122.6011 Ops/s	$\textbf{\color{#d91a1a}-5.14\%}$
test_iql_speed[True-backward]	17.4690ms	17.0414ms	58.6807 Ops/s	63.9240 Ops/s	$\textbf{\color{#d91a1a}-8.20\%}$
test_iql_speed[reduce-overhead-None]	9.0715ms	8.6478ms	115.6364 Ops/s	124.1699 Ops/s	$\textbf{\color{#d91a1a}-6.87\%}$
test_iql_speed[reduce-overhead-backward]	17.7513ms	17.4465ms	57.3180 Ops/s	56.9887 Ops/s	$\color{#35bf28}+0.58\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	7.4018ms	6.0383ms	165.6104 Ops/s	165.6173 Ops/s	$-0.00\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.5726ms	0.3457ms	2.8931 KOps/s	2.9022 KOps/s	$\color{#d91a1a}-0.32\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.5438ms	0.3303ms	3.0280 KOps/s	3.0447 KOps/s	$\color{#d91a1a}-0.55\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	6.0356ms	5.7621ms	173.5486 Ops/s	171.9761 Ops/s	$\color{#35bf28}+0.91\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.7138ms	0.3215ms	3.1105 KOps/s	3.2176 KOps/s	$\color{#d91a1a}-3.33\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.5326ms	0.3304ms	3.0267 KOps/s	3.0592 KOps/s	$\color{#d91a1a}-1.06\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	1.6360ms	1.4607ms	684.5912 Ops/s	710.7004 Ops/s	$\color{#d91a1a}-3.67\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	1.5597ms	1.3819ms	723.6311 Ops/s	754.8851 Ops/s	$\color{#d91a1a}-4.14\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	9.8274ms	6.0728ms	164.6696 Ops/s	168.5114 Ops/s	$\color{#d91a1a}-2.28\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.8083ms	0.5100ms	1.9608 KOps/s	2.0185 KOps/s	$\color{#d91a1a}-2.86\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.6935ms	0.4667ms	2.1429 KOps/s	2.4090 KOps/s	$\textbf{\color{#d91a1a}-11.04\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	6.0319ms	5.8196ms	171.8327 Ops/s	171.0055 Ops/s	$\color{#35bf28}+0.48\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	1.7685ms	0.3501ms	2.8567 KOps/s	3.0594 KOps/s	$\textbf{\color{#d91a1a}-6.63\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6551ms	0.3168ms	3.1562 KOps/s	3.3761 KOps/s	$\textbf{\color{#d91a1a}-6.51\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.9483ms	5.7214ms	174.7827 Ops/s	174.0030 Ops/s	$\color{#35bf28}+0.45\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.6953s	1.2312ms	812.2435 Ops/s	2.8466 KOps/s	$\textbf{\color{#d91a1a}-71.47\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.6903ms	0.3289ms	3.0403 KOps/s	3.0283 KOps/s	$\color{#35bf28}+0.40\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	6.0357ms	5.9380ms	168.4079 Ops/s	167.6439 Ops/s	$\color{#35bf28}+0.46\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	2.1143ms	0.4693ms	2.1307 KOps/s	2.1815 KOps/s	$\color{#d91a1a}-2.33\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.6345ms	0.4339ms	2.3047 KOps/s	2.2579 KOps/s	$\color{#35bf28}+2.07\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	6.6695ms	5.0774ms	196.9493 Ops/s	195.2104 Ops/s	$\color{#35bf28}+0.89\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	10.6570ms	2.0783ms	481.1613 Ops/s	493.9600 Ops/s	$\color{#d91a1a}-2.59\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	5.8053ms	1.2067ms	828.7311 Ops/s	849.2396 Ops/s	$\color{#d91a1a}-2.41\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.4974s	14.9112ms	67.0636 Ops/s	191.9493 Ops/s	$\textbf{\color{#d91a1a}-65.06\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	3.9909ms	1.7465ms	572.5694 Ops/s	482.3508 Ops/s	$\textbf{\color{#35bf28}+18.70\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	10.8080ms	1.2410ms	805.8225 Ops/s	894.8826 Ops/s	$\textbf{\color{#d91a1a}-9.95\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	8.1314ms	5.2947ms	188.8681 Ops/s	55.1004 Ops/s	$\textbf{\color{#35bf28}+242.77\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	8.0315ms	2.1777ms	459.2042 Ops/s	542.0873 Ops/s	$\textbf{\color{#d91a1a}-15.29\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	8.3579ms	1.3663ms	731.9059 Ops/s	987.0115 Ops/s	$\textbf{\color{#d91a1a}-25.85\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True]	35.7517ms	33.3080ms	30.0228 Ops/s	30.3232 Ops/s	$\color{#d91a1a}-0.99\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False]	19.0206ms	17.4714ms	57.2363 Ops/s	55.0293 Ops/s	$\color{#35bf28}+4.01\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True]	36.5840ms	33.9176ms	29.4832 Ops/s	29.4916 Ops/s	$\color{#d91a1a}-0.03\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False]	19.6202ms	18.0213ms	55.4899 Ops/s	55.3907 Ops/s	$\color{#35bf28}+0.18\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True]	37.7683ms	35.8225ms	27.9154 Ops/s	28.2244 Ops/s	$\color{#d91a1a}-1.09\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False]	20.0372ms	18.9580ms	52.7483 Ops/s	51.4775 Ops/s	$\color{#35bf28}+2.47\%$