[BugFix] Fix sampling of last item in SliceSampler

[vmoens]

Fixed a bug reported here where it was pointed that the last item of a trajectory was always missed by SliceSampler.

This was due to sampling the start item of the trajectory according to floor((traj_len - sample_len) * rand()) + start_idx instead of floor((traj_len - sample_len + 1) * rand()) + start_idx. To see why the second is right and first wrong, one can thing of a trajectory of traj_len=2, sample_len=1 and start_idx=0. It is clear that the first will only sample a start index of 0 and never 1.

cc @nicklashansen @dasGringuen

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1774

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (23 Unrelated Failures)

As of commit f75ce5e with merge base 975a205 ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Brax Tests on Linux / unittests (3.9, 12.1) / linux-job (gh)
  The process '/usr/bin/git' failed with exit code 128
• Build M1 Wheels / pytorch/rl / wheel-py3_8-cpu (gh)
  ImportError: cannot import name 'MemoryMappedTensor' from 'tensordict' (/Users/ec2-user/runner/_work/_temp/conda_environment_7445110520/lib/python3.8/site-packages/tensordict/__init__.py)
• Envpool Tests on Linux / unittests (3.9, 12.1) / linux-job (gh)
  The process '/usr/bin/git' failed with exit code 128
• Examples Tests on Linux / tests (3.9, 12.1) / linux-job (gh)
  The process '/usr/bin/git' failed with exit code 128
• Generate documentation / build-docs (3.9, 12.1) / linux-job (gh)
  The process '/usr/bin/git' failed with exit code 128
• Habitat Tests on Linux / tests (3.9, 11.6) / linux-job (gh)
  The process '/usr/bin/git' failed with exit code 128
• Jumanji Tests on Linux / unittests (3.9, 12.1) / linux-job (gh)
  The process '/usr/bin/git' failed with exit code 128
• Olddeps Tests on Linux / unittests (3.8, 11.6) / linux-job (gh)
  The process '/usr/bin/git' failed with exit code 128
• Optdeps Tests on Linux / tests (3.9, 12.1) / linux-job (gh)
  The process '/usr/bin/git' failed with exit code 128
• PettingZoo Tests on Linux / unittests / linux-job (gh)
  The process '/usr/bin/git' failed with exit code 128
• RLHF Tests on Linux / unittests (3.9, 12.1) / linux-job (gh)
  The process '/usr/bin/git' failed with exit code 128
• Robohive Tests on Linux / unittests (3.9, 12.1) / linux-job (gh)
  The process '/usr/bin/git' failed with exit code 128
• Sklearn Tests on Linux / unittests (3.9, 12.1) / linux-job (gh)
  The process '/usr/bin/git' failed with exit code 128
• Tests on Linux GPU (Gym) / unittests (3.9, 12.1) / linux-job (gh)
  The process '/usr/bin/git' failed with exit code 128
• Unit-tests on Linux CPU / tests (3.10) / linux-job (gh)
  test/test_libs.py::TestOpenML::test_env[batch_size0-mushroom_num]
• Unit-tests on Linux CPU / tests (3.8) / linux-job (gh)
  test/test_libs.py::TestOpenML::test_env[batch_size0-mushroom_num]
• Unit-tests on Linux CPU / tests (3.9) / linux-job (gh)
  test/test_libs.py::TestOpenML::test_env[batch_size0-mushroom_num]
• Unit-tests on Linux GPU / tests (3.8, 12.1) / linux-job (gh)
  The process '/usr/bin/git' failed with exit code 128
• Unit-tests on Linux GPU, latest stable release / tests (3.8, 11.8) / linux-job (gh)
  The process '/usr/bin/git' failed with exit code 128
• Unit-tests on MacOS CPU / tests (3.11) / macos-job (gh)
  The process '/usr/local/bin/git' failed with exit code 128
• Unit-tests on MacOS CPU / tests (3.8) / macos-job (gh)
  The process '/usr/local/bin/git' failed with exit code 128
• Unit-tests on Windows / unittests-cpu / windows-job (gh)
  The process 'C:\Program Files\Git\cmd\git.exe' failed with exit code 128
• Vmas Tests on Linux / unittests (3.9, 12.1) / linux-job (gh)
  The process '/usr/bin/git' failed with exit code 128

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 89. Improved: $\large\color{#35bf28}2$. Worsened: $\large\color{#d91a1a}3$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	0.1462s	71.1276ms	14.0592 Ops/s	14.9668 Ops/s	$\textbf{\color{#d91a1a}-6.06\%}$
test_sync	41.8154ms	36.1443ms	27.6669 Ops/s	28.0447 Ops/s	$\color{#d91a1a}-1.35\%$
test_async	0.1197s	35.4810ms	28.1841 Ops/s	28.4183 Ops/s	$\color{#d91a1a}-0.82\%$
test_simple	0.5287s	0.4600s	2.1738 Ops/s	2.1387 Ops/s	$\color{#35bf28}+1.64\%$
test_transformed	0.6910s	0.6281s	1.5922 Ops/s	1.5746 Ops/s	$\color{#35bf28}+1.12\%$
test_serial	1.4726s	1.4087s	0.7099 Ops/s	0.7007 Ops/s	$\color{#35bf28}+1.31\%$
test_parallel	1.4370s	1.4024s	0.7131 Ops/s	0.7267 Ops/s	$\color{#d91a1a}-1.87\%$
test_step_mdp_speed[True-True-True-True-True]	0.2037ms	21.9060μs	45.6497 KOps/s	45.5782 KOps/s	$\color{#35bf28}+0.16\%$
test_step_mdp_speed[True-True-True-True-False]	50.1140μs	13.3782μs	74.7482 KOps/s	74.8967 KOps/s	$\color{#d91a1a}-0.20\%$
test_step_mdp_speed[True-True-True-False-True]	55.9450μs	12.9602μs	77.1591 KOps/s	76.7109 KOps/s	$\color{#35bf28}+0.58\%$
test_step_mdp_speed[True-True-True-False-False]	67.1260μs	7.9363μs	126.0034 KOps/s	129.3621 KOps/s	$\color{#d91a1a}-2.60\%$
test_step_mdp_speed[True-True-False-True-True]	56.7170μs	23.6030μs	42.3675 KOps/s	43.2989 KOps/s	$\color{#d91a1a}-2.15\%$
test_step_mdp_speed[True-True-False-True-False]	88.4320μs	14.4862μs	69.0314 KOps/s	68.7298 KOps/s	$\color{#35bf28}+0.44\%$
test_step_mdp_speed[True-True-False-False-True]	0.1176ms	14.5661μs	68.6527 KOps/s	69.6660 KOps/s	$\color{#d91a1a}-1.45\%$
test_step_mdp_speed[True-True-False-False-False]	46.0060μs	9.0500μs	110.4974 KOps/s	110.2928 KOps/s	$\color{#35bf28}+0.19\%$
test_step_mdp_speed[True-False-True-True-True]	73.8480μs	24.4232μs	40.9446 KOps/s	40.6671 KOps/s	$\color{#35bf28}+0.68\%$
test_step_mdp_speed[True-False-True-True-False]	43.6410μs	15.9681μs	62.6249 KOps/s	62.4414 KOps/s	$\color{#35bf28}+0.29\%$
test_step_mdp_speed[True-False-True-False-True]	61.9760μs	14.1799μs	70.5225 KOps/s	71.1687 KOps/s	$\color{#d91a1a}-0.91\%$
test_step_mdp_speed[True-False-True-False-False]	41.1170μs	9.0939μs	109.9639 KOps/s	110.5971 KOps/s	$\color{#d91a1a}-0.57\%$
test_step_mdp_speed[True-False-False-True-True]	80.0590μs	26.0007μs	38.4605 KOps/s	38.6393 KOps/s	$\color{#d91a1a}-0.46\%$
test_step_mdp_speed[True-False-False-True-False]	61.0150μs	17.1479μs	58.3163 KOps/s	57.9619 KOps/s	$\color{#35bf28}+0.61\%$
test_step_mdp_speed[True-False-False-False-True]	47.3190μs	15.5036μs	64.5010 KOps/s	65.2394 KOps/s	$\color{#d91a1a}-1.13\%$
test_step_mdp_speed[True-False-False-False-False]	53.8500μs	10.2682μs	97.3885 KOps/s	97.3510 KOps/s	$\color{#35bf28}+0.04\%$
test_step_mdp_speed[False-True-True-True-True]	56.9360μs	24.4673μs	40.8709 KOps/s	40.4115 KOps/s	$\color{#35bf28}+1.14\%$
test_step_mdp_speed[False-True-True-True-False]	67.4360μs	16.0213μs	62.4167 KOps/s	61.6863 KOps/s	$\color{#35bf28}+1.18\%$
test_step_mdp_speed[False-True-True-False-True]	73.8980μs	16.4129μs	60.9276 KOps/s	60.1364 KOps/s	$\color{#35bf28}+1.32\%$
test_step_mdp_speed[False-True-True-False-False]	86.9520μs	10.4274μs	95.9008 KOps/s	96.4513 KOps/s	$\color{#d91a1a}-0.57\%$
test_step_mdp_speed[False-True-False-True-True]	60.0930μs	25.6495μs	38.9871 KOps/s	39.7470 KOps/s	$\color{#d91a1a}-1.91\%$
test_step_mdp_speed[False-True-False-True-False]	63.3580μs	16.9896μs	58.8595 KOps/s	57.1332 KOps/s	$\color{#35bf28}+3.02\%$
test_step_mdp_speed[False-True-False-False-True]	45.3350μs	17.4593μs	57.2759 KOps/s	56.4071 KOps/s	$\color{#35bf28}+1.54\%$
test_step_mdp_speed[False-True-False-False-False]	41.5680μs	11.6480μs	85.8517 KOps/s	85.1998 KOps/s	$\color{#35bf28}+0.77\%$
test_step_mdp_speed[False-False-True-True-True]	79.4090μs	26.8698μs	37.2165 KOps/s	36.0049 KOps/s	$\color{#35bf28}+3.37\%$
test_step_mdp_speed[False-False-True-True-False]	60.9750μs	18.7757μs	53.2602 KOps/s	53.5921 KOps/s	$\color{#d91a1a}-0.62\%$
test_step_mdp_speed[False-False-True-False-True]	67.3260μs	17.6746μs	56.5784 KOps/s	55.9895 KOps/s	$\color{#35bf28}+1.05\%$
test_step_mdp_speed[False-False-True-False-False]	39.9750μs	11.7390μs	85.1864 KOps/s	85.8227 KOps/s	$\color{#d91a1a}-0.74\%$
test_step_mdp_speed[False-False-False-True-True]	79.4490μs	28.5917μs	34.9752 KOps/s	35.2101 KOps/s	$\color{#d91a1a}-0.67\%$
test_step_mdp_speed[False-False-False-True-False]	51.3560μs	20.0003μs	49.9992 KOps/s	50.8782 KOps/s	$\color{#d91a1a}-1.73\%$
test_step_mdp_speed[False-False-False-False-True]	44.4130μs	18.9076μs	52.8888 KOps/s	53.6971 KOps/s	$\color{#d91a1a}-1.51\%$
test_step_mdp_speed[False-False-False-False-False]	55.9250μs	12.9559μs	77.1847 KOps/s	77.9810 KOps/s	$\color{#d91a1a}-1.02\%$
test_values[generalized_advantage_estimate-True-True]	23.4583ms	12.5918ms	79.4168 Ops/s	76.8376 Ops/s	$\color{#35bf28}+3.36\%$
test_values[vec_generalized_advantage_estimate-True-True]	36.3500ms	26.7496ms	37.3837 Ops/s	36.4620 Ops/s	$\color{#35bf28}+2.53\%$
test_values[td0_return_estimate-False-False]	0.3423ms	0.2106ms	4.7487 KOps/s	4.9593 KOps/s	$\color{#d91a1a}-4.25\%$
test_values[td1_return_estimate-False-False]	30.1480ms	26.3110ms	38.0069 Ops/s	37.4268 Ops/s	$\color{#35bf28}+1.55\%$
test_values[vec_td1_return_estimate-False-False]	36.8711ms	27.2376ms	36.7140 Ops/s	35.9013 Ops/s	$\color{#35bf28}+2.26\%$
test_values[td_lambda_return_estimate-True-False]	37.6995ms	36.7806ms	27.1882 Ops/s	26.9237 Ops/s	$\color{#35bf28}+0.98\%$
test_values[vec_td_lambda_return_estimate-True-False]	35.2963ms	26.9601ms	37.0918 Ops/s	36.0397 Ops/s	$\color{#35bf28}+2.92\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	9.0150ms	8.1696ms	122.4049 Ops/s	123.0735 Ops/s	$\color{#d91a1a}-0.54\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	3.6225ms	1.8672ms	535.5632 Ops/s	530.8563 Ops/s	$\color{#35bf28}+0.89\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	11.4400ms	0.4463ms	2.2409 KOps/s	2.2717 KOps/s	$\color{#d91a1a}-1.36\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	41.2029ms	34.5551ms	28.9393 Ops/s	25.4237 Ops/s	$\textbf{\color{#35bf28}+13.83\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	12.9669ms	2.7451ms	364.2914 Ops/s	364.3785 Ops/s	$\color{#d91a1a}-0.02\%$
test_dqn_speed	93.1921ms	8.7275ms	114.5805 Ops/s	124.3014 Ops/s	$\textbf{\color{#d91a1a}-7.82\%}$
test_ddpg_speed	23.3740ms	15.4981ms	64.5241 Ops/s	64.7122 Ops/s	$\color{#d91a1a}-0.29\%$
test_sac_speed	43.2298ms	32.0453ms	31.2058 Ops/s	31.6054 Ops/s	$\color{#d91a1a}-1.26\%$
test_redq_speed	48.4012ms	37.7035ms	26.5228 Ops/s	26.6524 Ops/s	$\color{#d91a1a}-0.49\%$
test_redq_deprec_speed	33.5329ms	27.9346ms	35.7979 Ops/s	35.4317 Ops/s	$\color{#35bf28}+1.03\%$
test_td3_speed	31.7302ms	22.1894ms	45.0665 Ops/s	44.1874 Ops/s	$\color{#35bf28}+1.99\%$
test_cql_speed	0.1010s	92.7211ms	10.7850 Ops/s	10.7652 Ops/s	$\color{#35bf28}+0.18\%$
test_a2c_speed	42.1403ms	28.2884ms	35.3502 Ops/s	34.4973 Ops/s	$\color{#35bf28}+2.47\%$
test_ppo_speed	37.1348ms	28.5716ms	34.9998 Ops/s	34.5778 Ops/s	$\color{#35bf28}+1.22\%$
test_reinforce_speed	33.4342ms	27.3132ms	36.6123 Ops/s	36.8570 Ops/s	$\color{#d91a1a}-0.66\%$
test_iql_speed	77.1145ms	67.1371ms	14.8949 Ops/s	14.7631 Ops/s	$\color{#35bf28}+0.89\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	2.4183ms	1.6709ms	598.4911 Ops/s	583.1099 Ops/s	$\color{#35bf28}+2.64\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	12.1514ms	0.5458ms	1.8322 KOps/s	1.8339 KOps/s	$\color{#d91a1a}-0.09\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	8.9855ms	0.5303ms	1.8856 KOps/s	1.9020 KOps/s	$\color{#d91a1a}-0.86\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	2.5765ms	1.7295ms	578.2112 Ops/s	563.6724 Ops/s	$\color{#35bf28}+2.58\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	8.9621ms	0.5443ms	1.8373 KOps/s	1.8162 KOps/s	$\color{#35bf28}+1.16\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	6.3178ms	0.5293ms	1.8892 KOps/s	1.8648 KOps/s	$\color{#35bf28}+1.31\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	2.9172ms	1.9483ms	513.2758 Ops/s	499.9691 Ops/s	$\color{#35bf28}+2.66\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	4.3123ms	0.6845ms	1.4609 KOps/s	1.4164 KOps/s	$\color{#35bf28}+3.14\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	9.2473ms	0.6807ms	1.4691 KOps/s	1.5012 KOps/s	$\color{#d91a1a}-2.14\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	0.1413s	1.9725ms	506.9671 Ops/s	612.0183 Ops/s	$\textbf{\color{#d91a1a}-17.16\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	13.1250ms	0.5612ms	1.7818 KOps/s	1.8034 KOps/s	$\color{#d91a1a}-1.20\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	7.0924ms	0.5394ms	1.8540 KOps/s	1.7952 KOps/s	$\color{#35bf28}+3.28\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.2572ms	1.7539ms	570.1460 Ops/s	565.9688 Ops/s	$\color{#35bf28}+0.74\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	2.4180ms	0.5421ms	1.8446 KOps/s	1.8168 KOps/s	$\color{#35bf28}+1.53\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	12.5591ms	0.5512ms	1.8142 KOps/s	1.8845 KOps/s	$\color{#d91a1a}-3.73\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.3466ms	2.0913ms	478.1722 Ops/s	486.8862 Ops/s	$\color{#d91a1a}-1.79\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	9.2822ms	0.7112ms	1.4062 KOps/s	1.3593 KOps/s	$\color{#35bf28}+3.45\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8150ms	0.6766ms	1.4780 KOps/s	1.4682 KOps/s	$\color{#35bf28}+0.67\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1630s	18.9112ms	52.8787 Ops/s	53.8049 Ops/s	$\color{#d91a1a}-1.72\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	16.8048ms	12.4845ms	80.0991 Ops/s	77.9269 Ops/s	$\color{#35bf28}+2.79\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	4.7870ms	1.6066ms	622.4447 Ops/s	614.9227 Ops/s	$\color{#35bf28}+1.22\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1318s	17.8465ms	56.0335 Ops/s	55.6547 Ops/s	$\color{#35bf28}+0.68\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	14.7058ms	12.2597ms	81.5681 Ops/s	78.4142 Ops/s	$\color{#35bf28}+4.02\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	5.1255ms	1.6385ms	610.3178 Ops/s	603.3788 Ops/s	$\color{#35bf28}+1.15\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1326s	18.1078ms	55.2247 Ops/s	53.5297 Ops/s	$\color{#35bf28}+3.17\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	15.0254ms	12.5664ms	79.5771 Ops/s	75.5301 Ops/s	$\textbf{\color{#35bf28}+5.36\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	2.4139ms	1.7043ms	586.7559 Ops/s	578.8096 Ops/s	$\color{#35bf28}+1.37\%$

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 92. Improved: $\large\color{#35bf28}7$. Worsened: $\large\color{#d91a1a}3$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	0.1270s	0.1217s	8.2153 Ops/s	8.1044 Ops/s	$\color{#35bf28}+1.37\%$
test_sync	0.1768s	0.1104s	9.0577 Ops/s	9.1165 Ops/s	$\color{#d91a1a}-0.64\%$
test_async	0.2707s	0.1001s	9.9947 Ops/s	9.9486 Ops/s	$\color{#35bf28}+0.46\%$
test_single_pixels	0.1456s	0.1445s	6.9196 Ops/s	6.8587 Ops/s	$\color{#35bf28}+0.89\%$
test_sync_pixels	95.6771ms	94.9353ms	10.5335 Ops/s	10.4508 Ops/s	$\color{#35bf28}+0.79\%$
test_async_pixels	0.2513s	92.6057ms	10.7985 Ops/s	10.8515 Ops/s	$\color{#d91a1a}-0.49\%$
test_simple	0.9657s	0.9016s	1.1091 Ops/s	1.1020 Ops/s	$\color{#35bf28}+0.64\%$
test_transformed	1.1892s	1.1223s	0.8911 Ops/s	0.8838 Ops/s	$\color{#35bf28}+0.82\%$
test_serial	2.5259s	2.4713s	0.4046 Ops/s	0.3912 Ops/s	$\color{#35bf28}+3.45\%$
test_parallel	2.5655s	2.4927s	0.4012 Ops/s	0.4021 Ops/s	$\color{#d91a1a}-0.24\%$
test_step_mdp_speed[True-True-True-True-True]	94.4510μs	34.2000μs	29.2398 KOps/s	29.7793 KOps/s	$\color{#d91a1a}-1.81\%$
test_step_mdp_speed[True-True-True-True-False]	45.3800μs	19.6789μs	50.8159 KOps/s	50.4966 KOps/s	$\color{#35bf28}+0.63\%$
test_step_mdp_speed[True-True-True-False-True]	44.7910μs	19.2047μs	52.0707 KOps/s	53.1505 KOps/s	$\color{#d91a1a}-2.03\%$
test_step_mdp_speed[True-True-True-False-False]	30.4510μs	11.1524μs	89.6671 KOps/s	88.8882 KOps/s	$\color{#35bf28}+0.88\%$
test_step_mdp_speed[True-True-False-True-True]	67.0010μs	35.5440μs	28.1341 KOps/s	29.2136 KOps/s	$\color{#d91a1a}-3.70\%$
test_step_mdp_speed[True-True-False-True-False]	44.9310μs	21.3135μs	46.9187 KOps/s	46.8607 KOps/s	$\color{#35bf28}+0.12\%$
test_step_mdp_speed[True-True-False-False-True]	53.0910μs	21.1055μs	47.3810 KOps/s	49.2232 KOps/s	$\color{#d91a1a}-3.74\%$
test_step_mdp_speed[True-True-False-False-False]	34.3010μs	13.0595μs	76.5725 KOps/s	75.7022 KOps/s	$\color{#35bf28}+1.15\%$
test_step_mdp_speed[True-False-True-True-True]	66.4710μs	37.3551μs	26.7701 KOps/s	26.9062 KOps/s	$\color{#d91a1a}-0.51\%$
test_step_mdp_speed[True-False-True-True-False]	47.8000μs	23.3348μs	42.8545 KOps/s	42.4145 KOps/s	$\color{#35bf28}+1.04\%$
test_step_mdp_speed[True-False-True-False-True]	51.1610μs	20.7784μs	48.1268 KOps/s	48.7855 KOps/s	$\color{#d91a1a}-1.35\%$
test_step_mdp_speed[True-False-True-False-False]	34.5610μs	12.9995μs	76.9260 KOps/s	75.9796 KOps/s	$\color{#35bf28}+1.25\%$
test_step_mdp_speed[True-False-False-True-True]	75.9710μs	39.0669μs	25.5971 KOps/s	26.2530 KOps/s	$\color{#d91a1a}-2.50\%$
test_step_mdp_speed[True-False-False-True-False]	48.4410μs	24.9448μs	40.0885 KOps/s	39.6466 KOps/s	$\color{#35bf28}+1.11\%$
test_step_mdp_speed[True-False-False-False-True]	50.3310μs	22.5999μs	44.2480 KOps/s	44.9218 KOps/s	$\color{#d91a1a}-1.50\%$
test_step_mdp_speed[True-False-False-False-False]	59.7810μs	14.8871μs	67.1723 KOps/s	66.5993 KOps/s	$\color{#35bf28}+0.86\%$
test_step_mdp_speed[False-True-True-True-True]	64.3710μs	37.1699μs	26.9035 KOps/s	27.1564 KOps/s	$\color{#d91a1a}-0.93\%$
test_step_mdp_speed[False-True-True-True-False]	51.2910μs	23.2010μs	43.1016 KOps/s	42.2973 KOps/s	$\color{#35bf28}+1.90\%$
test_step_mdp_speed[False-True-True-False-True]	56.3600μs	25.0532μs	39.9150 KOps/s	39.7144 KOps/s	$\color{#35bf28}+0.51\%$
test_step_mdp_speed[False-True-True-False-False]	45.8200μs	14.8965μs	67.1297 KOps/s	66.7939 KOps/s	$\color{#35bf28}+0.50\%$
test_step_mdp_speed[False-True-False-True-True]	69.9710μs	39.1696μs	25.5300 KOps/s	26.2397 KOps/s	$\color{#d91a1a}-2.70\%$
test_step_mdp_speed[False-True-False-True-False]	50.9200μs	25.1209μs	39.8075 KOps/s	38.8593 KOps/s	$\color{#35bf28}+2.44\%$
test_step_mdp_speed[False-True-False-False-True]	61.8510μs	27.0679μs	36.9441 KOps/s	37.5159 KOps/s	$\color{#d91a1a}-1.52\%$
test_step_mdp_speed[False-True-False-False-False]	41.0410μs	16.7598μs	59.6666 KOps/s	58.8126 KOps/s	$\color{#35bf28}+1.45\%$
test_step_mdp_speed[False-False-True-True-True]	75.4410μs	40.5929μs	24.6348 KOps/s	24.8418 KOps/s	$\color{#d91a1a}-0.83\%$
test_step_mdp_speed[False-False-True-True-False]	59.8610μs	26.8347μs	37.2652 KOps/s	36.4237 KOps/s	$\color{#35bf28}+2.31\%$
test_step_mdp_speed[False-False-True-False-True]	51.3000μs	27.0991μs	36.9016 KOps/s	37.6438 KOps/s	$\color{#d91a1a}-1.97\%$
test_step_mdp_speed[False-False-True-False-False]	53.1700μs	16.7689μs	59.6342 KOps/s	59.7490 KOps/s	$\color{#d91a1a}-0.19\%$
test_step_mdp_speed[False-False-False-True-True]	91.4910μs	42.1021μs	23.7518 KOps/s	23.9237 KOps/s	$\color{#d91a1a}-0.72\%$
test_step_mdp_speed[False-False-False-True-False]	55.2510μs	28.9917μs	34.4927 KOps/s	34.1644 KOps/s	$\color{#35bf28}+0.96\%$
test_step_mdp_speed[False-False-False-False-True]	56.2200μs	28.3232μs	35.3068 KOps/s	35.3310 KOps/s	$\color{#d91a1a}-0.07\%$
test_step_mdp_speed[False-False-False-False-False]	38.7400μs	18.7839μs	53.2371 KOps/s	53.6252 KOps/s	$\color{#d91a1a}-0.72\%$
test_values[generalized_advantage_estimate-True-True]	25.1546ms	24.5911ms	40.6651 Ops/s	37.4754 Ops/s	$\textbf{\color{#35bf28}+8.51\%}$
test_values[vec_generalized_advantage_estimate-True-True]	91.5792ms	3.3986ms	294.2409 Ops/s	307.6874 Ops/s	$\color{#d91a1a}-4.37\%$
test_values[td0_return_estimate-False-False]	95.5310μs	62.0002μs	16.1290 KOps/s	16.1403 KOps/s	$\color{#d91a1a}-0.07\%$
test_values[td1_return_estimate-False-False]	53.1061ms	52.3985ms	19.0845 Ops/s	18.6301 Ops/s	$\color{#35bf28}+2.44\%$
test_values[vec_td1_return_estimate-False-False]	1.9964ms	1.7681ms	565.5748 Ops/s	566.3714 Ops/s	$\color{#d91a1a}-0.14\%$
test_values[td_lambda_return_estimate-True-False]	85.8299ms	83.7023ms	11.9471 Ops/s	11.7522 Ops/s	$\color{#35bf28}+1.66\%$
test_values[vec_td_lambda_return_estimate-True-False]	2.0221ms	1.7592ms	568.4275 Ops/s	565.8841 Ops/s	$\color{#35bf28}+0.45\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	23.4752ms	23.1784ms	43.1436 Ops/s	42.2958 Ops/s	$\color{#35bf28}+2.00\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	0.8451ms	0.6998ms	1.4290 KOps/s	1.3445 KOps/s	$\textbf{\color{#35bf28}+6.29\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.7103ms	0.6535ms	1.5302 KOps/s	1.5135 KOps/s	$\color{#35bf28}+1.10\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	1.5015ms	1.4559ms	686.8829 Ops/s	683.9761 Ops/s	$\color{#35bf28}+0.42\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	0.9427ms	0.6742ms	1.4832 KOps/s	1.4711 KOps/s	$\color{#35bf28}+0.83\%$
test_dqn_speed	13.8517ms	7.2611ms	137.7201 Ops/s	134.1147 Ops/s	$\color{#35bf28}+2.69\%$
test_ddpg_speed	15.0419ms	14.1254ms	70.7946 Ops/s	62.8884 Ops/s	$\textbf{\color{#35bf28}+12.57\%}$
test_sac_speed	29.8406ms	28.8805ms	34.6254 Ops/s	33.9587 Ops/s	$\color{#35bf28}+1.96\%$
test_redq_speed	35.7333ms	34.7708ms	28.7597 Ops/s	28.4745 Ops/s	$\color{#35bf28}+1.00\%$
test_redq_deprec_speed	24.9427ms	24.0223ms	41.6280 Ops/s	41.4801 Ops/s	$\color{#35bf28}+0.36\%$
test_td3_speed	28.5264ms	19.7162ms	50.7196 Ops/s	49.8940 Ops/s	$\color{#35bf28}+1.65\%$
test_cql_speed	84.5261ms	83.3997ms	11.9905 Ops/s	11.9027 Ops/s	$\color{#35bf28}+0.74\%$
test_a2c_speed	26.5127ms	26.3742ms	37.9159 Ops/s	37.1451 Ops/s	$\color{#35bf28}+2.08\%$
test_ppo_speed	27.8711ms	26.8081ms	37.3021 Ops/s	36.7049 Ops/s	$\color{#35bf28}+1.63\%$
test_reinforce_speed	26.1971ms	25.6258ms	39.0231 Ops/s	38.2839 Ops/s	$\color{#35bf28}+1.93\%$
test_iql_speed	57.4448ms	57.0462ms	17.5296 Ops/s	17.3915 Ops/s	$\color{#35bf28}+0.79\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	2.3447ms	1.9333ms	517.2572 Ops/s	518.7456 Ops/s	$\color{#d91a1a}-0.29\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	2.2192ms	0.7957ms	1.2568 KOps/s	1.2531 KOps/s	$\color{#35bf28}+0.29\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.9975ms	0.7833ms	1.2766 KOps/s	1.2718 KOps/s	$\color{#35bf28}+0.38\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	2.0367ms	1.8804ms	531.8068 Ops/s	530.8162 Ops/s	$\color{#35bf28}+0.19\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	2.0703ms	0.7828ms	1.2774 KOps/s	1.2723 KOps/s	$\color{#35bf28}+0.41\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.9983ms	0.7720ms	1.2954 KOps/s	1.2795 KOps/s	$\color{#35bf28}+1.25\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.2685ms	2.2253ms	449.3689 Ops/s	398.2726 Ops/s	$\textbf{\color{#35bf28}+12.83\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	3.4204ms	0.9214ms	1.0853 KOps/s	1.0827 KOps/s	$\color{#35bf28}+0.24\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	1.1334ms	0.9129ms	1.0954 KOps/s	1.0853 KOps/s	$\color{#35bf28}+0.93\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	2.4851ms	1.9441ms	514.3835 Ops/s	518.9631 Ops/s	$\color{#d91a1a}-0.88\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	2.1131ms	0.7950ms	1.2579 KOps/s	1.2533 KOps/s	$\color{#35bf28}+0.37\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.9724ms	0.7823ms	1.2783 KOps/s	1.2634 KOps/s	$\color{#35bf28}+1.19\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	2.6149ms	1.9184ms	521.2795 Ops/s	523.7799 Ops/s	$\color{#d91a1a}-0.48\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.9127ms	0.7827ms	1.2777 KOps/s	1.2668 KOps/s	$\color{#35bf28}+0.86\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	4.8328ms	0.7784ms	1.2848 KOps/s	1.2846 KOps/s	$+0.01\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.1049ms	2.2281ms	448.8110 Ops/s	449.5141 Ops/s	$\color{#d91a1a}-0.16\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	3.3654ms	0.9251ms	1.0810 KOps/s	1.0790 KOps/s	$\color{#35bf28}+0.18\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.1491s	1.0841ms	922.4268 Ops/s	1.0920 KOps/s	$\textbf{\color{#d91a1a}-15.53\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1231s	15.1911ms	65.8282 Ops/s	54.3671 Ops/s	$\textbf{\color{#35bf28}+21.08\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	17.6370ms	12.4702ms	80.1912 Ops/s	77.4356 Ops/s	$\color{#35bf28}+3.56\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	2.5330ms	1.8541ms	539.3527 Ops/s	525.7856 Ops/s	$\color{#35bf28}+2.58\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1241s	17.5315ms	57.0402 Ops/s	64.3640 Ops/s	$\textbf{\color{#d91a1a}-11.38\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	15.2349ms	12.3958ms	80.6724 Ops/s	65.6737 Ops/s	$\textbf{\color{#35bf28}+22.84\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	2.5587ms	1.8628ms	536.8372 Ops/s	552.8467 Ops/s	$\color{#d91a1a}-2.90\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1257s	17.7251ms	56.4171 Ops/s	63.6370 Ops/s	$\textbf{\color{#d91a1a}-11.35\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	15.4444ms	12.5478ms	79.6950 Ops/s	65.5948 Ops/s	$\textbf{\color{#35bf28}+21.50\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	5.4880ms	2.0443ms	489.1542 Ops/s	483.2184 Ops/s	$\color{#35bf28}+1.23\%$