[BugFix] Fix async gym env with non-sync resets

[vmoens]

No description provided.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2170

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 2 Unrelated Failures

As of commit ee2bab5 with merge base 78b4717 ():

NEW FAILURES - The following jobs have failed:

• Habitat Tests on Linux / tests (3.9, 12.1) / linux-job (gh)
  RuntimeError: Command docker exec -t 0334bd06a3ada815227541e67b341a754b30ae1ddf2f8877c9fbfd0fd5d82f0b /exec failed with exit code 139
• Unit-tests on Linux / tests-optdeps (3.10, 12.1) / linux-job (gh)
  RuntimeError: Command docker exec -t 7da452d8c4d79e157d308dd02b68a58a660a82ab36dec376f46c4faa2a071fc2 /exec failed with exit code 1
• Unit-tests on Windows / unittests-cpu / windows-job (gh)
  The process 'C:\Program Files\Git\cmd\git.exe' failed with exit code 128

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Examples Tests on Linux / tests (3.9, 12.1) / linux-job (gh) (trunk failure)
• Unit-tests on Linux / tests-olddeps (3.8, 11.6) / linux-job (gh) (trunk failure)
  test/test_transforms.py::TestVecNorm::test_state_dict_vecnorm

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 91. Improved: $\large\color{#35bf28}6$. Worsened: $\large\color{#d91a1a}5$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	0.1002s	55.7119ms	17.9495 Ops/s	18.4183 Ops/s	$\color{#d91a1a}-2.55\%$
test_sync	30.8612ms	29.5630ms	33.8260 Ops/s	33.9603 Ops/s	$\color{#d91a1a}-0.40\%$
test_async	56.3676ms	28.4222ms	35.1838 Ops/s	35.5096 Ops/s	$\color{#d91a1a}-0.92\%$
test_simple	0.4316s	0.3536s	2.8277 Ops/s	2.9443 Ops/s	$\color{#d91a1a}-3.96\%$
test_transformed	0.4842s	0.4818s	2.0757 Ops/s	2.0021 Ops/s	$\color{#35bf28}+3.67\%$
test_serial	1.2533s	1.2056s	0.8295 Ops/s	0.8126 Ops/s	$\color{#35bf28}+2.07\%$
test_parallel	1.0856s	1.0222s	0.9783 Ops/s	0.9804 Ops/s	$\color{#d91a1a}-0.22\%$
test_step_mdp_speed[True-True-True-True-True]	0.1699ms	21.5064μs	46.4977 KOps/s	45.6254 KOps/s	$\color{#35bf28}+1.91\%$
test_step_mdp_speed[True-True-True-True-False]	46.4560μs	13.0580μs	76.5816 KOps/s	74.4752 KOps/s	$\color{#35bf28}+2.83\%$
test_step_mdp_speed[True-True-True-False-True]	46.7270μs	12.9862μs	77.0049 KOps/s	77.1633 KOps/s	$\color{#d91a1a}-0.21\%$
test_step_mdp_speed[True-True-True-False-False]	31.8490μs	7.7408μs	129.1863 KOps/s	126.1562 KOps/s	$\color{#35bf28}+2.40\%$
test_step_mdp_speed[True-True-False-True-True]	61.8150μs	22.7808μs	43.8967 KOps/s	42.7651 KOps/s	$\color{#35bf28}+2.65\%$
test_step_mdp_speed[True-True-False-True-False]	50.3330μs	14.2841μs	70.0080 KOps/s	67.5723 KOps/s	$\color{#35bf28}+3.60\%$
test_step_mdp_speed[True-True-False-False-True]	76.5950μs	13.8517μs	72.1931 KOps/s	70.3717 KOps/s	$\color{#35bf28}+2.59\%$
test_step_mdp_speed[True-True-False-False-False]	29.4950μs	8.8358μs	113.1754 KOps/s	110.0841 KOps/s	$\color{#35bf28}+2.81\%$
test_step_mdp_speed[True-False-True-True-True]	63.3180μs	24.0299μs	41.6148 KOps/s	40.5109 KOps/s	$\color{#35bf28}+2.72\%$
test_step_mdp_speed[True-False-True-True-False]	37.5500μs	15.4741μs	64.6239 KOps/s	62.6179 KOps/s	$\color{#35bf28}+3.20\%$
test_step_mdp_speed[True-False-True-False-True]	41.8570μs	13.7586μs	72.6820 KOps/s	69.5463 KOps/s	$\color{#35bf28}+4.51\%$
test_step_mdp_speed[True-False-True-False-False]	32.0200μs	8.8074μs	113.5403 KOps/s	110.5904 KOps/s	$\color{#35bf28}+2.67\%$
test_step_mdp_speed[True-False-False-True-True]	64.2500μs	25.0701μs	39.8882 KOps/s	38.6987 KOps/s	$\color{#35bf28}+3.07\%$
test_step_mdp_speed[True-False-False-True-False]	45.6650μs	16.6817μs	59.9458 KOps/s	57.8161 KOps/s	$\color{#35bf28}+3.68\%$
test_step_mdp_speed[True-False-False-False-True]	53.3590μs	14.8210μs	67.4716 KOps/s	65.5640 KOps/s	$\color{#35bf28}+2.91\%$
test_step_mdp_speed[True-False-False-False-False]	39.6540μs	9.9739μs	100.2614 KOps/s	98.1075 KOps/s	$\color{#35bf28}+2.20\%$
test_step_mdp_speed[False-True-True-True-True]	61.0630μs	23.6610μs	42.2636 KOps/s	40.6986 KOps/s	$\color{#35bf28}+3.85\%$
test_step_mdp_speed[False-True-True-True-False]	34.9950μs	15.4374μs	64.7779 KOps/s	62.9664 KOps/s	$\color{#35bf28}+2.88\%$
test_step_mdp_speed[False-True-True-False-True]	47.3280μs	16.0052μs	62.4798 KOps/s	61.4017 KOps/s	$\color{#35bf28}+1.76\%$
test_step_mdp_speed[False-True-True-False-False]	43.5610μs	9.9449μs	100.5537 KOps/s	97.1678 KOps/s	$\color{#35bf28}+3.48\%$
test_step_mdp_speed[False-True-False-True-True]	54.1110μs	24.8044μs	40.3154 KOps/s	38.9592 KOps/s	$\color{#35bf28}+3.48\%$
test_step_mdp_speed[False-True-False-True-False]	39.0530μs	16.6098μs	60.2055 KOps/s	57.8150 KOps/s	$\color{#35bf28}+4.13\%$
test_step_mdp_speed[False-True-False-False-True]	64.2590μs	16.9784μs	58.8985 KOps/s	56.6057 KOps/s	$\color{#35bf28}+4.05\%$
test_step_mdp_speed[False-True-False-False-False]	33.3820μs	11.1347μs	89.8093 KOps/s	87.1999 KOps/s	$\color{#35bf28}+2.99\%$
test_step_mdp_speed[False-False-True-True-True]	57.1660μs	26.1595μs	38.2271 KOps/s	36.9114 KOps/s	$\color{#35bf28}+3.56\%$
test_step_mdp_speed[False-False-True-True-False]	45.1340μs	17.8453μs	56.0370 KOps/s	53.8911 KOps/s	$\color{#35bf28}+3.98\%$
test_step_mdp_speed[False-False-True-False-True]	46.7670μs	16.9313μs	59.0620 KOps/s	57.0533 KOps/s	$\color{#35bf28}+3.52\%$
test_step_mdp_speed[False-False-True-False-False]	31.8800μs	11.1330μs	89.8234 KOps/s	87.3441 KOps/s	$\color{#35bf28}+2.84\%$
test_step_mdp_speed[False-False-False-True-True]	48.3200μs	27.5202μs	36.3370 KOps/s	35.1808 KOps/s	$\color{#35bf28}+3.29\%$
test_step_mdp_speed[False-False-False-True-False]	0.2600ms	19.7514μs	50.6293 KOps/s	51.0895 KOps/s	$\color{#d91a1a}-0.90\%$
test_step_mdp_speed[False-False-False-False-True]	52.4670μs	17.9345μs	55.7585 KOps/s	53.9493 KOps/s	$\color{#35bf28}+3.35\%$
test_step_mdp_speed[False-False-False-False-False]	53.5290μs	12.2542μs	81.6048 KOps/s	77.7984 KOps/s	$\color{#35bf28}+4.89\%$
test_values[generalized_advantage_estimate-True-True]	15.8609ms	9.4828ms	105.4545 Ops/s	107.6857 Ops/s	$\color{#d91a1a}-2.07\%$
test_values[vec_generalized_advantage_estimate-True-True]	36.2416ms	33.4158ms	29.9259 Ops/s	28.3784 Ops/s	$\textbf{\color{#35bf28}+5.45\%}$
test_values[td0_return_estimate-False-False]	0.2283ms	0.1706ms	5.8629 KOps/s	5.9242 KOps/s	$\color{#d91a1a}-1.04\%$
test_values[td1_return_estimate-False-False]	23.8485ms	23.0128ms	43.4541 Ops/s	42.4916 Ops/s	$\color{#35bf28}+2.27\%$
test_values[vec_td1_return_estimate-False-False]	35.0070ms	33.5920ms	29.7690 Ops/s	28.3152 Ops/s	$\textbf{\color{#35bf28}+5.13\%}$
test_values[td_lambda_return_estimate-True-False]	36.9205ms	33.5881ms	29.7724 Ops/s	29.7025 Ops/s	$\color{#35bf28}+0.24\%$
test_values[vec_td_lambda_return_estimate-True-False]	34.8846ms	33.5194ms	29.8334 Ops/s	28.1808 Ops/s	$\textbf{\color{#35bf28}+5.86\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512]	8.3109ms	8.1794ms	122.2580 Ops/s	121.7915 Ops/s	$\color{#35bf28}+0.38\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	1.9499ms	1.8528ms	539.7243 Ops/s	559.4591 Ops/s	$\color{#d91a1a}-3.53\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.4712ms	0.3545ms	2.8211 KOps/s	2.8472 KOps/s	$\color{#d91a1a}-0.92\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	48.0150ms	46.2016ms	21.6443 Ops/s	22.5388 Ops/s	$\color{#d91a1a}-3.97\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	3.8109ms	3.0467ms	328.2272 Ops/s	325.0145 Ops/s	$\color{#35bf28}+0.99\%$
test_dqn_speed	2.5405ms	1.3474ms	742.1529 Ops/s	734.4932 Ops/s	$\color{#35bf28}+1.04\%$
test_ddpg_speed	3.5220ms	2.8588ms	349.7920 Ops/s	351.4139 Ops/s	$\color{#d91a1a}-0.46\%$
test_sac_speed	8.9782ms	8.3572ms	119.6567 Ops/s	118.7701 Ops/s	$\color{#35bf28}+0.75\%$
test_redq_speed	15.0132ms	13.2586ms	75.4227 Ops/s	76.4335 Ops/s	$\color{#d91a1a}-1.32\%$
test_redq_deprec_speed	14.5351ms	13.3837ms	74.7180 Ops/s	69.3327 Ops/s	$\textbf{\color{#35bf28}+7.77\%}$
test_td3_speed	15.9380ms	8.3837ms	119.2795 Ops/s	118.9158 Ops/s	$\color{#35bf28}+0.31\%$
test_cql_speed	55.2582ms	37.7431ms	26.4949 Ops/s	27.4557 Ops/s	$\color{#d91a1a}-3.50\%$
test_a2c_speed	8.6692ms	7.5170ms	133.0326 Ops/s	130.2821 Ops/s	$\color{#35bf28}+2.11\%$
test_ppo_speed	8.1609ms	7.7045ms	129.7945 Ops/s	127.9465 Ops/s	$\color{#35bf28}+1.44\%$
test_reinforce_speed	7.6429ms	6.6584ms	150.1859 Ops/s	150.0764 Ops/s	$\color{#35bf28}+0.07\%$
test_iql_speed	33.9712ms	32.9696ms	30.3310 Ops/s	30.2616 Ops/s	$\color{#35bf28}+0.23\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.3801ms	2.2932ms	436.0654 Ops/s	455.5286 Ops/s	$\color{#d91a1a}-4.27\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	1.0987ms	0.5234ms	1.9107 KOps/s	2.0108 KOps/s	$\color{#d91a1a}-4.98\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6825ms	0.4740ms	2.1096 KOps/s	2.1301 KOps/s	$\color{#d91a1a}-0.96\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.3739ms	2.3243ms	430.2355 Ops/s	474.0699 Ops/s	$\textbf{\color{#d91a1a}-9.25\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	1.0910ms	0.4950ms	2.0201 KOps/s	2.0599 KOps/s	$\color{#d91a1a}-1.93\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.7714ms	0.4692ms	2.1313 KOps/s	2.1634 KOps/s	$\color{#d91a1a}-1.48\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	1.8184ms	1.2256ms	815.8966 Ops/s	803.9242 Ops/s	$\color{#35bf28}+1.49\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	4.1541ms	1.1642ms	858.9276 Ops/s	848.7758 Ops/s	$\color{#35bf28}+1.20\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.3700ms	2.3334ms	428.5676 Ops/s	428.6161 Ops/s	$\color{#d91a1a}-0.01\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.1372ms	0.6134ms	1.6301 KOps/s	1.6178 KOps/s	$\color{#35bf28}+0.76\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.1091s	0.6682ms	1.4965 KOps/s	1.6329 KOps/s	$\textbf{\color{#d91a1a}-8.35\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	2.3105ms	2.1732ms	460.1601 Ops/s	451.3320 Ops/s	$\color{#35bf28}+1.96\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	1.0288ms	0.4947ms	2.0215 KOps/s	1.9831 KOps/s	$\color{#35bf28}+1.94\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.8183ms	0.4753ms	2.1040 KOps/s	2.0726 KOps/s	$\color{#35bf28}+1.52\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	2.4526ms	2.1499ms	465.1479 Ops/s	446.4225 Ops/s	$\color{#35bf28}+4.19\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.7747ms	0.4857ms	2.0589 KOps/s	2.0326 KOps/s	$\color{#35bf28}+1.29\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.1022s	0.5971ms	1.6748 KOps/s	2.1194 KOps/s	$\textbf{\color{#d91a1a}-20.98\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	2.5626ms	2.2816ms	438.2911 Ops/s	434.3418 Ops/s	$\color{#35bf28}+0.91\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.9259ms	0.6147ms	1.6267 KOps/s	1.6051 KOps/s	$\color{#35bf28}+1.35\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8259ms	0.5851ms	1.7091 KOps/s	1.6607 KOps/s	$\color{#35bf28}+2.92\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1070s	7.7606ms	128.8567 Ops/s	115.1991 Ops/s	$\textbf{\color{#35bf28}+11.86\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	16.2834ms	12.5879ms	79.4412 Ops/s	78.5477 Ops/s	$\color{#35bf28}+1.14\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	1.8166ms	1.0728ms	932.1781 Ops/s	921.0171 Ops/s	$\color{#35bf28}+1.21\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1133s	6.0921ms	164.1461 Ops/s	174.7255 Ops/s	$\textbf{\color{#d91a1a}-6.05\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	15.0747ms	12.2972ms	81.3193 Ops/s	79.0269 Ops/s	$\color{#35bf28}+2.90\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	1.1197ms	1.0440ms	957.8172 Ops/s	927.6466 Ops/s	$\color{#35bf28}+3.25\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1092s	6.0963ms	164.0338 Ops/s	125.9543 Ops/s	$\textbf{\color{#35bf28}+30.23\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	0.1191s	14.8280ms	67.4401 Ops/s	76.1771 Ops/s	$\textbf{\color{#d91a1a}-11.47\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	1.8996ms	1.3813ms	723.9695 Ops/s	726.3743 Ops/s	$\color{#d91a1a}-0.33\%$

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 94. Improved: $\large\color{#35bf28}2$. Worsened: $\large\color{#d91a1a}3$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	0.1173s	0.1165s	8.5809 Ops/s	8.7189 Ops/s	$\color{#d91a1a}-1.58\%$
test_sync	0.1008s	99.4566ms	10.0546 Ops/s	9.6996 Ops/s	$\color{#35bf28}+3.66\%$
test_async	0.1921s	97.2860ms	10.2790 Ops/s	10.1670 Ops/s	$\color{#35bf28}+1.10\%$
test_single_pixels	0.1297s	0.1284s	7.7875 Ops/s	7.9743 Ops/s	$\color{#d91a1a}-2.34\%$
test_sync_pixels	83.3231ms	81.7269ms	12.2359 Ops/s	12.3681 Ops/s	$\color{#d91a1a}-1.07\%$
test_async_pixels	0.1618s	66.6587ms	15.0018 Ops/s	14.7976 Ops/s	$\color{#35bf28}+1.38\%$
test_simple	0.7925s	0.7796s	1.2827 Ops/s	1.2928 Ops/s	$\color{#d91a1a}-0.78\%$
test_transformed	1.0316s	1.0271s	0.9737 Ops/s	0.9867 Ops/s	$\color{#d91a1a}-1.33\%$
test_serial	2.5473s	2.4817s	0.4030 Ops/s	0.4117 Ops/s	$\color{#d91a1a}-2.12\%$
test_parallel	2.3797s	2.3082s	0.4332 Ops/s	0.4306 Ops/s	$\color{#35bf28}+0.61\%$
test_step_mdp_speed[True-True-True-True-True]	92.3120μs	33.7265μs	29.6503 KOps/s	29.7946 KOps/s	$\color{#d91a1a}-0.48\%$
test_step_mdp_speed[True-True-True-True-False]	40.5310μs	20.0476μs	49.8813 KOps/s	50.2000 KOps/s	$\color{#d91a1a}-0.63\%$
test_step_mdp_speed[True-True-True-False-True]	37.8700μs	19.1465μs	52.2290 KOps/s	52.3302 KOps/s	$\color{#d91a1a}-0.19\%$
test_step_mdp_speed[True-True-True-False-False]	36.7820μs	11.4125μs	87.6232 KOps/s	86.9242 KOps/s	$\color{#35bf28}+0.80\%$
test_step_mdp_speed[True-True-False-True-True]	56.6310μs	34.9129μs	28.6427 KOps/s	28.3820 KOps/s	$\color{#35bf28}+0.92\%$
test_step_mdp_speed[True-True-False-True-False]	42.3600μs	21.6868μs	46.1109 KOps/s	46.1685 KOps/s	$\color{#d91a1a}-0.12\%$
test_step_mdp_speed[True-True-False-False-True]	43.7910μs	20.7538μs	48.1839 KOps/s	47.4673 KOps/s	$\color{#35bf28}+1.51\%$
test_step_mdp_speed[True-True-False-False-False]	62.0610μs	13.2553μs	75.4415 KOps/s	74.8467 KOps/s	$\color{#35bf28}+0.79\%$
test_step_mdp_speed[True-False-True-True-True]	61.9910μs	36.6264μs	27.3027 KOps/s	27.0203 KOps/s	$\color{#35bf28}+1.05\%$
test_step_mdp_speed[True-False-True-True-False]	51.8610μs	23.6077μs	42.3591 KOps/s	41.8335 KOps/s	$\color{#35bf28}+1.26\%$
test_step_mdp_speed[True-False-True-False-True]	61.9210μs	21.0826μs	47.4324 KOps/s	47.8901 KOps/s	$\color{#d91a1a}-0.96\%$
test_step_mdp_speed[True-False-True-False-False]	48.5800μs	13.3429μs	74.9462 KOps/s	74.3625 KOps/s	$\color{#35bf28}+0.78\%$
test_step_mdp_speed[True-False-False-True-True]	63.1910μs	38.5067μs	25.9695 KOps/s	25.6046 KOps/s	$\color{#35bf28}+1.43\%$
test_step_mdp_speed[True-False-False-True-False]	60.9210μs	25.2346μs	39.6282 KOps/s	39.0002 KOps/s	$\color{#35bf28}+1.61\%$
test_step_mdp_speed[True-False-False-False-True]	53.9410μs	23.1166μs	43.2589 KOps/s	43.6532 KOps/s	$\color{#d91a1a}-0.90\%$
test_step_mdp_speed[True-False-False-False-False]	33.0000μs	15.0695μs	66.3591 KOps/s	65.3397 KOps/s	$\color{#35bf28}+1.56\%$
test_step_mdp_speed[False-True-True-True-True]	55.7510μs	36.9122μs	27.0913 KOps/s	26.7392 KOps/s	$\color{#35bf28}+1.32\%$
test_step_mdp_speed[False-True-True-True-False]	57.5520μs	23.7686μs	42.0723 KOps/s	42.2113 KOps/s	$\color{#d91a1a}-0.33\%$
test_step_mdp_speed[False-True-True-False-True]	49.6300μs	24.8390μs	40.2592 KOps/s	39.9552 KOps/s	$\color{#35bf28}+0.76\%$
test_step_mdp_speed[False-True-True-False-False]	39.7110μs	15.0697μs	66.3582 KOps/s	65.1176 KOps/s	$\color{#35bf28}+1.91\%$
test_step_mdp_speed[False-True-False-True-True]	63.5810μs	38.9658μs	25.6635 KOps/s	25.4471 KOps/s	$\color{#35bf28}+0.85\%$
test_step_mdp_speed[False-True-False-True-False]	43.0510μs	25.3334μs	39.4735 KOps/s	39.1129 KOps/s	$\color{#35bf28}+0.92\%$
test_step_mdp_speed[False-True-False-False-True]	61.0510μs	26.6124μs	37.5764 KOps/s	37.0291 KOps/s	$\color{#35bf28}+1.48\%$
test_step_mdp_speed[False-True-False-False-False]	36.4900μs	17.1864μs	58.1856 KOps/s	58.5744 KOps/s	$\color{#d91a1a}-0.66\%$
test_step_mdp_speed[False-False-True-True-True]	65.7110μs	40.2975μs	24.8154 KOps/s	24.7064 KOps/s	$\color{#35bf28}+0.44\%$
test_step_mdp_speed[False-False-True-True-False]	45.5220μs	27.5174μs	36.3407 KOps/s	35.7315 KOps/s	$\color{#35bf28}+1.70\%$
test_step_mdp_speed[False-False-True-False-True]	45.1800μs	26.3354μs	37.9716 KOps/s	37.1105 KOps/s	$\color{#35bf28}+2.32\%$
test_step_mdp_speed[False-False-True-False-False]	41.1600μs	17.0149μs	58.7721 KOps/s	59.2332 KOps/s	$\color{#d91a1a}-0.78\%$
test_step_mdp_speed[False-False-False-True-True]	55.4100μs	42.5751μs	23.4879 KOps/s	23.1432 KOps/s	$\color{#35bf28}+1.49\%$
test_step_mdp_speed[False-False-False-True-False]	53.4910μs	29.3982μs	34.0157 KOps/s	33.4755 KOps/s	$\color{#35bf28}+1.61\%$
test_step_mdp_speed[False-False-False-False-True]	60.0910μs	28.0555μs	35.6437 KOps/s	35.4710 KOps/s	$\color{#35bf28}+0.49\%$
test_step_mdp_speed[False-False-False-False-False]	40.9010μs	18.4611μs	54.1680 KOps/s	53.6822 KOps/s	$\color{#35bf28}+0.91\%$
test_values[generalized_advantage_estimate-True-True]	25.4523ms	24.8729ms	40.2044 Ops/s	40.2145 Ops/s	$\color{#d91a1a}-0.03\%$
test_values[vec_generalized_advantage_estimate-True-True]	81.9926ms	3.2111ms	311.4242 Ops/s	299.2362 Ops/s	$\color{#35bf28}+4.07\%$
test_values[td0_return_estimate-False-False]	91.2910μs	64.4155μs	15.5242 KOps/s	15.5568 KOps/s	$\color{#d91a1a}-0.21\%$
test_values[td1_return_estimate-False-False]	53.7171ms	53.0337ms	18.8559 Ops/s	18.8187 Ops/s	$\color{#35bf28}+0.20\%$
test_values[vec_td1_return_estimate-False-False]	2.0644ms	1.7738ms	563.7698 Ops/s	565.2890 Ops/s	$\color{#d91a1a}-0.27\%$
test_values[td_lambda_return_estimate-True-False]	85.1759ms	84.4379ms	11.8430 Ops/s	11.8241 Ops/s	$\color{#35bf28}+0.16\%$
test_values[vec_td_lambda_return_estimate-True-False]	2.0731ms	1.7666ms	566.0568 Ops/s	566.2285 Ops/s	$\color{#d91a1a}-0.03\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	23.9051ms	23.5973ms	42.3778 Ops/s	42.5030 Ops/s	$\color{#d91a1a}-0.29\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	0.9237ms	0.7072ms	1.4140 KOps/s	1.4265 KOps/s	$\color{#d91a1a}-0.88\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.7205ms	0.6546ms	1.5276 KOps/s	1.5278 KOps/s	$\color{#d91a1a}-0.01\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	1.5147ms	1.4574ms	686.1581 Ops/s	685.3174 Ops/s	$\color{#35bf28}+0.12\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	0.9426ms	0.6714ms	1.4895 KOps/s	1.4782 KOps/s	$\color{#35bf28}+0.77\%$
test_dqn_speed	78.2010ms	1.5938ms	627.4235 Ops/s	696.2557 Ops/s	$\textbf{\color{#d91a1a}-9.89\%}$
test_ddpg_speed	3.2201ms	2.9504ms	338.9323 Ops/s	343.0322 Ops/s	$\color{#d91a1a}-1.20\%$
test_sac_speed	9.4596ms	8.5215ms	117.3507 Ops/s	119.1986 Ops/s	$\color{#d91a1a}-1.55\%$
test_redq_speed	11.6622ms	10.5104ms	95.1440 Ops/s	85.2750 Ops/s	$\textbf{\color{#35bf28}+11.57\%}$
test_redq_deprec_speed	12.1778ms	11.6981ms	85.4838 Ops/s	86.6372 Ops/s	$\color{#d91a1a}-1.33\%$
test_td3_speed	17.5458ms	8.5480ms	116.9871 Ops/s	119.7737 Ops/s	$\color{#d91a1a}-2.33\%$
test_cql_speed	27.6644ms	26.0695ms	38.3590 Ops/s	39.1105 Ops/s	$\color{#d91a1a}-1.92\%$
test_a2c_speed	6.0189ms	5.7439ms	174.0973 Ops/s	180.5788 Ops/s	$\color{#d91a1a}-3.59\%$
test_ppo_speed	6.3928ms	6.0216ms	166.0695 Ops/s	170.2577 Ops/s	$\color{#d91a1a}-2.46\%$
test_reinforce_speed	4.9469ms	4.6985ms	212.8347 Ops/s	222.2074 Ops/s	$\color{#d91a1a}-4.22\%$
test_iql_speed	21.3043ms	20.1997ms	49.5057 Ops/s	51.4042 Ops/s	$\color{#d91a1a}-3.69\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.0060ms	2.8565ms	350.0812 Ops/s	349.7636 Ops/s	$\color{#35bf28}+0.09\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	1.2450ms	0.6018ms	1.6618 KOps/s	1.6782 KOps/s	$\color{#d91a1a}-0.98\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.7639ms	0.5821ms	1.7178 KOps/s	1.7353 KOps/s	$\color{#d91a1a}-1.01\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.0920ms	2.8539ms	350.3927 Ops/s	348.8197 Ops/s	$\color{#35bf28}+0.45\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.7298ms	0.5922ms	1.6886 KOps/s	1.7133 KOps/s	$\color{#d91a1a}-1.44\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	4.8938ms	0.5742ms	1.7417 KOps/s	1.7711 KOps/s	$\color{#d91a1a}-1.66\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	1.6743ms	1.5470ms	646.4211 Ops/s	654.5390 Ops/s	$\color{#d91a1a}-1.24\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	5.7875ms	1.4833ms	674.1673 Ops/s	684.1929 Ops/s	$\color{#d91a1a}-1.47\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.0864ms	2.9734ms	336.3165 Ops/s	334.6113 Ops/s	$\color{#35bf28}+0.51\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.7264ms	0.7274ms	1.3748 KOps/s	1.3915 KOps/s	$\color{#d91a1a}-1.20\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8609ms	0.7040ms	1.4204 KOps/s	1.4237 KOps/s	$\color{#d91a1a}-0.24\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	2.9617ms	2.8425ms	351.8038 Ops/s	351.4966 Ops/s	$\color{#35bf28}+0.09\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	1.4245ms	0.6023ms	1.6603 KOps/s	1.6889 KOps/s	$\color{#d91a1a}-1.70\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.7330ms	0.5811ms	1.7210 KOps/s	1.7446 KOps/s	$\color{#d91a1a}-1.35\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.1306ms	2.8396ms	352.1683 Ops/s	348.1546 Ops/s	$\color{#35bf28}+1.15\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.7182ms	0.5920ms	1.6893 KOps/s	1.6993 KOps/s	$\color{#d91a1a}-0.59\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	4.9742ms	0.5807ms	1.7222 KOps/s	1.7655 KOps/s	$\color{#d91a1a}-2.46\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.0892ms	2.9918ms	334.2512 Ops/s	332.2514 Ops/s	$\color{#35bf28}+0.60\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.6355ms	0.7347ms	1.3611 KOps/s	1.3866 KOps/s	$\color{#d91a1a}-1.84\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8807ms	0.7021ms	1.4243 KOps/s	1.4154 KOps/s	$\color{#35bf28}+0.62\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1306s	7.4600ms	134.0482 Ops/s	132.3480 Ops/s	$\color{#35bf28}+1.28\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	22.8148ms	15.6777ms	63.7849 Ops/s	54.9789 Ops/s	$\textbf{\color{#35bf28}+16.02\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	1.3567ms	1.1935ms	837.8568 Ops/s	857.1143 Ops/s	$\color{#d91a1a}-2.25\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1179s	7.2079ms	138.7362 Ops/s	136.7714 Ops/s	$\color{#35bf28}+1.44\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	0.1303s	18.0476ms	55.4090 Ops/s	62.7851 Ops/s	$\textbf{\color{#d91a1a}-11.75\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	2.2641ms	1.1962ms	835.9600 Ops/s	849.8542 Ops/s	$\color{#d91a1a}-1.63\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1195s	7.6098ms	131.4097 Ops/s	131.3397 Ops/s	$\color{#35bf28}+0.05\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	18.7086ms	15.9839ms	62.5628 Ops/s	61.4290 Ops/s	$\color{#35bf28}+1.85\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	7.2335ms	1.6856ms	593.2719 Ops/s	650.9411 Ops/s	$\textbf{\color{#d91a1a}-8.86\%}$