[BugFix] Fix envpool

[vmoens]

Description

Fix envpool

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 89. Improved: $\large\color{#35bf28}4$. Worsened: $\large\color{#d91a1a}9$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	73.2706ms	72.7806ms	13.7399 Ops/s	13.7349 Ops/s	$\color{#35bf28}+0.04\%$
test_sync	0.1143s	42.1852ms	23.7050 Ops/s	23.7634 Ops/s	$\color{#d91a1a}-0.25\%$
test_async	0.1395s	39.0701ms	25.5950 Ops/s	25.4170 Ops/s	$\color{#35bf28}+0.70\%$
test_simple	0.6859s	0.6186s	1.6164 Ops/s	1.6268 Ops/s	$\color{#d91a1a}-0.64\%$
test_transformed	0.8378s	0.7842s	1.2752 Ops/s	1.2622 Ops/s	$\color{#35bf28}+1.03\%$
test_serial	1.7642s	1.7110s	0.5844 Ops/s	0.5831 Ops/s	$\color{#35bf28}+0.23\%$
test_parallel	1.5641s	1.4993s	0.6670 Ops/s	0.6575 Ops/s	$\color{#35bf28}+1.44\%$
test_step_mdp_speed[True-True-True-True-True]	0.2409ms	44.2272μs	22.6105 KOps/s	22.5366 KOps/s	$\color{#35bf28}+0.33\%$
test_step_mdp_speed[True-True-True-True-False]	2.3360ms	25.0578μs	39.9077 KOps/s	39.5451 KOps/s	$\color{#35bf28}+0.92\%$
test_step_mdp_speed[True-True-True-False-True]	0.1066ms	31.8124μs	31.4343 KOps/s	31.2751 KOps/s	$\color{#35bf28}+0.51\%$
test_step_mdp_speed[True-True-True-False-False]	0.1278ms	17.6691μs	56.5961 KOps/s	56.8836 KOps/s	$\color{#d91a1a}-0.51\%$
test_step_mdp_speed[True-True-False-True-True]	0.1300ms	45.5857μs	21.9367 KOps/s	21.5747 KOps/s	$\color{#35bf28}+1.68\%$
test_step_mdp_speed[True-True-False-True-False]	53.9010μs	26.7476μs	37.3865 KOps/s	36.9472 KOps/s	$\color{#35bf28}+1.19\%$
test_step_mdp_speed[True-True-False-False-True]	0.1251ms	33.6540μs	29.7142 KOps/s	29.8705 KOps/s	$\color{#d91a1a}-0.52\%$
test_step_mdp_speed[True-True-False-False-False]	0.1039ms	19.3972μs	51.5537 KOps/s	50.5542 KOps/s	$\color{#35bf28}+1.98\%$
test_step_mdp_speed[True-False-True-True-True]	96.5020μs	47.7644μs	20.9361 KOps/s	20.9235 KOps/s	$\color{#35bf28}+0.06\%$
test_step_mdp_speed[True-False-True-True-False]	0.1037ms	28.6397μs	34.9166 KOps/s	33.8049 KOps/s	$\color{#35bf28}+3.29\%$
test_step_mdp_speed[True-False-True-False-True]	64.2010μs	33.4169μs	29.9250 KOps/s	29.9520 KOps/s	$\color{#d91a1a}-0.09\%$
test_step_mdp_speed[True-False-True-False-False]	0.2201ms	19.3784μs	51.6039 KOps/s	51.5809 KOps/s	$\color{#35bf28}+0.04\%$
test_step_mdp_speed[True-False-False-True-True]	1.2514ms	49.3638μs	20.2578 KOps/s	20.1673 KOps/s	$\color{#35bf28}+0.45\%$
test_step_mdp_speed[True-False-False-True-False]	0.1976ms	30.2762μs	33.0293 KOps/s	33.0027 KOps/s	$\color{#35bf28}+0.08\%$
test_step_mdp_speed[True-False-False-False-True]	61.6010μs	35.0607μs	28.5220 KOps/s	28.5740 KOps/s	$\color{#d91a1a}-0.18\%$
test_step_mdp_speed[True-False-False-False-False]	66.0010μs	21.1338μs	47.3175 KOps/s	47.6328 KOps/s	$\color{#d91a1a}-0.66\%$
test_step_mdp_speed[False-True-True-True-True]	80.2010μs	47.1998μs	21.1865 KOps/s	20.6955 KOps/s	$\color{#35bf28}+2.37\%$
test_step_mdp_speed[False-True-True-True-False]	61.2010μs	28.7897μs	34.7347 KOps/s	34.7809 KOps/s	$\color{#d91a1a}-0.13\%$
test_step_mdp_speed[False-True-True-False-True]	79.0010μs	36.7727μs	27.1941 KOps/s	26.0574 KOps/s	$\color{#35bf28}+4.36\%$
test_step_mdp_speed[False-True-True-False-False]	3.4957ms	21.8396μs	45.7883 KOps/s	46.1628 KOps/s	$\color{#d91a1a}-0.81\%$
test_step_mdp_speed[False-True-False-True-True]	81.7010μs	48.7950μs	20.4939 KOps/s	19.9873 KOps/s	$\color{#35bf28}+2.53\%$
test_step_mdp_speed[False-True-False-True-False]	64.5010μs	30.2999μs	33.0034 KOps/s	32.9197 KOps/s	$\color{#35bf28}+0.25\%$
test_step_mdp_speed[False-True-False-False-True]	64.2010μs	38.3613μs	26.0679 KOps/s	25.2422 KOps/s	$\color{#35bf28}+3.27\%$
test_step_mdp_speed[False-True-False-False-False]	4.0738ms	23.0338μs	43.4144 KOps/s	43.0788 KOps/s	$\color{#35bf28}+0.78\%$
test_step_mdp_speed[False-False-True-True-True]	74.8010μs	50.7479μs	19.7052 KOps/s	19.5940 KOps/s	$\color{#35bf28}+0.57\%$
test_step_mdp_speed[False-False-True-True-False]	66.2010μs	32.2817μs	30.9773 KOps/s	31.0622 KOps/s	$\color{#d91a1a}-0.27\%$
test_step_mdp_speed[False-False-True-False-True]	66.8010μs	38.5807μs	25.9197 KOps/s	25.7134 KOps/s	$\color{#35bf28}+0.80\%$
test_step_mdp_speed[False-False-True-False-False]	91.5010μs	22.9683μs	43.5382 KOps/s	43.5423 KOps/s	$-0.01\%$
test_step_mdp_speed[False-False-False-True-True]	99.3010μs	52.2979μs	19.1212 KOps/s	19.0607 KOps/s	$\color{#35bf28}+0.32\%$
test_step_mdp_speed[False-False-False-True-False]	61.0010μs	33.6881μs	29.6841 KOps/s	29.4341 KOps/s	$\color{#35bf28}+0.85\%$
test_step_mdp_speed[False-False-False-False-True]	76.9010μs	40.5344μs	24.6704 KOps/s	25.0829 KOps/s	$\color{#d91a1a}-1.64\%$
test_step_mdp_speed[False-False-False-False-False]	54.1010μs	24.5026μs	40.8120 KOps/s	40.1520 KOps/s	$\color{#35bf28}+1.64\%$
test_values[generalized_advantage_estimate-True-True]	14.1824ms	13.7527ms	72.7132 Ops/s	75.9198 Ops/s	$\color{#d91a1a}-4.22\%$
test_values[vec_generalized_advantage_estimate-True-True]	51.1932ms	41.5279ms	24.0802 Ops/s	23.6649 Ops/s	$\color{#35bf28}+1.76\%$
test_values[td0_return_estimate-False-False]	0.3489ms	0.2107ms	4.7455 KOps/s	4.6037 KOps/s	$\color{#35bf28}+3.08\%$
test_values[td1_return_estimate-False-False]	13.6191ms	13.3556ms	74.8750 Ops/s	76.2334 Ops/s	$\color{#d91a1a}-1.78\%$
test_values[vec_td1_return_estimate-False-False]	48.6578ms	41.1937ms	24.2756 Ops/s	24.2390 Ops/s	$\color{#35bf28}+0.15\%$
test_values[td_lambda_return_estimate-True-False]	34.9413ms	32.1950ms	31.0607 Ops/s	31.8116 Ops/s	$\color{#d91a1a}-2.36\%$
test_values[vec_td_lambda_return_estimate-True-False]	48.0827ms	41.2092ms	24.2664 Ops/s	24.2603 Ops/s	$\color{#35bf28}+0.03\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	12.7978ms	12.1440ms	82.3449 Ops/s	85.5413 Ops/s	$\color{#d91a1a}-3.74\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	13.0717ms	3.4518ms	289.7065 Ops/s	295.9185 Ops/s	$\color{#d91a1a}-2.10\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	1.6326ms	0.4761ms	2.1003 KOps/s	2.1236 KOps/s	$\color{#d91a1a}-1.10\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	63.2175ms	58.0172ms	17.2363 Ops/s	17.8205 Ops/s	$\color{#d91a1a}-3.28\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	9.4012ms	2.8467ms	351.2795 Ops/s	351.7222 Ops/s	$\color{#d91a1a}-0.13\%$
test_dqn_speed	8.4787ms	1.8308ms	546.2090 Ops/s	506.2982 Ops/s	$\textbf{\color{#35bf28}+7.88\%}$
test_ddpg_speed	11.3800ms	2.8005ms	357.0852 Ops/s	365.9396 Ops/s	$\color{#d91a1a}-2.42\%$
test_sac_speed	12.9156ms	8.1965ms	122.0027 Ops/s	119.0959 Ops/s	$\color{#35bf28}+2.44\%$
test_redq_speed	27.5759ms	16.4213ms	60.8964 Ops/s	62.1480 Ops/s	$\color{#d91a1a}-2.01\%$
test_redq_deprec_speed	36.1251ms	13.6868ms	73.0630 Ops/s	76.8881 Ops/s	$\color{#d91a1a}-4.97\%$
test_td3_speed	11.1724ms	10.3623ms	96.5034 Ops/s	98.4437 Ops/s	$\color{#d91a1a}-1.97\%$
test_cql_speed	38.5868ms	29.0966ms	34.3683 Ops/s	38.6408 Ops/s	$\textbf{\color{#d91a1a}-11.06\%}$
test_a2c_speed	12.0773ms	5.3303ms	187.6074 Ops/s	192.5321 Ops/s	$\color{#d91a1a}-2.56\%$
test_ppo_speed	12.9251ms	5.6443ms	177.1693 Ops/s	183.6577 Ops/s	$\color{#d91a1a}-3.53\%$
test_reinforce_speed	13.0137ms	4.1407ms	241.5052 Ops/s	244.4137 Ops/s	$\color{#d91a1a}-1.19\%$
test_iql_speed	29.7261ms	21.9870ms	45.4815 Ops/s	46.8769 Ops/s	$\color{#d91a1a}-2.98\%$
test_sample_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.5917ms	2.7137ms	368.5010 Ops/s	386.2475 Ops/s	$\color{#d91a1a}-4.59\%$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	4.7501ms	2.7881ms	358.6724 Ops/s	362.2427 Ops/s	$\color{#d91a1a}-0.99\%$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.1474s	3.2153ms	311.0109 Ops/s	365.7588 Ops/s	$\textbf{\color{#d91a1a}-14.97\%}$
test_sample_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.6544ms	2.6059ms	383.7504 Ops/s	390.6295 Ops/s	$\color{#d91a1a}-1.76\%$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	4.4410ms	2.7842ms	359.1755 Ops/s	370.0844 Ops/s	$\color{#d91a1a}-2.95\%$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	5.5180ms	2.8129ms	355.5052 Ops/s	366.5683 Ops/s	$\color{#d91a1a}-3.02\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.3008ms	2.5568ms	391.1181 Ops/s	388.2603 Ops/s	$\color{#35bf28}+0.74\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	4.8286ms	2.7889ms	358.5667 Ops/s	370.3957 Ops/s	$\color{#d91a1a}-3.19\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.1420s	3.2714ms	305.6792 Ops/s	368.6591 Ops/s	$\textbf{\color{#d91a1a}-17.08\%}$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	4.8032ms	2.7503ms	363.5984 Ops/s	388.9889 Ops/s	$\textbf{\color{#d91a1a}-6.53\%}$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	4.9107ms	2.8666ms	348.8446 Ops/s	363.5244 Ops/s	$\color{#d91a1a}-4.04\%$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	4.6092ms	2.8579ms	349.9112 Ops/s	358.6840 Ops/s	$\color{#d91a1a}-2.45\%$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.0056ms	2.6194ms	381.7648 Ops/s	382.3571 Ops/s	$\color{#d91a1a}-0.15\%$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	3.9655ms	2.7661ms	361.5249 Ops/s	367.5131 Ops/s	$\color{#d91a1a}-1.63\%$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	4.5685ms	2.7914ms	358.2428 Ops/s	366.3526 Ops/s	$\color{#d91a1a}-2.21\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	0.2449s	3.3033ms	302.7318 Ops/s	390.5691 Ops/s	$\textbf{\color{#d91a1a}-22.49\%}$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	5.4188ms	2.7981ms	357.3846 Ops/s	358.4796 Ops/s	$\color{#d91a1a}-0.31\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	5.1635ms	2.8334ms	352.9348 Ops/s	354.3122 Ops/s	$\color{#d91a1a}-0.39\%$
test_populate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.2484s	31.8323ms	31.4146 Ops/s	33.3603 Ops/s	$\textbf{\color{#d91a1a}-5.83\%}$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	0.1445s	27.2246ms	36.7315 Ops/s	34.1384 Ops/s	$\textbf{\color{#35bf28}+7.60\%}$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	0.1465s	29.7499ms	33.6136 Ops/s	37.3466 Ops/s	$\textbf{\color{#d91a1a}-10.00\%}$
test_populate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1469s	27.2620ms	36.6811 Ops/s	33.7464 Ops/s	$\textbf{\color{#35bf28}+8.70\%}$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	0.1431s	29.6230ms	33.7575 Ops/s	37.4803 Ops/s	$\textbf{\color{#d91a1a}-9.93\%}$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	0.1488s	26.9764ms	37.0695 Ops/s	34.3160 Ops/s	$\textbf{\color{#35bf28}+8.02\%}$
test_populate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1489s	29.5287ms	33.8653 Ops/s	37.3565 Ops/s	$\textbf{\color{#d91a1a}-9.35\%}$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	0.1498s	29.4794ms	33.9220 Ops/s	33.4779 Ops/s	$\color{#35bf28}+1.33\%$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	0.1449s	27.0227ms	37.0059 Ops/s	36.8835 Ops/s	$\color{#35bf28}+0.33\%$

[matteobettini]

LGTM

Is this just a refactoring essentially?

[vmoens]

LGTM

Is this just a refactoring essentially?

No there's also a bug fix in the batch size and spec assignment