[CI] Fix CI

[vmoens]

Some CI fixes

• Windows does not find temp directory when closing it
• public mjkey is gone, now uploaded in pytorch S3

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 89. Improved: $\large\color{#35bf28}9$. Worsened: $\large\color{#d91a1a}5$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	75.9489ms	73.1381ms	13.6728 Ops/s	13.5815 Ops/s	$\color{#35bf28}+0.67\%$
test_sync	0.1311s	42.7426ms	23.3959 Ops/s	22.9701 Ops/s	$\color{#35bf28}+1.85\%$
test_async	0.1507s	39.1603ms	25.5361 Ops/s	25.5101 Ops/s	$\color{#35bf28}+0.10\%$
test_simple	0.6966s	0.6238s	1.6031 Ops/s	1.6033 Ops/s	$\color{#d91a1a}-0.01\%$
test_transformed	0.8733s	0.8100s	1.2346 Ops/s	1.2190 Ops/s	$\color{#35bf28}+1.28\%$
test_serial	1.8184s	1.7553s	0.5697 Ops/s	0.5666 Ops/s	$\color{#35bf28}+0.55\%$
test_parallel	1.5855s	1.5055s	0.6642 Ops/s	0.6452 Ops/s	$\color{#35bf28}+2.95\%$
test_step_mdp_speed[True-True-True-True-True]	0.3880ms	44.8188μs	22.3121 KOps/s	22.3304 KOps/s	$\color{#d91a1a}-0.08\%$
test_step_mdp_speed[True-True-True-True-False]	0.1059ms	25.4501μs	39.2925 KOps/s	39.8420 KOps/s	$\color{#d91a1a}-1.38\%$
test_step_mdp_speed[True-True-True-False-True]	65.8000μs	31.7463μs	31.4998 KOps/s	31.5223 KOps/s	$\color{#d91a1a}-0.07\%$
test_step_mdp_speed[True-True-True-False-False]	37.5000μs	17.7509μs	56.3351 KOps/s	56.7917 KOps/s	$\color{#d91a1a}-0.80\%$
test_step_mdp_speed[True-True-False-True-True]	99.5010μs	46.6916μs	21.4171 KOps/s	21.6009 KOps/s	$\color{#d91a1a}-0.85\%$
test_step_mdp_speed[True-True-False-True-False]	71.3010μs	27.2443μs	36.7049 KOps/s	37.0171 KOps/s	$\color{#d91a1a}-0.84\%$
test_step_mdp_speed[True-True-False-False-True]	84.3010μs	33.6235μs	29.7411 KOps/s	29.5930 KOps/s	$\color{#35bf28}+0.50\%$
test_step_mdp_speed[True-True-False-False-False]	85.8000μs	19.4584μs	51.3917 KOps/s	51.0228 KOps/s	$\color{#35bf28}+0.72\%$
test_step_mdp_speed[True-False-True-True-True]	72.6010μs	47.8537μs	20.8970 KOps/s	20.6854 KOps/s	$\color{#35bf28}+1.02\%$
test_step_mdp_speed[True-False-True-True-False]	92.1010μs	29.0584μs	34.4135 KOps/s	34.5583 KOps/s	$\color{#d91a1a}-0.42\%$
test_step_mdp_speed[True-False-True-False-True]	0.1178ms	33.2718μs	30.0555 KOps/s	29.7756 KOps/s	$\color{#35bf28}+0.94\%$
test_step_mdp_speed[True-False-True-False-False]	44.6000μs	19.4470μs	51.4218 KOps/s	50.9998 KOps/s	$\color{#35bf28}+0.83\%$
test_step_mdp_speed[True-False-False-True-True]	0.1255ms	49.5351μs	20.1877 KOps/s	20.1761 KOps/s	$\color{#35bf28}+0.06\%$
test_step_mdp_speed[True-False-False-True-False]	60.7000μs	30.8334μs	32.4324 KOps/s	32.5910 KOps/s	$\color{#d91a1a}-0.49\%$
test_step_mdp_speed[True-False-False-False-True]	68.7000μs	34.9918μs	28.5781 KOps/s	28.3471 KOps/s	$\color{#35bf28}+0.81\%$
test_step_mdp_speed[True-False-False-False-False]	51.3000μs	21.2130μs	47.1409 KOps/s	47.2188 KOps/s	$\color{#d91a1a}-0.16\%$
test_step_mdp_speed[False-True-True-True-True]	0.1077ms	48.0695μs	20.8032 KOps/s	20.7202 KOps/s	$\color{#35bf28}+0.40\%$
test_step_mdp_speed[False-True-True-True-False]	0.1051ms	29.3576μs	34.0628 KOps/s	34.2089 KOps/s	$\color{#d91a1a}-0.43\%$
test_step_mdp_speed[False-True-True-False-True]	61.7000μs	37.2667μs	26.8336 KOps/s	26.6503 KOps/s	$\color{#35bf28}+0.69\%$
test_step_mdp_speed[False-True-True-False-False]	4.5532ms	22.0163μs	45.4209 KOps/s	45.1069 KOps/s	$\color{#35bf28}+0.70\%$
test_step_mdp_speed[False-True-False-True-True]	0.1327ms	49.7506μs	20.1002 KOps/s	19.9407 KOps/s	$\color{#35bf28}+0.80\%$
test_step_mdp_speed[False-True-False-True-False]	55.6000μs	30.5516μs	32.7316 KOps/s	32.0589 KOps/s	$\color{#35bf28}+2.10\%$
test_step_mdp_speed[False-True-False-False-True]	0.1092ms	38.5743μs	25.9240 KOps/s	25.8264 KOps/s	$\color{#35bf28}+0.38\%$
test_step_mdp_speed[False-True-False-False-False]	0.3539ms	23.4736μs	42.6011 KOps/s	42.6599 KOps/s	$\color{#d91a1a}-0.14\%$
test_step_mdp_speed[False-False-True-True-True]	0.1246ms	51.5494μs	19.3989 KOps/s	19.3566 KOps/s	$\color{#35bf28}+0.22\%$
test_step_mdp_speed[False-False-True-True-False]	96.9010μs	32.5668μs	30.7061 KOps/s	30.6063 KOps/s	$\color{#35bf28}+0.33\%$
test_step_mdp_speed[False-False-True-False-True]	90.4000μs	39.1156μs	25.5653 KOps/s	25.8643 KOps/s	$\color{#d91a1a}-1.16\%$
test_step_mdp_speed[False-False-True-False-False]	94.8010μs	23.3364μs	42.8515 KOps/s	43.0033 KOps/s	$\color{#d91a1a}-0.35\%$
test_step_mdp_speed[False-False-False-True-True]	99.9010μs	53.4107μs	18.7228 KOps/s	18.9894 KOps/s	$\color{#d91a1a}-1.40\%$
test_step_mdp_speed[False-False-False-True-False]	55.5000μs	33.8485μs	29.5434 KOps/s	29.2431 KOps/s	$\color{#35bf28}+1.03\%$
test_step_mdp_speed[False-False-False-False-True]	73.7000μs	40.9080μs	24.4451 KOps/s	24.7956 KOps/s	$\color{#d91a1a}-1.41\%$
test_step_mdp_speed[False-False-False-False-False]	47.5010μs	24.6835μs	40.5129 KOps/s	40.5651 KOps/s	$\color{#d91a1a}-0.13\%$
test_values[generalized_advantage_estimate-True-True]	16.4524ms	13.7097ms	72.9413 Ops/s	71.9426 Ops/s	$\color{#35bf28}+1.39\%$
test_values[vec_generalized_advantage_estimate-True-True]	49.7401ms	42.5777ms	23.4865 Ops/s	23.2114 Ops/s	$\color{#35bf28}+1.19\%$
test_values[td0_return_estimate-False-False]	0.3962ms	0.1776ms	5.6300 KOps/s	982.5175 Ops/s	$\textbf{\color{#35bf28}+473.02\%}$
test_values[td1_return_estimate-False-False]	13.4929ms	13.3421ms	74.9509 Ops/s	72.6115 Ops/s	$\color{#35bf28}+3.22\%$
test_values[vec_td1_return_estimate-False-False]	56.9189ms	42.4639ms	23.5494 Ops/s	23.7862 Ops/s	$\color{#d91a1a}-1.00\%$
test_values[td_lambda_return_estimate-True-False]	33.4652ms	31.9341ms	31.3145 Ops/s	30.5545 Ops/s	$\color{#35bf28}+2.49\%$
test_values[vec_td_lambda_return_estimate-True-False]	49.5318ms	42.2322ms	23.6786 Ops/s	23.6497 Ops/s	$\color{#35bf28}+0.12\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	12.5659ms	12.2207ms	81.8285 Ops/s	80.2125 Ops/s	$\color{#35bf28}+2.01\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	9.7368ms	3.3501ms	298.4941 Ops/s	303.8043 Ops/s	$\color{#d91a1a}-1.75\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.9109ms	0.4602ms	2.1730 KOps/s	2.1170 KOps/s	$\color{#35bf28}+2.65\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	61.6551ms	57.2419ms	17.4697 Ops/s	17.8024 Ops/s	$\color{#d91a1a}-1.87\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	8.4328ms	2.7606ms	362.2432 Ops/s	367.9350 Ops/s	$\color{#d91a1a}-1.55\%$
test_dqn_speed	10.0835ms	1.8135ms	551.4327 Ops/s	499.2698 Ops/s	$\textbf{\color{#35bf28}+10.45\%}$
test_ddpg_speed	9.3756ms	2.7030ms	369.9636 Ops/s	361.9773 Ops/s	$\color{#35bf28}+2.21\%$
test_sac_speed	13.9734ms	7.9213ms	126.2412 Ops/s	123.6373 Ops/s	$\color{#35bf28}+2.11\%$
test_redq_speed	20.6652ms	15.8124ms	63.2416 Ops/s	63.0548 Ops/s	$\color{#35bf28}+0.30\%$
test_redq_deprec_speed	19.0560ms	12.3655ms	80.8703 Ops/s	78.3011 Ops/s	$\color{#35bf28}+3.28\%$
test_td3_speed	13.0052ms	9.9127ms	100.8808 Ops/s	94.4537 Ops/s	$\textbf{\color{#35bf28}+6.80\%}$
test_cql_speed	32.4838ms	25.9675ms	38.5097 Ops/s	34.1862 Ops/s	$\textbf{\color{#35bf28}+12.65\%}$
test_a2c_speed	12.1608ms	5.1433ms	194.4269 Ops/s	192.8072 Ops/s	$\color{#35bf28}+0.84\%$
test_ppo_speed	13.6060ms	5.4881ms	182.2136 Ops/s	167.9371 Ops/s	$\textbf{\color{#35bf28}+8.50\%}$
test_reinforce_speed	10.9569ms	4.0250ms	248.4503 Ops/s	244.6158 Ops/s	$\color{#35bf28}+1.57\%$
test_iql_speed	28.2627ms	21.0355ms	47.5387 Ops/s	47.4880 Ops/s	$\color{#35bf28}+0.11\%$
test_sample_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.3589ms	2.5474ms	392.5545 Ops/s	385.9835 Ops/s	$\color{#35bf28}+1.70\%$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	3.9285ms	2.7046ms	369.7393 Ops/s	371.9784 Ops/s	$\color{#d91a1a}-0.60\%$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.1608s	3.2324ms	309.3705 Ops/s	365.2185 Ops/s	$\textbf{\color{#d91a1a}-15.29\%}$
test_sample_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.4560ms	2.5726ms	388.7102 Ops/s	384.6201 Ops/s	$\color{#35bf28}+1.06\%$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	3.8684ms	2.7153ms	368.2880 Ops/s	363.0821 Ops/s	$\color{#35bf28}+1.43\%$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	4.2947ms	2.7971ms	357.5148 Ops/s	368.1035 Ops/s	$\color{#d91a1a}-2.88\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	2.8206ms	2.5241ms	396.1865 Ops/s	382.4045 Ops/s	$\color{#35bf28}+3.60\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	5.7952ms	2.8023ms	356.8436 Ops/s	367.8567 Ops/s	$\color{#d91a1a}-2.99\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	4.4096ms	2.7968ms	357.5507 Ops/s	366.6606 Ops/s	$\color{#d91a1a}-2.48\%$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.1638ms	2.5585ms	390.8493 Ops/s	384.2495 Ops/s	$\color{#35bf28}+1.72\%$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	3.6452ms	2.7434ms	364.5071 Ops/s	367.5736 Ops/s	$\color{#d91a1a}-0.83\%$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	4.4126ms	2.7818ms	359.4765 Ops/s	363.4064 Ops/s	$\color{#d91a1a}-1.08\%$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.3668ms	2.5891ms	386.2337 Ops/s	389.4290 Ops/s	$\color{#d91a1a}-0.82\%$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	3.9019ms	2.7529ms	363.2564 Ops/s	368.9293 Ops/s	$\color{#d91a1a}-1.54\%$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	3.9751ms	2.6913ms	371.5707 Ops/s	365.9699 Ops/s	$\color{#35bf28}+1.53\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	2.7880ms	2.5540ms	391.5421 Ops/s	386.6003 Ops/s	$\color{#35bf28}+1.28\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	4.6487ms	2.7699ms	361.0260 Ops/s	361.6982 Ops/s	$\color{#d91a1a}-0.19\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	4.7189ms	2.7752ms	360.3341 Ops/s	360.6862 Ops/s	$\color{#d91a1a}-0.10\%$
test_populate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.2602s	32.7512ms	30.5332 Ops/s	31.9743 Ops/s	$\color{#d91a1a}-4.51\%$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	0.1532s	27.7643ms	36.0175 Ops/s	32.7194 Ops/s	$\textbf{\color{#35bf28}+10.08\%}$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	0.1519s	30.2151ms	33.0961 Ops/s	36.0390 Ops/s	$\textbf{\color{#d91a1a}-8.17\%}$
test_populate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1514s	27.9827ms	35.7364 Ops/s	32.7004 Ops/s	$\textbf{\color{#35bf28}+9.28\%}$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	0.1533s	30.2860ms	33.0186 Ops/s	35.6449 Ops/s	$\textbf{\color{#d91a1a}-7.37\%}$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	0.1611s	27.9991ms	35.7154 Ops/s	32.2999 Ops/s	$\textbf{\color{#35bf28}+10.57\%}$
test_populate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1644s	31.0438ms	32.2125 Ops/s	35.7816 Ops/s	$\textbf{\color{#d91a1a}-9.97\%}$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	0.1543s	28.1073ms	35.5779 Ops/s	32.6570 Ops/s	$\textbf{\color{#35bf28}+8.94\%}$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	0.1584s	30.7127ms	32.5599 Ops/s	35.7814 Ops/s	$\textbf{\color{#d91a1a}-9.00\%}$

[matteobettini]

for uniformity we could do this in all libs

[matteobettini]

also this makes it an object property, but we might want it as a class property as users will have to know whats available before instantiating

[vmoens]

Yes that's right!

[vmoens]

I made it a generator in all cases such that one can do

env_generator = MyEnvClass.available_envs()

but that won't call anything until you iterate over it. In some cases it can avoid importing the lib.
The only problem with that is when you do

env_generator = MyEnvClass.available_envs()
for name in env_generator:
    print(name)
for name in env_generator:
    # never reached
    print(name)

Wdyt?

[matteobettini]

Personally I always just print it in one go, what is the need for iterating?
The purpose of the attribute is to show all the available envs.
Also this would BC breaking.

What is the problem of a normal list? I think I am missing the reason why we are changing this

[vmoens]

It is true it's bc-breaking but only mildly: you can still iterate. If the goal is to print you will soon realize you should put the result in a list. If you iterate (once) nothing changes. The issue is when you iterate twice and/or when you index (but I wonder who would index that).
By the way: generator makes more sense here to me also because indexing should be unimportant (GymEnv.available_envs[3] is meaningless)

[matteobettini]

Ok I see, if the constraint is that we do not want to import anything when callingLibrary.available_envs()it seems like something like this is the only option.

[vmoens]

Yep
This is on my mcbook:

>>> import time
>>> def import_gym():
...     t0 = time.time()
...     import gymnasium
...     print(time.time()-t0)
... 
>>> import_gym()
8.547016143798828

Now imagine we have a test run for collectors + gym. It spawns 2 workers, each having 2 envs in parallel.
So we wait at least 16 seconds just to launch the test, without counting execution. Imagine with the parametrization how fast it grows! We quickly end up with a single test taking > 5 minutes...

[matteobettini]

there might still be a few libs to update, like smac and vmas

[vmoens]

On it but it will be a separate PR

[matteobettini]

LGTM