Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Add time_dim arg in value modules #1946

Merged
merged 2 commits into from
Feb 22, 2024
Merged

[Feature] Add time_dim arg in value modules #1946

merged 2 commits into from
Feb 22, 2024

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Feb 22, 2024 •
edited
Loading

No description provided.

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Feb 22, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1946

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Merge Blocking SEVs

There is 1 active merge blocking SEVs. Please view them below:

If you must merge, use @pytorchbot merge -f.

✅ You can merge normally! (2 Unrelated Failures)

As of commit 684ab56 with merge base bb44067 (image):

FLAKY - The following job failed but was likely due to flakiness present on trunk:

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 22, 2024
@vmoens vmoens added the enhancement New feature or request label Feb 22, 2024
@vmoens vmoens linked an issue Feb 22, 2024 that may be closed by this pull request
[Feature Request] Expose time_dim parameter on GAE and other advantage estimators #1915
Closed
@github-actions GitHub Actions
Copy link

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 89. Improved: $\large\color{#35bf28}4$. Worsened: $\large\color{#d91a1a}2$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 62.2345ms 61.8212ms 16.1757 Ops/s 15.4937 Ops/s $\color{#35bf28}+4.40\%$
test_sync 37.2181ms 33.0315ms 30.2741 Ops/s 29.8178 Ops/s $\color{#35bf28}+1.53\%$
test_async 66.2269ms 32.1695ms 31.0854 Ops/s 32.5169 Ops/s $\color{#d91a1a}-4.40\%$
test_simple 0.4888s 0.4368s 2.2894 Ops/s 2.3329 Ops/s $\color{#d91a1a}-1.86\%$
test_transformed 0.6366s 0.5877s 1.7015 Ops/s 1.7167 Ops/s $\color{#d91a1a}-0.88\%$
test_serial 1.4926s 1.4421s 0.6934 Ops/s 0.7070 Ops/s $\color{#d91a1a}-1.91\%$
test_parallel 1.4288s 1.3698s 0.7300 Ops/s 0.7353 Ops/s $\color{#d91a1a}-0.72\%$
test_step_mdp_speed[True-True-True-True-True] 0.2586ms 21.0113μs 47.5935 KOps/s 47.2238 KOps/s $\color{#35bf28}+0.78\%$
test_step_mdp_speed[True-True-True-True-False] 32.4610μs 12.6955μs 78.7683 KOps/s 77.7168 KOps/s $\color{#35bf28}+1.35\%$
test_step_mdp_speed[True-True-True-False-True] 45.4450μs 12.2970μs 81.3204 KOps/s 81.2365 KOps/s $\color{#35bf28}+0.10\%$
test_step_mdp_speed[True-True-True-False-False] 53.8410μs 7.4207μs 134.7573 KOps/s 132.9072 KOps/s $\color{#35bf28}+1.39\%$
test_step_mdp_speed[True-True-False-True-True] 60.9340μs 22.4517μs 44.5400 KOps/s 44.2841 KOps/s $\color{#35bf28}+0.58\%$
test_step_mdp_speed[True-True-False-True-False] 60.4740μs 14.1260μs 70.7917 KOps/s 70.1686 KOps/s $\color{#35bf28}+0.89\%$
test_step_mdp_speed[True-True-False-False-True] 33.2020μs 13.5468μs 73.8184 KOps/s 73.5968 KOps/s $\color{#35bf28}+0.30\%$
test_step_mdp_speed[True-True-False-False-False] 48.8510μs 8.7144μs 114.7528 KOps/s 113.3356 KOps/s $\color{#35bf28}+1.25\%$
test_step_mdp_speed[True-False-True-True-True] 59.3320μs 23.8262μs 41.9706 KOps/s 41.8299 KOps/s $\color{#35bf28}+0.34\%$
test_step_mdp_speed[True-False-True-True-False] 58.4900μs 15.5279μs 64.4002 KOps/s 63.6059 KOps/s $\color{#35bf28}+1.25\%$
test_step_mdp_speed[True-False-True-False-True] 65.8930μs 13.4572μs 74.3099 KOps/s 73.3447 KOps/s $\color{#35bf28}+1.32\%$
test_step_mdp_speed[True-False-True-False-False] 38.2020μs 8.6862μs 115.1248 KOps/s 113.2090 KOps/s $\color{#35bf28}+1.69\%$
test_step_mdp_speed[True-False-False-True-True] 49.4930μs 24.8659μs 40.2158 KOps/s 39.7934 KOps/s $\color{#35bf28}+1.06\%$
test_step_mdp_speed[True-False-False-True-False] 61.4760μs 16.5486μs 60.4282 KOps/s 59.6410 KOps/s $\color{#35bf28}+1.32\%$
test_step_mdp_speed[True-False-False-False-True] 59.0210μs 14.7859μs 67.6322 KOps/s 67.4604 KOps/s $\color{#35bf28}+0.25\%$
test_step_mdp_speed[True-False-False-False-False] 30.9680μs 9.9377μs 100.6274 KOps/s 100.1613 KOps/s $\color{#35bf28}+0.47\%$
test_step_mdp_speed[False-True-True-True-True] 69.9120μs 23.7845μs 42.0443 KOps/s 41.8165 KOps/s $\color{#35bf28}+0.54\%$
test_step_mdp_speed[False-True-True-True-False] 40.1460μs 15.7923μs 63.3221 KOps/s 63.9539 KOps/s $\color{#d91a1a}-0.99\%$
test_step_mdp_speed[False-True-True-False-True] 60.1730μs 15.8262μs 63.1862 KOps/s 62.9476 KOps/s $\color{#35bf28}+0.38\%$
test_step_mdp_speed[False-True-True-False-False] 51.1960μs 9.9037μs 100.9722 KOps/s 97.7731 KOps/s $\color{#35bf28}+3.27\%$
test_step_mdp_speed[False-True-False-True-True] 41.8490μs 25.0973μs 39.8450 KOps/s 39.0125 KOps/s $\color{#35bf28}+2.13\%$
test_step_mdp_speed[False-True-False-True-False] 64.4210μs 16.5515μs 60.4175 KOps/s 59.3536 KOps/s $\color{#35bf28}+1.79\%$
test_step_mdp_speed[False-True-False-False-True] 74.0800μs 16.9310μs 59.0631 KOps/s 59.7104 KOps/s $\color{#d91a1a}-1.08\%$
test_step_mdp_speed[False-True-False-False-False] 45.0640μs 11.0458μs 90.5324 KOps/s 89.0499 KOps/s $\color{#35bf28}+1.66\%$
test_step_mdp_speed[False-False-True-True-True] 69.7910μs 26.2613μs 38.0788 KOps/s 37.8665 KOps/s $\color{#35bf28}+0.56\%$
test_step_mdp_speed[False-False-True-True-False] 62.4670μs 17.9854μs 55.6008 KOps/s 55.7326 KOps/s $\color{#d91a1a}-0.24\%$
test_step_mdp_speed[False-False-True-False-True] 60.2230μs 16.9638μs 58.9489 KOps/s 58.9376 KOps/s $\color{#35bf28}+0.02\%$
test_step_mdp_speed[False-False-True-False-False] 34.4240μs 11.1803μs 89.4431 KOps/s 88.8782 KOps/s $\color{#35bf28}+0.64\%$
test_step_mdp_speed[False-False-False-True-True] 73.5580μs 26.9628μs 37.0882 KOps/s 36.6357 KOps/s $\color{#35bf28}+1.24\%$
test_step_mdp_speed[False-False-False-True-False] 46.2070μs 18.9775μs 52.6940 KOps/s 52.5311 KOps/s $\color{#35bf28}+0.31\%$
test_step_mdp_speed[False-False-False-False-True] 63.4990μs 17.9487μs 55.7143 KOps/s 55.8760 KOps/s $\color{#d91a1a}-0.29\%$
test_step_mdp_speed[False-False-False-False-False] 59.1910μs 12.2605μs 81.5628 KOps/s 82.1072 KOps/s $\color{#d91a1a}-0.66\%$
test_values[generalized_advantage_estimate-True-True] 10.6045ms 9.2172ms 108.4934 Ops/s 110.5131 Ops/s $\color{#d91a1a}-1.83\%$
test_values[vec_generalized_advantage_estimate-True-True] 49.1229ms 34.6189ms 28.8860 Ops/s 28.4710 Ops/s $\color{#35bf28}+1.46\%$
test_values[td0_return_estimate-False-False] 0.2569ms 0.1687ms 5.9281 KOps/s 5.6973 KOps/s $\color{#35bf28}+4.05\%$
test_values[td1_return_estimate-False-False] 23.1325ms 22.8333ms 43.7957 Ops/s 44.2053 Ops/s $\color{#d91a1a}-0.93\%$
test_values[vec_td1_return_estimate-False-False] 35.1983ms 33.4726ms 29.8751 Ops/s 28.4687 Ops/s $\color{#35bf28}+4.94\%$
test_values[td_lambda_return_estimate-True-False] 36.0502ms 32.9785ms 30.3228 Ops/s 30.5337 Ops/s $\color{#d91a1a}-0.69\%$
test_values[vec_td_lambda_return_estimate-True-False] 34.6011ms 33.4432ms 29.9015 Ops/s 28.3929 Ops/s $\textbf{\color{#35bf28}+5.31\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512] 11.0788ms 8.1485ms 122.7217 Ops/s 126.1782 Ops/s $\color{#d91a1a}-2.74\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.2945ms 1.9703ms 507.5402 Ops/s 519.2402 Ops/s $\color{#d91a1a}-2.25\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.4272ms 0.3431ms 2.9145 KOps/s 2.8718 KOps/s $\color{#35bf28}+1.49\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 41.4043ms 39.5111ms 25.3094 Ops/s 22.5369 Ops/s $\textbf{\color{#35bf28}+12.30\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 3.6690ms 3.0161ms 331.5549 Ops/s 331.0121 Ops/s $\color{#35bf28}+0.16\%$
test_dqn_speed 68.5430ms 1.5011ms 666.1688 Ops/s 709.8838 Ops/s $\textbf{\color{#d91a1a}-6.16\%}$
test_ddpg_speed 3.4058ms 2.7936ms 357.9626 Ops/s 356.0772 Ops/s $\color{#35bf28}+0.53\%$
test_sac_speed 8.8163ms 8.3960ms 119.1040 Ops/s 119.3481 Ops/s $\color{#d91a1a}-0.20\%$
test_redq_speed 14.8528ms 13.1903ms 75.8133 Ops/s 75.3962 Ops/s $\color{#35bf28}+0.55\%$
test_redq_deprec_speed 15.7915ms 13.4877ms 74.1415 Ops/s 74.1877 Ops/s $\color{#d91a1a}-0.06\%$
test_td3_speed 8.8189ms 8.4252ms 118.6913 Ops/s 119.2953 Ops/s $\color{#d91a1a}-0.51\%$
test_cql_speed 38.0162ms 36.8843ms 27.1118 Ops/s 27.2457 Ops/s $\color{#d91a1a}-0.49\%$
test_a2c_speed 8.3910ms 7.4599ms 134.0509 Ops/s 135.6804 Ops/s $\color{#d91a1a}-1.20\%$
test_ppo_speed 8.6507ms 7.7980ms 128.2381 Ops/s 127.3257 Ops/s $\color{#35bf28}+0.72\%$
test_reinforce_speed 8.8712ms 6.6769ms 149.7690 Ops/s 150.6175 Ops/s $\color{#d91a1a}-0.56\%$
test_iql_speed 33.9630ms 32.7837ms 30.5030 Ops/s 30.4383 Ops/s $\color{#35bf28}+0.21\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 4.2882ms 2.8034ms 356.7085 Ops/s 347.8889 Ops/s $\color{#35bf28}+2.54\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7943ms 0.5203ms 1.9220 KOps/s 1.9308 KOps/s $\color{#d91a1a}-0.45\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6213ms 0.4868ms 2.0542 KOps/s 2.0552 KOps/s $\color{#d91a1a}-0.05\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 4.1324ms 2.8097ms 355.9115 Ops/s 340.5712 Ops/s $\color{#35bf28}+4.50\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.8013ms 0.5208ms 1.9201 KOps/s 1.9725 KOps/s $\color{#d91a1a}-2.65\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.8308ms 0.4866ms 2.0552 KOps/s 2.0831 KOps/s $\color{#d91a1a}-1.34\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.7089ms 2.8953ms 345.3911 Ops/s 338.5854 Ops/s $\color{#35bf28}+2.01\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.9430ms 0.6337ms 1.5779 KOps/s 1.5735 KOps/s $\color{#35bf28}+0.28\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.9405ms 0.6123ms 1.6333 KOps/s 1.6609 KOps/s $\color{#d91a1a}-1.66\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 2.9715ms 2.7629ms 361.9438 Ops/s 359.1962 Ops/s $\color{#35bf28}+0.76\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.6292ms 0.5143ms 1.9443 KOps/s 1.9181 KOps/s $\color{#35bf28}+1.36\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7810ms 0.5006ms 1.9976 KOps/s 2.0506 KOps/s $\color{#d91a1a}-2.59\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.1733ms 2.8313ms 353.1952 Ops/s 353.3197 Ops/s $\color{#d91a1a}-0.04\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.6215ms 0.5093ms 1.9636 KOps/s 1.9537 KOps/s $\color{#35bf28}+0.51\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7526ms 0.4871ms 2.0531 KOps/s 2.0439 KOps/s $\color{#35bf28}+0.45\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.2301ms 2.9330ms 340.9475 Ops/s 331.6788 Ops/s $\color{#35bf28}+2.79\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.7436ms 0.6337ms 1.5782 KOps/s 1.5721 KOps/s $\color{#35bf28}+0.38\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8402ms 0.6048ms 1.6535 KOps/s 1.6578 KOps/s $\color{#d91a1a}-0.26\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 98.5732ms 7.6335ms 131.0019 Ops/s 122.9425 Ops/s $\textbf{\color{#35bf28}+6.56\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 98.6789ms 14.9657ms 66.8195 Ops/s 75.5926 Ops/s $\textbf{\color{#d91a1a}-11.61\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 4.7825ms 2.5686ms 389.3224 Ops/s 396.5121 Ops/s $\color{#d91a1a}-1.81\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 91.1303ms 7.5904ms 131.7453 Ops/s 107.0496 Ops/s $\textbf{\color{#35bf28}+23.07\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 15.4370ms 13.2642ms 75.3910 Ops/s 75.6955 Ops/s $\color{#d91a1a}-0.40\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 4.8429ms 2.6131ms 382.6930 Ops/s 397.0391 Ops/s $\color{#d91a1a}-3.61\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 91.2852ms 9.5389ms 104.8343 Ops/s 107.1175 Ops/s $\color{#d91a1a}-2.13\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 15.6068ms 13.5405ms 73.8523 Ops/s 73.8002 Ops/s $\color{#35bf28}+0.07\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 4.0148ms 2.8203ms 354.5730 Ops/s 356.7827 Ops/s $\color{#d91a1a}-0.62\%$

@github-actions GitHub Actions
Copy link

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 92. Improved: $\large\color{#35bf28}14$. Worsened: $\large\color{#d91a1a}1$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1136s 0.1110s 9.0080 Ops/s 8.3455 Ops/s $\textbf{\color{#35bf28}+7.94\%}$
test_sync 95.3361ms 95.1265ms 10.5123 Ops/s 10.3128 Ops/s $\color{#35bf28}+1.94\%$
test_async 0.1807s 91.4626ms 10.9334 Ops/s 10.9381 Ops/s $\color{#d91a1a}-0.04\%$
test_single_pixels 0.1241s 0.1240s 8.0672 Ops/s 7.4200 Ops/s $\textbf{\color{#35bf28}+8.72\%}$
test_sync_pixels 81.8287ms 79.6328ms 12.5576 Ops/s 13.1969 Ops/s $\color{#d91a1a}-4.84\%$
test_async_pixels 0.1492s 73.6285ms 13.5817 Ops/s 13.7334 Ops/s $\color{#d91a1a}-1.10\%$
test_simple 0.8613s 0.8065s 1.2400 Ops/s 1.2085 Ops/s $\color{#35bf28}+2.61\%$
test_transformed 1.0832s 1.0257s 0.9749 Ops/s 0.9504 Ops/s $\color{#35bf28}+2.59\%$
test_serial 2.4395s 2.3830s 0.4196 Ops/s 0.4120 Ops/s $\color{#35bf28}+1.85\%$
test_parallel 2.1190s 2.0501s 0.4878 Ops/s 0.4863 Ops/s $\color{#35bf28}+0.30\%$
test_step_mdp_speed[True-True-True-True-True] 0.1022ms 32.7645μs 30.5209 KOps/s 28.9345 KOps/s $\textbf{\color{#35bf28}+5.48\%}$
test_step_mdp_speed[True-True-True-True-False] 50.0500μs 19.1615μs 52.1879 KOps/s 49.7510 KOps/s $\color{#35bf28}+4.90\%$
test_step_mdp_speed[True-True-True-False-True] 45.6200μs 18.2836μs 54.6938 KOps/s 52.1113 KOps/s $\color{#35bf28}+4.96\%$
test_step_mdp_speed[True-True-True-False-False] 76.2810μs 11.0443μs 90.5446 KOps/s 87.1447 KOps/s $\color{#35bf28}+3.90\%$
test_step_mdp_speed[True-True-False-True-True] 58.4120μs 34.0534μs 29.3656 KOps/s 27.5709 KOps/s $\textbf{\color{#35bf28}+6.51\%}$
test_step_mdp_speed[True-True-False-True-False] 44.0510μs 21.5245μs 46.4587 KOps/s 45.6297 KOps/s $\color{#35bf28}+1.82\%$
test_step_mdp_speed[True-True-False-False-True] 47.5610μs 20.0967μs 49.7594 KOps/s 46.9890 KOps/s $\textbf{\color{#35bf28}+5.90\%}$
test_step_mdp_speed[True-True-False-False-False] 47.2310μs 12.8802μs 77.6388 KOps/s 75.0465 KOps/s $\color{#35bf28}+3.45\%$
test_step_mdp_speed[True-False-True-True-True] 58.8100μs 36.1293μs 27.6783 KOps/s 25.7160 KOps/s $\textbf{\color{#35bf28}+7.63\%}$
test_step_mdp_speed[True-False-True-True-False] 57.7410μs 23.2586μs 42.9949 KOps/s 42.2945 KOps/s $\color{#35bf28}+1.66\%$
test_step_mdp_speed[True-False-True-False-True] 43.7200μs 20.1136μs 49.7175 KOps/s 46.7850 KOps/s $\textbf{\color{#35bf28}+6.27\%}$
test_step_mdp_speed[True-False-True-False-False] 28.4300μs 13.0284μs 76.7552 KOps/s 75.3757 KOps/s $\color{#35bf28}+1.83\%$
test_step_mdp_speed[True-False-False-True-True] 70.8500μs 37.9794μs 26.3301 KOps/s 25.3736 KOps/s $\color{#35bf28}+3.77\%$
test_step_mdp_speed[True-False-False-True-False] 60.8210μs 25.3125μs 39.5062 KOps/s 38.4427 KOps/s $\color{#35bf28}+2.77\%$
test_step_mdp_speed[True-False-False-False-True] 48.6900μs 22.0165μs 45.4204 KOps/s 42.9972 KOps/s $\textbf{\color{#35bf28}+5.64\%}$
test_step_mdp_speed[True-False-False-False-False] 28.8200μs 14.9778μs 66.7653 KOps/s 66.5075 KOps/s $\color{#35bf28}+0.39\%$
test_step_mdp_speed[False-True-True-True-True] 67.6220μs 36.2603μs 27.5784 KOps/s 26.2408 KOps/s $\textbf{\color{#35bf28}+5.10\%}$
test_step_mdp_speed[False-True-True-True-False] 58.9510μs 23.8265μs 41.9700 KOps/s 41.4194 KOps/s $\color{#35bf28}+1.33\%$
test_step_mdp_speed[False-True-True-False-True] 61.4910μs 24.7289μs 40.4385 KOps/s 38.9431 KOps/s $\color{#35bf28}+3.84\%$
test_step_mdp_speed[False-True-True-False-False] 37.1710μs 14.9957μs 66.6859 KOps/s 65.6388 KOps/s $\color{#35bf28}+1.60\%$
test_step_mdp_speed[False-True-False-True-True] 76.8400μs 38.0087μs 26.3097 KOps/s 24.9439 KOps/s $\textbf{\color{#35bf28}+5.48\%}$
test_step_mdp_speed[False-True-False-True-False] 48.9410μs 25.3954μs 39.3773 KOps/s 37.8856 KOps/s $\color{#35bf28}+3.94\%$
test_step_mdp_speed[False-True-False-False-True] 53.2100μs 25.9967μs 38.4665 KOps/s 37.4023 KOps/s $\color{#35bf28}+2.85\%$
test_step_mdp_speed[False-True-False-False-False] 37.1800μs 16.6616μs 60.0183 KOps/s 59.1776 KOps/s $\color{#35bf28}+1.42\%$
test_step_mdp_speed[False-False-True-True-True] 85.2410μs 40.2873μs 24.8217 KOps/s 24.0017 KOps/s $\color{#35bf28}+3.42\%$
test_step_mdp_speed[False-False-True-True-False] 42.6300μs 27.6123μs 36.2158 KOps/s 35.1636 KOps/s $\color{#35bf28}+2.99\%$
test_step_mdp_speed[False-False-True-False-True] 56.0400μs 25.9141μs 38.5891 KOps/s 37.4363 KOps/s $\color{#35bf28}+3.08\%$
test_step_mdp_speed[False-False-True-False-False] 35.3510μs 16.7347μs 59.7561 KOps/s 58.1491 KOps/s $\color{#35bf28}+2.76\%$
test_step_mdp_speed[False-False-False-True-True] 73.7610μs 41.7953μs 23.9261 KOps/s 23.2126 KOps/s $\color{#35bf28}+3.07\%$
test_step_mdp_speed[False-False-False-True-False] 58.2110μs 29.5692μs 33.8190 KOps/s 33.7095 KOps/s $\color{#35bf28}+0.32\%$
test_step_mdp_speed[False-False-False-False-True] 51.9810μs 28.0831μs 35.6086 KOps/s 35.3099 KOps/s $\color{#35bf28}+0.85\%$
test_step_mdp_speed[False-False-False-False-False] 49.5910μs 18.7615μs 53.3008 KOps/s 52.6954 KOps/s $\color{#35bf28}+1.15\%$
test_values[generalized_advantage_estimate-True-True] 24.4779ms 23.4103ms 42.7162 Ops/s 43.6806 Ops/s $\color{#d91a1a}-2.21\%$
test_values[vec_generalized_advantage_estimate-True-True] 84.3587ms 3.2184ms 310.7131 Ops/s 309.8781 Ops/s $\color{#35bf28}+0.27\%$
test_values[td0_return_estimate-False-False] 91.2410μs 55.6950μs 17.9549 KOps/s 17.3550 KOps/s $\color{#35bf28}+3.46\%$
test_values[td1_return_estimate-False-False] 51.8691ms 51.1600ms 19.5465 Ops/s 19.9734 Ops/s $\color{#d91a1a}-2.14\%$
test_values[vec_td1_return_estimate-False-False] 2.0339ms 1.7270ms 579.0384 Ops/s 577.2286 Ops/s $\color{#35bf28}+0.31\%$
test_values[td_lambda_return_estimate-True-False] 82.7791ms 80.5072ms 12.4212 Ops/s 12.4601 Ops/s $\color{#d91a1a}-0.31\%$
test_values[vec_td_lambda_return_estimate-True-False] 4.2694ms 1.7624ms 567.3980 Ops/s 566.9203 Ops/s $\color{#35bf28}+0.08\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 23.0256ms 21.3427ms 46.8544 Ops/s 45.8072 Ops/s $\color{#35bf28}+2.29\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 0.8211ms 0.6579ms 1.5201 KOps/s 1.4746 KOps/s $\color{#35bf28}+3.08\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.6622ms 0.6093ms 1.6411 KOps/s 1.6188 KOps/s $\color{#35bf28}+1.38\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.4435ms 1.4123ms 708.0784 Ops/s 697.9215 Ops/s $\color{#35bf28}+1.46\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.9074ms 0.6314ms 1.5838 KOps/s 1.5652 KOps/s $\color{#35bf28}+1.19\%$
test_dqn_speed 8.1068ms 1.5019ms 665.8085 Ops/s 687.8798 Ops/s $\color{#d91a1a}-3.21\%$
test_ddpg_speed 3.2047ms 2.7909ms 358.3015 Ops/s 355.2186 Ops/s $\color{#35bf28}+0.87\%$
test_sac_speed 8.4017ms 8.0347ms 124.4596 Ops/s 123.7339 Ops/s $\color{#35bf28}+0.59\%$
test_redq_speed 11.6656ms 10.3105ms 96.9889 Ops/s 96.2658 Ops/s $\color{#35bf28}+0.75\%$
test_redq_deprec_speed 11.6489ms 11.0374ms 90.6007 Ops/s 88.5665 Ops/s $\color{#35bf28}+2.30\%$
test_td3_speed 16.0408ms 8.1182ms 123.1802 Ops/s 123.3602 Ops/s $\color{#d91a1a}-0.15\%$
test_cql_speed 26.0734ms 25.3095ms 39.5109 Ops/s 36.3291 Ops/s $\textbf{\color{#35bf28}+8.76\%}$
test_a2c_speed 5.8505ms 5.4696ms 182.8297 Ops/s 176.4440 Ops/s $\color{#35bf28}+3.62\%$
test_ppo_speed 6.9128ms 5.7763ms 173.1202 Ops/s 166.9881 Ops/s $\color{#35bf28}+3.67\%$
test_reinforce_speed 4.7867ms 4.4915ms 222.6431 Ops/s 217.0009 Ops/s $\color{#35bf28}+2.60\%$
test_iql_speed 20.2178ms 19.5206ms 51.2278 Ops/s 50.2213 Ops/s $\color{#35bf28}+2.00\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.8847ms 3.6566ms 273.4777 Ops/s 268.7329 Ops/s $\color{#35bf28}+1.77\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7279ms 0.5759ms 1.7365 KOps/s 1.7327 KOps/s $\color{#35bf28}+0.22\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7122ms 0.5508ms 1.8156 KOps/s 1.8077 KOps/s $\color{#35bf28}+0.44\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.8505ms 3.6550ms 273.5985 Ops/s 267.6298 Ops/s $\color{#35bf28}+2.23\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.7061ms 0.5626ms 1.7776 KOps/s 1.7696 KOps/s $\color{#35bf28}+0.45\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6914ms 0.5432ms 1.8409 KOps/s 1.8277 KOps/s $\color{#35bf28}+0.72\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.9328ms 3.7736ms 264.9978 Ops/s 259.7684 Ops/s $\color{#35bf28}+2.01\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.8506ms 0.6953ms 1.4382 KOps/s 1.2694 KOps/s $\textbf{\color{#35bf28}+13.29\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8078ms 0.6653ms 1.5030 KOps/s 1.4775 KOps/s $\color{#35bf28}+1.73\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.8409ms 3.6585ms 273.3379 Ops/s 269.7457 Ops/s $\color{#35bf28}+1.33\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7716ms 0.5706ms 1.7524 KOps/s 1.7402 KOps/s $\color{#35bf28}+0.71\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7159ms 0.5491ms 1.8213 KOps/s 1.8049 KOps/s $\color{#35bf28}+0.91\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.9412ms 3.6626ms 273.0273 Ops/s 269.3144 Ops/s $\color{#35bf28}+1.38\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 97.3661ms 0.6697ms 1.4932 KOps/s 1.7625 KOps/s $\textbf{\color{#d91a1a}-15.27\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7006ms 0.5405ms 1.8501 KOps/s 1.5496 KOps/s $\textbf{\color{#35bf28}+19.39\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.9907ms 3.7879ms 263.9997 Ops/s 260.9871 Ops/s $\color{#35bf28}+1.15\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.8579ms 0.6991ms 1.4303 KOps/s 1.4379 KOps/s $\color{#d91a1a}-0.53\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8354ms 0.6766ms 1.4781 KOps/s 1.4902 KOps/s $\color{#d91a1a}-0.81\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1017s 9.2922ms 107.6175 Ops/s 89.9909 Ops/s $\textbf{\color{#35bf28}+19.59\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 17.7593ms 15.5740ms 64.2095 Ops/s 61.6999 Ops/s $\color{#35bf28}+4.07\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 7.7927ms 3.0337ms 329.6269 Ops/s 321.7578 Ops/s $\color{#35bf28}+2.45\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1009s 11.0353ms 90.6180 Ops/s 90.4464 Ops/s $\color{#35bf28}+0.19\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 17.6970ms 15.5051ms 64.4950 Ops/s 61.4989 Ops/s $\color{#35bf28}+4.87\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 8.1743ms 3.0232ms 330.7761 Ops/s 323.3411 Ops/s $\color{#35bf28}+2.30\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1041s 11.3932ms 87.7717 Ops/s 86.6134 Ops/s $\color{#35bf28}+1.34\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 18.2729ms 16.0090ms 62.4647 Ops/s 60.4226 Ops/s $\color{#35bf28}+3.38\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 8.4458ms 3.3576ms 297.8352 Ops/s 293.6412 Ops/s $\color{#35bf28}+1.43\%$

@vmoens vmoens merged commit b28bbfe into main Feb 22, 2024
66 of 68 checks passed
@vmoens vmoens deleted the add-dims-values branch February 27, 2024 00:47
@wbinventor
Copy link

It looks like this didn’t make it in the new 0.3.1 release. Will this be included in the following release?

@vmoens
Copy link
Contributor Author

vmoens commented Mar 5, 2024

Yes since it's a new feature and not a bug fix or a speed improvement it's to be included in 0.4.
Hope that makes sense!

@wbinventor
Copy link

That makes sense! Just wondered about the release plan since the new parameter is rendered in the online documentation even though it’s not yet available.

@vmoens
Copy link
Contributor Author

vmoens commented Mar 5, 2024

You're pointing at a longstanding issue here which is that the doc of the main branch is automatically synced and we don't have yet a separate doc for the release and for main.
We should fix this! It's one of these "one maintainer cannot do it all" sort of things :/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature Request] Expose time_dim parameter on GAE and other advantage estimators
3 participants
@vmoens @wbinventor @facebook-github-bot