Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] Regroup tests #1791

Merged
merged 6 commits into from
Jan 12, 2024
Merged

[CI] Regroup tests #1791

merged 6 commits into from
Jan 12, 2024

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Jan 12, 2024 •
edited
Loading

Having tests spread all over the place is a pain, let's put them together as much as we can

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jan 12, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1791

Note: Links to docs will display an error until the docs builds have been completed.

❌ 36 New Failures, 1 Pending, 5 Unrelated Failures

As of commit 9d4d52c with merge base ac39c76 (image):

NEW FAILURES - The following jobs have failed:

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 12, 2024
@vmoens vmoens added the CI Has to do with CI setup (e.g. wheels & builds, tests...) label Jan 12, 2024
@vmoens vmoens added the Data Data-related PR, will launch data-related jobs label Jan 12, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 12, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 89. Improved: $\large\color{#35bf28}2$. Worsened: $\large\color{#d91a1a}2$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 62.9771ms 62.5539ms 15.9862 Ops/s 15.7243 Ops/s $\color{#35bf28}+1.67\%$
test_sync 35.1663ms 34.4799ms 29.0024 Ops/s 27.4079 Ops/s $\textbf{\color{#35bf28}+5.82\%}$
test_async 84.3983ms 34.8587ms 28.6872 Ops/s 29.8157 Ops/s $\color{#d91a1a}-3.78\%$
test_simple 0.4932s 0.4354s 2.2967 Ops/s 2.2464 Ops/s $\color{#35bf28}+2.24\%$
test_transformed 0.6566s 0.5986s 1.6706 Ops/s 1.6122 Ops/s $\color{#35bf28}+3.62\%$
test_serial 1.3996s 1.3429s 0.7447 Ops/s 0.7273 Ops/s $\color{#35bf28}+2.39\%$
test_parallel 1.4386s 1.3830s 0.7231 Ops/s 0.7306 Ops/s $\color{#d91a1a}-1.03\%$
test_step_mdp_speed[True-True-True-True-True] 0.1455ms 21.4936μs 46.5254 KOps/s 45.7711 KOps/s $\color{#35bf28}+1.65\%$
test_step_mdp_speed[True-True-True-True-False] 36.7680μs 13.2000μs 75.7576 KOps/s 75.1348 KOps/s $\color{#35bf28}+0.83\%$
test_step_mdp_speed[True-True-True-False-True] 38.7820μs 12.6403μs 79.1117 KOps/s 78.5068 KOps/s $\color{#35bf28}+0.77\%$
test_step_mdp_speed[True-True-True-False-False] 28.4220μs 7.7528μs 128.9863 KOps/s 128.3076 KOps/s $\color{#35bf28}+0.53\%$
test_step_mdp_speed[True-True-False-True-True] 48.6610μs 22.9173μs 43.6351 KOps/s 43.3821 KOps/s $\color{#35bf28}+0.58\%$
test_step_mdp_speed[True-True-False-True-False] 46.3960μs 14.7967μs 67.5826 KOps/s 67.8704 KOps/s $\color{#d91a1a}-0.42\%$
test_step_mdp_speed[True-True-False-False-True] 35.2050μs 13.9700μs 71.5818 KOps/s 71.0445 KOps/s $\color{#35bf28}+0.76\%$
test_step_mdp_speed[True-True-False-False-False] 37.7710μs 8.9942μs 111.1827 KOps/s 110.8484 KOps/s $\color{#35bf28}+0.30\%$
test_step_mdp_speed[True-False-True-True-True] 57.9190μs 24.2218μs 41.2851 KOps/s 41.5670 KOps/s $\color{#d91a1a}-0.68\%$
test_step_mdp_speed[True-False-True-True-False] 44.6730μs 15.8320μs 63.1631 KOps/s 62.4458 KOps/s $\color{#35bf28}+1.15\%$
test_step_mdp_speed[True-False-True-False-True] 48.7520μs 14.0246μs 71.3031 KOps/s 71.6581 KOps/s $\color{#d91a1a}-0.50\%$
test_step_mdp_speed[True-False-True-False-False] 32.1100μs 9.0350μs 110.6805 KOps/s 108.9781 KOps/s $\color{#35bf28}+1.56\%$
test_step_mdp_speed[True-False-False-True-True] 80.3400μs 25.5187μs 39.1869 KOps/s 38.7228 KOps/s $\color{#35bf28}+1.20\%$
test_step_mdp_speed[True-False-False-True-False] 71.9440μs 17.0756μs 58.5633 KOps/s 58.2065 KOps/s $\color{#35bf28}+0.61\%$
test_step_mdp_speed[True-False-False-False-True] 44.2320μs 15.1446μs 66.0300 KOps/s 65.3968 KOps/s $\color{#35bf28}+0.97\%$
test_step_mdp_speed[True-False-False-False-False] 42.7800μs 10.0912μs 99.0962 KOps/s 98.9342 KOps/s $\color{#35bf28}+0.16\%$
test_step_mdp_speed[False-True-True-True-True] 91.9020μs 25.0362μs 39.9421 KOps/s 40.7061 KOps/s $\color{#d91a1a}-1.88\%$
test_step_mdp_speed[False-True-True-True-False] 46.2870μs 15.8584μs 63.0581 KOps/s 63.2673 KOps/s $\color{#d91a1a}-0.33\%$
test_step_mdp_speed[False-True-True-False-True] 74.5290μs 16.4680μs 60.7237 KOps/s 60.9007 KOps/s $\color{#d91a1a}-0.29\%$
test_step_mdp_speed[False-True-True-False-False] 31.4490μs 10.2449μs 97.6092 KOps/s 98.0271 KOps/s $\color{#d91a1a}-0.43\%$
test_step_mdp_speed[False-True-False-True-True] 58.4600μs 25.4650μs 39.2695 KOps/s 39.1270 KOps/s $\color{#35bf28}+0.36\%$
test_step_mdp_speed[False-True-False-True-False] 0.2135ms 17.5671μs 56.9246 KOps/s 58.7519 KOps/s $\color{#d91a1a}-3.11\%$
test_step_mdp_speed[False-True-False-False-True] 41.9780μs 17.4608μs 57.2712 KOps/s 57.3444 KOps/s $\color{#d91a1a}-0.13\%$
test_step_mdp_speed[False-True-False-False-False] 31.2980μs 11.4113μs 87.6322 KOps/s 86.9661 KOps/s $\color{#35bf28}+0.77\%$
test_step_mdp_speed[False-False-True-True-True] 71.8840μs 26.7096μs 37.4398 KOps/s 37.0180 KOps/s $\color{#35bf28}+1.14\%$
test_step_mdp_speed[False-False-True-True-False] 46.4370μs 18.2874μs 54.6824 KOps/s 54.0586 KOps/s $\color{#35bf28}+1.15\%$
test_step_mdp_speed[False-False-True-False-True] 43.1100μs 17.3735μs 57.5589 KOps/s 56.3480 KOps/s $\color{#35bf28}+2.15\%$
test_step_mdp_speed[False-False-True-False-False] 37.8800μs 11.3943μs 87.7632 KOps/s 86.6149 KOps/s $\color{#35bf28}+1.33\%$
test_step_mdp_speed[False-False-False-True-True] 70.8630μs 27.5429μs 36.3070 KOps/s 35.2300 KOps/s $\color{#35bf28}+3.06\%$
test_step_mdp_speed[False-False-False-True-False] 73.4770μs 19.3435μs 51.6969 KOps/s 51.1203 KOps/s $\color{#35bf28}+1.13\%$
test_step_mdp_speed[False-False-False-False-True] 55.6540μs 18.3409μs 54.5230 KOps/s 53.5520 KOps/s $\color{#35bf28}+1.81\%$
test_step_mdp_speed[False-False-False-False-False] 0.1522ms 12.6734μs 78.9052 KOps/s 79.9173 KOps/s $\color{#d91a1a}-1.27\%$
test_values[generalized_advantage_estimate-True-True] 16.3549ms 12.3469ms 80.9922 Ops/s 80.3893 Ops/s $\color{#35bf28}+0.75\%$
test_values[vec_generalized_advantage_estimate-True-True] 34.9145ms 27.6101ms 36.2187 Ops/s 36.1903 Ops/s $\color{#35bf28}+0.08\%$
test_values[td0_return_estimate-False-False] 0.2402ms 0.1743ms 5.7361 KOps/s 5.6841 KOps/s $\color{#35bf28}+0.91\%$
test_values[td1_return_estimate-False-False] 26.8887ms 26.0086ms 38.4489 Ops/s 37.9366 Ops/s $\color{#35bf28}+1.35\%$
test_values[vec_td1_return_estimate-False-False] 36.2550ms 27.9698ms 35.7529 Ops/s 35.6230 Ops/s $\color{#35bf28}+0.36\%$
test_values[td_lambda_return_estimate-True-False] 51.8458ms 36.7993ms 27.1745 Ops/s 27.3648 Ops/s $\color{#d91a1a}-0.70\%$
test_values[vec_td_lambda_return_estimate-True-False] 35.0960ms 27.7276ms 36.0651 Ops/s 36.0486 Ops/s $\color{#35bf28}+0.05\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 8.4865ms 8.1564ms 122.6032 Ops/s 119.0411 Ops/s $\color{#35bf28}+2.99\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.2532ms 1.8321ms 545.8271 Ops/s 538.4831 Ops/s $\color{#35bf28}+1.36\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 8.9961ms 0.4277ms 2.3379 KOps/s 2.2931 KOps/s $\color{#35bf28}+1.95\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 45.1153ms 38.3014ms 26.1087 Ops/s 25.7566 Ops/s $\color{#35bf28}+1.37\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 11.1991ms 2.6328ms 379.8193 Ops/s 379.6925 Ops/s $\color{#35bf28}+0.03\%$
test_dqn_speed 80.4727ms 8.0261ms 124.5942 Ops/s 121.8127 Ops/s $\color{#35bf28}+2.28\%$
test_ddpg_speed 15.7821ms 14.3098ms 69.8824 Ops/s 68.6146 Ops/s $\color{#35bf28}+1.85\%$
test_sac_speed 32.2980ms 28.9934ms 34.4906 Ops/s 34.0421 Ops/s $\color{#35bf28}+1.32\%$
test_redq_speed 37.4554ms 35.2891ms 28.3374 Ops/s 28.0052 Ops/s $\color{#35bf28}+1.19\%$
test_redq_deprec_speed 34.5112ms 25.3735ms 39.4113 Ops/s 39.4209 Ops/s $\color{#d91a1a}-0.02\%$
test_td3_speed 28.3459ms 20.0167ms 49.9584 Ops/s 49.4805 Ops/s $\color{#35bf28}+0.97\%$
test_cql_speed 96.0287ms 88.0027ms 11.3633 Ops/s 11.4825 Ops/s $\color{#d91a1a}-1.04\%$
test_a2c_speed 36.0535ms 27.2519ms 36.6946 Ops/s 37.4574 Ops/s $\color{#d91a1a}-2.04\%$
test_ppo_speed 28.1016ms 27.1858ms 36.7839 Ops/s 36.7769 Ops/s $\color{#35bf28}+0.02\%$
test_reinforce_speed 26.9093ms 25.9784ms 38.4935 Ops/s 38.6725 Ops/s $\color{#d91a1a}-0.46\%$
test_iql_speed 64.9926ms 63.6109ms 15.7206 Ops/s 15.7906 Ops/s $\color{#d91a1a}-0.44\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 1.8386ms 1.4318ms 698.4100 Ops/s 693.5483 Ops/s $\color{#35bf28}+0.70\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 9.7939ms 0.5197ms 1.9242 KOps/s 1.9298 KOps/s $\color{#d91a1a}-0.29\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 8.7846ms 0.5018ms 1.9930 KOps/s 1.9809 KOps/s $\color{#35bf28}+0.61\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 1.9085ms 1.4166ms 705.9073 Ops/s 699.8083 Ops/s $\color{#35bf28}+0.87\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.6337ms 0.5064ms 1.9749 KOps/s 1.8455 KOps/s $\textbf{\color{#35bf28}+7.01\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.1041s 0.5822ms 1.7177 KOps/s 1.7228 KOps/s $\color{#d91a1a}-0.29\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 2.2397ms 1.6218ms 616.5913 Ops/s 602.6443 Ops/s $\color{#35bf28}+2.31\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 10.0383ms 0.6500ms 1.5384 KOps/s 1.5491 KOps/s $\color{#d91a1a}-0.69\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 9.3871ms 0.6382ms 1.5669 KOps/s 1.5808 KOps/s $\color{#d91a1a}-0.88\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 2.0017ms 1.4671ms 681.6268 Ops/s 674.1931 Ops/s $\color{#35bf28}+1.10\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.1003s 0.6407ms 1.5607 KOps/s 1.5774 KOps/s $\color{#d91a1a}-1.06\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6489ms 0.4956ms 2.0179 KOps/s 2.0101 KOps/s $\color{#35bf28}+0.38\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 2.2354ms 1.4318ms 698.4016 Ops/s 702.8646 Ops/s $\color{#d91a1a}-0.63\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.1014s 0.6321ms 1.5821 KOps/s 1.9205 KOps/s $\textbf{\color{#d91a1a}-17.62\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 8.7559ms 0.5064ms 1.9749 KOps/s 2.0243 KOps/s $\color{#d91a1a}-2.44\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 2.3659ms 1.6570ms 603.5054 Ops/s 601.9315 Ops/s $\color{#35bf28}+0.26\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 10.3782ms 0.6786ms 1.4736 KOps/s 1.5233 KOps/s $\color{#d91a1a}-3.27\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8869ms 0.6299ms 1.5875 KOps/s 1.5804 KOps/s $\color{#35bf28}+0.45\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1144s 17.6516ms 56.6522 Ops/s 57.5900 Ops/s $\color{#d91a1a}-1.63\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 15.5727ms 13.4980ms 74.0852 Ops/s 73.3391 Ops/s $\color{#35bf28}+1.02\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 4.5707ms 3.2475ms 307.9266 Ops/s 305.4912 Ops/s $\color{#35bf28}+0.80\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1089s 17.2678ms 57.9111 Ops/s 57.2091 Ops/s $\color{#35bf28}+1.23\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 15.9658ms 13.5538ms 73.7799 Ops/s 73.6779 Ops/s $\color{#35bf28}+0.14\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 11.6406ms 3.4373ms 290.9234 Ops/s 308.1673 Ops/s $\textbf{\color{#d91a1a}-5.60\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1101s 17.3528ms 57.6277 Ops/s 56.5549 Ops/s $\color{#35bf28}+1.90\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 19.0717ms 13.8986ms 71.9496 Ops/s 72.3936 Ops/s $\color{#d91a1a}-0.61\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 5.1262ms 3.5018ms 285.5665 Ops/s 285.9186 Ops/s $\color{#d91a1a}-0.12\%$

@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 12, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 92. Improved: $\large\color{#35bf28}1$. Worsened: $\large\color{#d91a1a}2$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1200s 0.1197s 8.3520 Ops/s 8.2295 Ops/s $\color{#35bf28}+1.49\%$
test_sync 0.1786s 0.1099s 9.0961 Ops/s 8.9879 Ops/s $\color{#35bf28}+1.20\%$
test_async 0.2704s 99.0553ms 10.0954 Ops/s 9.9230 Ops/s $\color{#35bf28}+1.74\%$
test_single_pixels 0.1451s 0.1447s 6.9114 Ops/s 6.8574 Ops/s $\color{#35bf28}+0.79\%$
test_sync_pixels 96.2725ms 95.4031ms 10.4818 Ops/s 10.5298 Ops/s $\color{#d91a1a}-0.46\%$
test_async_pixels 0.2657s 93.4975ms 10.6955 Ops/s 10.9493 Ops/s $\color{#d91a1a}-2.32\%$
test_simple 0.9452s 0.8773s 1.1399 Ops/s 1.1246 Ops/s $\color{#35bf28}+1.36\%$
test_transformed 1.1845s 1.1174s 0.8950 Ops/s 0.8919 Ops/s $\color{#35bf28}+0.34\%$
test_serial 2.5218s 2.4580s 0.4068 Ops/s 0.3966 Ops/s $\color{#35bf28}+2.59\%$
test_parallel 2.5456s 2.4872s 0.4021 Ops/s 0.4019 Ops/s $\color{#35bf28}+0.04\%$
test_step_mdp_speed[True-True-True-True-True] 0.2570ms 33.1551μs 30.1612 KOps/s 30.6288 KOps/s $\color{#d91a1a}-1.53\%$
test_step_mdp_speed[True-True-True-True-False] 0.2144ms 19.6907μs 50.7855 KOps/s 51.6361 KOps/s $\color{#d91a1a}-1.65\%$
test_step_mdp_speed[True-True-True-False-True] 0.2054ms 18.9010μs 52.9072 KOps/s 53.9226 KOps/s $\color{#d91a1a}-1.88\%$
test_step_mdp_speed[True-True-True-False-False] 27.0400μs 11.3957μs 87.7525 KOps/s 88.8495 KOps/s $\color{#d91a1a}-1.23\%$
test_step_mdp_speed[True-True-False-True-True] 0.2229ms 35.3457μs 28.2920 KOps/s 29.1232 KOps/s $\color{#d91a1a}-2.85\%$
test_step_mdp_speed[True-True-False-True-False] 0.2074ms 21.7308μs 46.0176 KOps/s 46.9232 KOps/s $\color{#d91a1a}-1.93\%$
test_step_mdp_speed[True-True-False-False-True] 0.1386ms 20.9515μs 47.7293 KOps/s 48.9810 KOps/s $\color{#d91a1a}-2.56\%$
test_step_mdp_speed[True-True-False-False-False] 0.1998ms 13.3771μs 74.7544 KOps/s 76.9811 KOps/s $\color{#d91a1a}-2.89\%$
test_step_mdp_speed[True-False-True-True-True] 0.2275ms 37.4124μs 26.7291 KOps/s 27.6406 KOps/s $\color{#d91a1a}-3.30\%$
test_step_mdp_speed[True-False-True-True-False] 0.2049ms 23.6192μs 42.3385 KOps/s 43.1577 KOps/s $\color{#d91a1a}-1.90\%$
test_step_mdp_speed[True-False-True-False-True] 0.2099ms 20.7303μs 48.2385 KOps/s 48.7892 KOps/s $\color{#d91a1a}-1.13\%$
test_step_mdp_speed[True-False-True-False-False] 0.1972ms 13.2382μs 75.5388 KOps/s 75.4195 KOps/s $\color{#35bf28}+0.16\%$
test_step_mdp_speed[True-False-False-True-True] 0.2353ms 38.9059μs 25.7030 KOps/s 26.3318 KOps/s $\color{#d91a1a}-2.39\%$
test_step_mdp_speed[True-False-False-True-False] 44.3410μs 25.1836μs 39.7084 KOps/s 39.6752 KOps/s $\color{#35bf28}+0.08\%$
test_step_mdp_speed[True-False-False-False-True] 0.2142ms 22.6281μs 44.1928 KOps/s 45.4458 KOps/s $\color{#d91a1a}-2.76\%$
test_step_mdp_speed[True-False-False-False-False] 30.6400μs 14.9540μs 66.8716 KOps/s 67.4468 KOps/s $\color{#d91a1a}-0.85\%$
test_step_mdp_speed[False-True-True-True-True] 0.2275ms 36.7072μs 27.2426 KOps/s 27.6321 KOps/s $\color{#d91a1a}-1.41\%$
test_step_mdp_speed[False-True-True-True-False] 40.0410μs 23.5512μs 42.4607 KOps/s 43.7621 KOps/s $\color{#d91a1a}-2.97\%$
test_step_mdp_speed[False-True-True-False-True] 0.2098ms 25.1203μs 39.8084 KOps/s 41.3100 KOps/s $\color{#d91a1a}-3.63\%$
test_step_mdp_speed[False-True-True-False-False] 0.2032ms 15.0129μs 66.6093 KOps/s 67.5914 KOps/s $\color{#d91a1a}-1.45\%$
test_step_mdp_speed[False-True-False-True-True] 0.1382ms 39.0631μs 25.5996 KOps/s 26.0709 KOps/s $\color{#d91a1a}-1.81\%$
test_step_mdp_speed[False-True-False-True-False] 0.2229ms 25.3716μs 39.4141 KOps/s 39.9981 KOps/s $\color{#d91a1a}-1.46\%$
test_step_mdp_speed[False-True-False-False-True] 0.2177ms 26.3193μs 37.9949 KOps/s 38.2862 KOps/s $\color{#d91a1a}-0.76\%$
test_step_mdp_speed[False-True-False-False-False] 0.1985ms 16.7143μs 59.8290 KOps/s 59.8093 KOps/s $\color{#35bf28}+0.03\%$
test_step_mdp_speed[False-False-True-True-True] 67.3710μs 40.8832μs 24.4599 KOps/s 25.4706 KOps/s $\color{#d91a1a}-3.97\%$
test_step_mdp_speed[False-False-True-True-False] 0.2108ms 27.4020μs 36.4937 KOps/s 37.0926 KOps/s $\color{#d91a1a}-1.61\%$
test_step_mdp_speed[False-False-True-False-True] 47.5010μs 26.7440μs 37.3915 KOps/s 37.1172 KOps/s $\color{#35bf28}+0.74\%$
test_step_mdp_speed[False-False-True-False-False] 0.2001ms 16.6991μs 59.8834 KOps/s 59.3406 KOps/s $\color{#35bf28}+0.91\%$
test_step_mdp_speed[False-False-False-True-True] 64.5610μs 42.2429μs 23.6726 KOps/s 23.9659 KOps/s $\color{#d91a1a}-1.22\%$
test_step_mdp_speed[False-False-False-True-False] 0.2164ms 29.1376μs 34.3199 KOps/s 34.5758 KOps/s $\color{#d91a1a}-0.74\%$
test_step_mdp_speed[False-False-False-False-True] 0.2132ms 28.2533μs 35.3940 KOps/s 35.7218 KOps/s $\color{#d91a1a}-0.92\%$
test_step_mdp_speed[False-False-False-False-False] 0.2005ms 18.6960μs 53.4873 KOps/s 53.7354 KOps/s $\color{#d91a1a}-0.46\%$
test_values[generalized_advantage_estimate-True-True] 25.0212ms 24.5986ms 40.6526 Ops/s 40.0976 Ops/s $\color{#35bf28}+1.38\%$
test_values[vec_generalized_advantage_estimate-True-True] 85.5991ms 3.2797ms 304.9025 Ops/s 309.8607 Ops/s $\color{#d91a1a}-1.60\%$
test_values[td0_return_estimate-False-False] 91.6420μs 62.2441μs 16.0658 KOps/s 16.2799 KOps/s $\color{#d91a1a}-1.32\%$
test_values[td1_return_estimate-False-False] 53.0632ms 52.6522ms 18.9926 Ops/s 18.9078 Ops/s $\color{#35bf28}+0.45\%$
test_values[vec_td1_return_estimate-False-False] 1.9693ms 1.7590ms 568.4944 Ops/s 568.5081 Ops/s $-0.00\%$
test_values[td_lambda_return_estimate-True-False] 86.2705ms 84.0916ms 11.8918 Ops/s 11.8336 Ops/s $\color{#35bf28}+0.49\%$
test_values[vec_td_lambda_return_estimate-True-False] 2.0084ms 1.7593ms 568.4166 Ops/s 568.1686 Ops/s $\color{#35bf28}+0.04\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 23.5965ms 23.3635ms 42.8018 Ops/s 42.5470 Ops/s $\color{#35bf28}+0.60\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 0.8341ms 0.6975ms 1.4338 KOps/s 1.4446 KOps/s $\color{#d91a1a}-0.75\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.7222ms 0.6516ms 1.5347 KOps/s 1.5391 KOps/s $\color{#d91a1a}-0.28\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.6087ms 1.4528ms 688.3106 Ops/s 688.9093 Ops/s $\color{#d91a1a}-0.09\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.9335ms 0.6761ms 1.4791 KOps/s 1.4936 KOps/s $\color{#d91a1a}-0.97\%$
test_dqn_speed 13.9055ms 7.3379ms 136.2779 Ops/s 133.2935 Ops/s $\color{#35bf28}+2.24\%$
test_ddpg_speed 15.3302ms 14.3941ms 69.4730 Ops/s 68.6335 Ops/s $\color{#35bf28}+1.22\%$
test_sac_speed 29.9424ms 29.1564ms 34.2978 Ops/s 33.7260 Ops/s $\color{#35bf28}+1.70\%$
test_redq_speed 35.5983ms 34.8234ms 28.7163 Ops/s 28.5511 Ops/s $\color{#35bf28}+0.58\%$
test_redq_deprec_speed 24.9615ms 24.1820ms 41.3531 Ops/s 40.9206 Ops/s $\color{#35bf28}+1.06\%$
test_td3_speed 28.9016ms 19.8154ms 50.4658 Ops/s 49.6943 Ops/s $\color{#35bf28}+1.55\%$
test_cql_speed 84.7697ms 83.6381ms 11.9563 Ops/s 11.7448 Ops/s $\color{#35bf28}+1.80\%$
test_a2c_speed 27.7789ms 26.8392ms 37.2590 Ops/s 36.7310 Ops/s $\color{#35bf28}+1.44\%$
test_ppo_speed 27.9581ms 27.1089ms 36.8882 Ops/s 36.2718 Ops/s $\color{#35bf28}+1.70\%$
test_reinforce_speed 28.6268ms 26.2626ms 38.0770 Ops/s 37.9126 Ops/s $\color{#35bf28}+0.43\%$
test_iql_speed 58.5220ms 57.5386ms 17.3796 Ops/s 17.1343 Ops/s $\color{#35bf28}+1.43\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 2.5311ms 1.9099ms 523.5867 Ops/s 524.7242 Ops/s $\color{#d91a1a}-0.22\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.0619ms 0.8352ms 1.1973 KOps/s 1.1980 KOps/s $\color{#d91a1a}-0.06\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.9861ms 0.8242ms 1.2133 KOps/s 1.2121 KOps/s $\color{#35bf28}+0.10\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 2.0939ms 1.8775ms 532.6194 Ops/s 530.8775 Ops/s $\color{#35bf28}+0.33\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.1086ms 0.8259ms 1.2108 KOps/s 1.2129 KOps/s $\color{#d91a1a}-0.17\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 1.0405ms 0.8175ms 1.2233 KOps/s 1.2287 KOps/s $\color{#d91a1a}-0.44\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 0.1102s 2.4170ms 413.7445 Ops/s 461.6088 Ops/s $\textbf{\color{#d91a1a}-10.37\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.1663ms 0.9539ms 1.0483 KOps/s 1.0487 KOps/s $\color{#d91a1a}-0.03\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 1.1539ms 0.9437ms 1.0597 KOps/s 1.0574 KOps/s $\color{#35bf28}+0.22\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 2.6837ms 1.9136ms 522.5731 Ops/s 523.9264 Ops/s $\color{#d91a1a}-0.26\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.0508ms 0.8358ms 1.1964 KOps/s 1.1941 KOps/s $\color{#35bf28}+0.20\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 1.0419ms 0.8266ms 1.2098 KOps/s 1.2109 KOps/s $\color{#d91a1a}-0.09\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 2.5213ms 1.8865ms 530.0721 Ops/s 528.4104 Ops/s $\color{#35bf28}+0.31\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.0559ms 0.8263ms 1.2102 KOps/s 1.2110 KOps/s $\color{#d91a1a}-0.07\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.9806ms 0.8177ms 1.2229 KOps/s 1.2258 KOps/s $\color{#d91a1a}-0.24\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 0.1131s 2.4259ms 412.2230 Ops/s 460.5650 Ops/s $\textbf{\color{#d91a1a}-10.50\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.1180ms 0.9555ms 1.0466 KOps/s 1.0449 KOps/s $\color{#35bf28}+0.16\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 1.0849ms 0.9456ms 1.0575 KOps/s 1.0574 KOps/s $+0.01\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1621s 18.8792ms 52.9683 Ops/s 54.7302 Ops/s $\color{#d91a1a}-3.22\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 16.3440ms 13.7635ms 72.6561 Ops/s 73.7118 Ops/s $\color{#d91a1a}-1.43\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 7.0385ms 3.3757ms 296.2382 Ops/s 303.4321 Ops/s $\color{#d91a1a}-2.37\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1209s 17.8626ms 55.9828 Ops/s 55.6945 Ops/s $\color{#35bf28}+0.52\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 15.8903ms 13.7321ms 72.8221 Ops/s 62.9555 Ops/s $\textbf{\color{#35bf28}+15.67\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 4.8856ms 3.2829ms 304.6127 Ops/s 302.6388 Ops/s $\color{#35bf28}+0.65\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1227s 18.1664ms 55.0466 Ops/s 55.1304 Ops/s $\color{#d91a1a}-0.15\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 16.7360ms 13.9777ms 71.5428 Ops/s 72.4970 Ops/s $\color{#d91a1a}-1.32\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 5.1117ms 3.5099ms 284.9079 Ops/s 286.3886 Ops/s $\color{#d91a1a}-0.52\%$

@vmoens vmoens merged commit 6667dd8 into main Jan 12, 2024
59 of 61 checks passed
@vmoens vmoens deleted the regroup-tests branch January 12, 2024 17:19
@vmoens vmoens restored the regroup-tests branch January 12, 2024 17:24
@vmoens vmoens deleted the regroup-tests branch February 27, 2024 00:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CI Has to do with CI setup (e.g. wheels & builds, tests...) CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Data Data-related PR, will launch data-related jobs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@vmoens @facebook-github-bot