Skip to content

[CI] Add flaky test tracking system#3408

Merged
vmoens merged 1 commit intomainfrom
flaky-ci
Jan 29, 2026
Merged

[CI] Add flaky test tracking system#3408
vmoens merged 1 commit intomainfrom
flaky-ci

Conversation

@vmoens
Copy link
Collaborator

@vmoens vmoens commented Jan 29, 2026

Summary

Adds a GitHub-native flaky CI test tracking system to identify, track, and visualize flaky tests over time on the main branch.

Features

  • Structured test output: Configures pytest to emit JSON results via pytest-json-report
  • Artifact collection: Uploads test result artifacts with CI metadata on every CI run
  • Daily aggregation: Scheduled workflow that analyzes recent CI runs to identify flaky tests
  • Flaky detection: Identifies tests with failure rates between 5-95% with configurable thresholds
  • Reports: Generates JSON and Markdown reports with flaky test details
  • Dashboard: Static HTML dashboard deployed to GitHub Pages with Chart.js visualization
  • Badge: Dynamic shields.io badge showing flaky test count
  • Notifications: Creates/updates a GitHub Issue with daily flaky test summaries

Workflows Analyzed

  • test-linux.yml - Main unit tests
  • test-linux-libs.yml - Library integrations (brax, gym, envpool, etc.)
  • test-linux-habitat.yml - Habitat environment tests
  • test-linux-llm.yml - LLM/RLHF tests
  • test-linux-sota.yml - SOTA implementation tests

Files Added

File Description
.github/workflows/flaky-test-tracker.yml Daily scheduled workflow for aggregation
.github/scripts/analyze_flaky_tests.py Main analysis script with GitHub API integration
.github/scripts/generate_dashboard.py Dashboard HTML generator
.github/unittest/helpers/upload_test_results.py Test result processor with metadata
.github/unittest/helpers/pytest_json_report.sh Reusable helper for JSON reporting

Configuration

Flaky test detection thresholds (configurable in analyze_flaky_tests.py):

  • Minimum failure rate: 5%
  • Maximum failure rate: 95%
  • Minimum failures: 2
  • Minimum executions: 3

Outputs

  • Badge: https://pytorch.github.io/rl/flaky/badge.json
  • Dashboard: https://pytorch.github.io/rl/flaky/
  • JSON Report: https://pytorch.github.io/rl/flaky/flaky-tests.json
  • Markdown Report: https://pytorch.github.io/rl/flaky/flaky-tests.md

Test plan

  • Verify workflow runs successfully on schedule or manual trigger
  • Verify test result JSON files are generated during CI runs
  • Verify dashboard is deployed to GitHub Pages
  • Verify badge displays correctly in README

@vmoens
[CI] Add flaky test tracking system
1155129 
Add a GitHub-native flaky CI test tracking system that:

- Configures pytest to emit JSON results via pytest-json-report
- Uploads test result artifacts with CI metadata on every run
- Runs a daily aggregation workflow that analyzes recent CI runs
- Identifies flaky tests based on failure rate heuristics
- Generates JSON/Markdown reports and a static dashboard
- Deploys to GitHub Pages with dynamic shields.io badge
- Creates/updates a GitHub Issue for notifications

Workflows analyzed:
- test-linux.yml (main unit tests)
- test-linux-libs.yml (library integrations)
- test-linux-habitat.yml (Habitat environment)
- test-linux-llm.yml (LLM/RLHF tests)
- test-linux-sota.yml (SOTA implementations)

Files added:
- .github/workflows/flaky-test-tracker.yml
- .github/scripts/analyze_flaky_tests.py
- .github/scripts/generate_dashboard.py
- .github/unittest/helpers/upload_test_results.py
- .github/unittest/helpers/pytest_json_report.sh
@pytorch-bot
Copy link

pytorch-bot bot commented Jan 29, 2026

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3408

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 29, 2026
@github-actions github-actions bot added the CI Has to do with CI setup (e.g. wheels & builds, tests...) label Jan 29, 2026
@vmoens vmoens merged commit e09ee4f into main Jan 29, 2026
94 of 113 checks passed
@github-actions
Copy link
Contributor

github-actions bot commented Jan 29, 2026

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 148. Improved: $\large\color{#35bf28}11$. Worsened: $\large\color{#d91a1a}11$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_tensor_to_bytestream_speed[pickle] 81.5882μs 80.5716μs 12.4113 KOps/s 12.4949 KOps/s $\color{#d91a1a}-0.67\%$
test_tensor_to_bytestream_speed[torch.save] 0.1413ms 0.1402ms 7.1336 KOps/s 7.2063 KOps/s $\color{#d91a1a}-1.01\%$
test_tensor_to_bytestream_speed[untyped_storage] 0.1297s 0.1291s 7.7486 Ops/s 8.2322 Ops/s $\textbf{\color{#d91a1a}-5.87\%}$
test_tensor_to_bytestream_speed[numpy] 2.6056μs 2.5862μs 386.6682 KOps/s 399.2190 KOps/s $\color{#d91a1a}-3.14\%$
test_tensor_to_bytestream_speed[safetensors] 38.9249μs 37.5879μs 26.6043 KOps/s 27.6907 KOps/s $\color{#d91a1a}-3.92\%$
test_simple 0.9139s 0.8230s 1.2151 Ops/s 1.2216 Ops/s $\color{#d91a1a}-0.53\%$
test_transformed 1.5452s 1.4528s 0.6883 Ops/s 0.6871 Ops/s $\color{#35bf28}+0.18\%$
test_serial 2.3227s 2.3194s 0.4311 Ops/s 0.4350 Ops/s $\color{#d91a1a}-0.88\%$
test_parallel 2.0667s 1.9974s 0.5006 Ops/s 0.5209 Ops/s $\color{#d91a1a}-3.88\%$
test_step_mdp_speed[True-True-True-True-True] 0.3481ms 44.2892μs 22.5789 KOps/s 22.3057 KOps/s $\color{#35bf28}+1.22\%$
test_step_mdp_speed[True-True-True-True-False] 56.2000μs 24.8907μs 40.1756 KOps/s 39.6036 KOps/s $\color{#35bf28}+1.44\%$
test_step_mdp_speed[True-True-True-False-True] 57.8810μs 24.6436μs 40.5785 KOps/s 39.5310 KOps/s $\color{#35bf28}+2.65\%$
test_step_mdp_speed[True-True-True-False-False] 42.3210μs 13.8435μs 72.2362 KOps/s 71.7815 KOps/s $\color{#35bf28}+0.63\%$
test_step_mdp_speed[True-True-False-True-True] 0.1042ms 48.3805μs 20.6695 KOps/s 20.6361 KOps/s $\color{#35bf28}+0.16\%$
test_step_mdp_speed[True-True-False-True-False] 79.7210μs 27.2535μs 36.6925 KOps/s 35.6593 KOps/s $\color{#35bf28}+2.90\%$
test_step_mdp_speed[True-True-False-False-True] 64.6610μs 27.6157μs 36.2113 KOps/s 35.7566 KOps/s $\color{#35bf28}+1.27\%$
test_step_mdp_speed[True-True-False-False-False] 59.1810μs 16.7262μs 59.7864 KOps/s 59.6045 KOps/s $\color{#35bf28}+0.31\%$
test_step_mdp_speed[True-False-True-True-True] 0.1169ms 50.9915μs 19.6111 KOps/s 20.1101 KOps/s $\color{#d91a1a}-2.48\%$
test_step_mdp_speed[True-False-True-True-False] 71.5710μs 30.6354μs 32.6420 KOps/s 33.2906 KOps/s $\color{#d91a1a}-1.95\%$
test_step_mdp_speed[True-False-True-False-True] 61.0210μs 27.5415μs 36.3088 KOps/s 36.0257 KOps/s $\color{#35bf28}+0.79\%$
test_step_mdp_speed[True-False-True-False-False] 54.6800μs 16.7465μs 59.7141 KOps/s 60.5012 KOps/s $\color{#d91a1a}-1.30\%$
test_step_mdp_speed[True-False-False-True-True] 94.1610μs 52.6170μs 19.0053 KOps/s 18.7881 KOps/s $\color{#35bf28}+1.16\%$
test_step_mdp_speed[True-False-False-True-False] 72.3910μs 32.6573μs 30.6210 KOps/s 30.1912 KOps/s $\color{#35bf28}+1.42\%$
test_step_mdp_speed[True-False-False-False-True] 0.1052ms 28.7905μs 34.7337 KOps/s 32.5590 KOps/s $\textbf{\color{#35bf28}+6.68\%}$
test_step_mdp_speed[True-False-False-False-False] 52.9510μs 18.9888μs 52.6627 KOps/s 51.8144 KOps/s $\color{#35bf28}+1.64\%$
test_step_mdp_speed[False-True-True-True-True] 0.1160ms 48.4762μs 20.6287 KOps/s 19.9047 KOps/s $\color{#35bf28}+3.64\%$
test_step_mdp_speed[False-True-True-True-False] 64.7910μs 29.8648μs 33.4842 KOps/s 32.6290 KOps/s $\color{#35bf28}+2.62\%$
test_step_mdp_speed[False-True-True-False-True] 68.7420μs 30.8973μs 32.3652 KOps/s 31.1329 KOps/s $\color{#35bf28}+3.96\%$
test_step_mdp_speed[False-True-True-False-False] 61.9810μs 18.0469μs 55.4112 KOps/s 55.1607 KOps/s $\color{#35bf28}+0.45\%$
test_step_mdp_speed[False-True-False-True-True] 2.6288ms 52.9491μs 18.8860 KOps/s 18.6465 KOps/s $\color{#35bf28}+1.28\%$
test_step_mdp_speed[False-True-False-True-False] 0.1016ms 32.7466μs 30.5376 KOps/s 29.9816 KOps/s $\color{#35bf28}+1.85\%$
test_step_mdp_speed[False-True-False-False-True] 98.9520μs 33.8301μs 29.5595 KOps/s 29.9057 KOps/s $\color{#d91a1a}-1.16\%$
test_step_mdp_speed[False-True-False-False-False] 50.8700μs 20.6129μs 48.5133 KOps/s 48.3277 KOps/s $\color{#35bf28}+0.38\%$
test_step_mdp_speed[False-False-True-True-True] 98.8810μs 56.0460μs 17.8425 KOps/s 17.8023 KOps/s $\color{#35bf28}+0.23\%$
test_step_mdp_speed[False-False-True-True-False] 82.7710μs 35.8989μs 27.8560 KOps/s 27.6854 KOps/s $\color{#35bf28}+0.62\%$
test_step_mdp_speed[False-False-True-False-True] 83.5910μs 33.9623μs 29.4444 KOps/s 29.3260 KOps/s $\color{#35bf28}+0.40\%$
test_step_mdp_speed[False-False-True-False-False] 60.7310μs 20.4744μs 48.8416 KOps/s 49.1300 KOps/s $\color{#d91a1a}-0.59\%$
test_step_mdp_speed[False-False-False-True-True] 0.1008ms 57.7167μs 17.3260 KOps/s 17.2627 KOps/s $\color{#35bf28}+0.37\%$
test_step_mdp_speed[False-False-False-True-False] 97.1820μs 38.5775μs 25.9218 KOps/s 26.2818 KOps/s $\color{#d91a1a}-1.37\%$
test_step_mdp_speed[False-False-False-False-True] 76.4110μs 35.8306μs 27.9091 KOps/s 27.9627 KOps/s $\color{#d91a1a}-0.19\%$
test_step_mdp_speed[False-False-False-False-False] 90.7910μs 23.1702μs 43.1589 KOps/s 43.4623 KOps/s $\color{#d91a1a}-0.70\%$
test_non_tensor_env_rollout_speed[1000-single-True] 0.7492s 0.7470s 1.3386 Ops/s 1.2884 Ops/s $\color{#35bf28}+3.89\%$
test_non_tensor_env_rollout_speed[1000-single-False] 0.7254s 0.6304s 1.5864 Ops/s 1.5664 Ops/s $\color{#35bf28}+1.27\%$
test_non_tensor_env_rollout_speed[1000-serial-no-buffers-True] 1.7490s 1.6736s 0.5975 Ops/s 0.5957 Ops/s $\color{#35bf28}+0.31\%$
test_non_tensor_env_rollout_speed[1000-serial-no-buffers-False] 1.5306s 1.4525s 0.6885 Ops/s 0.6843 Ops/s $\color{#35bf28}+0.61\%$
test_non_tensor_env_rollout_speed[1000-serial-buffers-True] 2.0027s 1.9273s 0.5188 Ops/s 0.5175 Ops/s $\color{#35bf28}+0.26\%$
test_non_tensor_env_rollout_speed[1000-serial-buffers-False] 1.7772s 1.7014s 0.5878 Ops/s 0.5848 Ops/s $\color{#35bf28}+0.51\%$
test_non_tensor_env_rollout_speed[1000-parallel-no-buffers-True] 4.7768s 4.6249s 0.2162 Ops/s 0.2163 Ops/s $\color{#d91a1a}-0.05\%$
test_non_tensor_env_rollout_speed[1000-parallel-no-buffers-False] 4.6616s 4.4927s 0.2226 Ops/s 0.2219 Ops/s $\color{#35bf28}+0.31\%$
test_non_tensor_env_rollout_speed[1000-parallel-buffers-True] 2.1035s 1.9843s 0.5040 Ops/s 0.5156 Ops/s $\color{#d91a1a}-2.26\%$
test_non_tensor_env_rollout_speed[1000-parallel-buffers-False] 1.7370s 1.6620s 0.6017 Ops/s 0.6022 Ops/s $\color{#d91a1a}-0.09\%$
test_values[generalized_advantage_estimate-True-True] 21.0348ms 20.6558ms 48.4125 Ops/s 49.5002 Ops/s $\color{#d91a1a}-2.20\%$
test_values[vec_generalized_advantage_estimate-True-True] 0.1443s 3.8178ms 261.9330 Ops/s 252.5185 Ops/s $\color{#35bf28}+3.73\%$
test_values[td0_return_estimate-False-False] 0.1215ms 85.5750μs 11.6857 KOps/s 11.8723 KOps/s $\color{#d91a1a}-1.57\%$
test_values[td1_return_estimate-False-False] 49.5309ms 49.1447ms 20.3481 Ops/s 20.6997 Ops/s $\color{#d91a1a}-1.70\%$
test_values[vec_td1_return_estimate-False-False] 1.3220ms 1.0989ms 909.9633 Ops/s 917.2290 Ops/s $\color{#d91a1a}-0.79\%$
test_values[td_lambda_return_estimate-True-False] 81.3602ms 81.0496ms 12.3381 Ops/s 12.6110 Ops/s $\color{#d91a1a}-2.16\%$
test_values[vec_td_lambda_return_estimate-True-False] 1.3063ms 1.0940ms 914.0672 Ops/s 919.4696 Ops/s $\color{#d91a1a}-0.59\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 21.0919ms 20.8110ms 48.0515 Ops/s 49.1298 Ops/s $\color{#d91a1a}-2.19\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 1.0641ms 0.7695ms 1.2995 KOps/s 1.3080 KOps/s $\color{#d91a1a}-0.65\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.7324ms 0.6854ms 1.4589 KOps/s 1.4730 KOps/s $\color{#d91a1a}-0.95\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.5426ms 1.5015ms 666.0188 Ops/s 668.9286 Ops/s $\color{#d91a1a}-0.44\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.7505ms 0.7015ms 1.4255 KOps/s 1.4387 KOps/s $\color{#d91a1a}-0.91\%$
test_dqn_speed[False-None] 1.6335ms 1.5445ms 647.4425 Ops/s 646.7712 Ops/s $\color{#35bf28}+0.10\%$
test_dqn_speed[False-backward] 2.4558ms 2.1971ms 455.1446 Ops/s 455.3982 Ops/s $\color{#d91a1a}-0.06\%$
test_dqn_speed[True-None] 0.6154ms 0.5387ms 1.8562 KOps/s 1.8379 KOps/s $\color{#35bf28}+0.99\%$
test_dqn_speed[True-backward] 1.2083ms 1.1809ms 846.8128 Ops/s 934.1345 Ops/s $\textbf{\color{#d91a1a}-9.35\%}$
test_dqn_speed[reduce-overhead-None] 0.6415ms 0.5598ms 1.7865 KOps/s 1.6767 KOps/s $\textbf{\color{#35bf28}+6.55\%}$
test_ddpg_speed[False-None] 3.3172ms 2.9229ms 342.1239 Ops/s 344.0852 Ops/s $\color{#d91a1a}-0.57\%$
test_ddpg_speed[False-backward] 4.7292ms 4.3438ms 230.2130 Ops/s 238.6651 Ops/s $\color{#d91a1a}-3.54\%$
test_ddpg_speed[True-None] 1.3652ms 1.2847ms 778.3747 Ops/s 765.8593 Ops/s $\color{#35bf28}+1.63\%$
test_ddpg_speed[True-backward] 2.5115ms 2.4628ms 406.0431 Ops/s 425.9239 Ops/s $\color{#d91a1a}-4.67\%$
test_ddpg_speed[reduce-overhead-None] 1.4072ms 1.3186ms 758.3892 Ops/s 759.4107 Ops/s $\color{#d91a1a}-0.13\%$
test_sac_speed[False-None] 8.8537ms 8.4096ms 118.9124 Ops/s 118.4554 Ops/s $\color{#35bf28}+0.39\%$
test_sac_speed[False-backward] 12.5083ms 11.7849ms 84.8544 Ops/s 87.0676 Ops/s $\color{#d91a1a}-2.54\%$
test_sac_speed[True-None] 1.8728ms 1.7873ms 559.5091 Ops/s 560.5391 Ops/s $\color{#d91a1a}-0.18\%$
test_sac_speed[True-backward] 3.6048ms 3.5370ms 282.7228 Ops/s 293.3166 Ops/s $\color{#d91a1a}-3.61\%$
test_sac_speed[reduce-overhead-None] 18.6295ms 10.6070ms 94.2772 Ops/s 93.3845 Ops/s $\color{#35bf28}+0.96\%$
test_redq_deprec_speed[False-None] 10.2705ms 9.4064ms 106.3103 Ops/s 105.5847 Ops/s $\color{#35bf28}+0.69\%$
test_redq_deprec_speed[False-backward] 13.4658ms 12.8270ms 77.9604 Ops/s 79.3665 Ops/s $\color{#d91a1a}-1.77\%$
test_redq_deprec_speed[True-None] 2.5857ms 2.5114ms 398.1823 Ops/s 395.7674 Ops/s $\color{#35bf28}+0.61\%$
test_redq_deprec_speed[True-backward] 4.5028ms 4.2698ms 234.2049 Ops/s 243.7804 Ops/s $\color{#d91a1a}-3.93\%$
test_redq_deprec_speed[reduce-overhead-None] 15.5413ms 9.6226ms 103.9217 Ops/s 105.7092 Ops/s $\color{#d91a1a}-1.69\%$
test_td3_speed[False-None] 8.6911ms 8.3282ms 120.0743 Ops/s 121.1898 Ops/s $\color{#d91a1a}-0.92\%$
test_td3_speed[False-backward] 11.6395ms 10.9742ms 91.1230 Ops/s 93.9312 Ops/s $\color{#d91a1a}-2.99\%$
test_td3_speed[True-None] 1.6286ms 1.5996ms 625.1603 Ops/s 627.6826 Ops/s $\color{#d91a1a}-0.40\%$
test_td3_speed[True-backward] 3.2510ms 3.1987ms 312.6236 Ops/s 326.0926 Ops/s $\color{#d91a1a}-4.13\%$
test_td3_speed[reduce-overhead-None] 55.5023ms 23.3103ms 42.8995 Ops/s 43.3858 Ops/s $\color{#d91a1a}-1.12\%$
test_cql_speed[False-None] 19.6478ms 17.4943ms 57.1614 Ops/s 57.6227 Ops/s $\color{#d91a1a}-0.80\%$
test_cql_speed[False-backward] 23.8111ms 23.2029ms 43.0981 Ops/s 44.1400 Ops/s $\color{#d91a1a}-2.36\%$
test_cql_speed[True-None] 3.5392ms 3.2704ms 305.7686 Ops/s 311.7069 Ops/s $\color{#d91a1a}-1.91\%$
test_cql_speed[True-backward] 5.7492ms 5.3389ms 187.3059 Ops/s 184.1514 Ops/s $\color{#35bf28}+1.71\%$
test_cql_speed[reduce-overhead-None] 18.3749ms 11.7438ms 85.1514 Ops/s 86.0292 Ops/s $\color{#d91a1a}-1.02\%$
test_a2c_speed[False-None] 4.3922ms 3.2924ms 303.7320 Ops/s 305.6855 Ops/s $\color{#d91a1a}-0.64\%$
test_a2c_speed[False-backward] 6.7401ms 6.2647ms 159.6240 Ops/s 154.4820 Ops/s $\color{#35bf28}+3.33\%$
test_a2c_speed[True-None] 1.4342ms 1.3230ms 755.8515 Ops/s 751.9632 Ops/s $\color{#35bf28}+0.52\%$
test_a2c_speed[True-backward] 3.0021ms 2.9131ms 343.2739 Ops/s 328.7429 Ops/s $\color{#35bf28}+4.42\%$
test_a2c_speed[reduce-overhead-None] 1.1556ms 0.9655ms 1.0357 KOps/s 1.0193 KOps/s $\color{#35bf28}+1.61\%$
test_ppo_speed[False-None] 4.0952ms 3.9189ms 255.1728 Ops/s 255.2195 Ops/s $\color{#d91a1a}-0.02\%$
test_ppo_speed[False-backward] 7.5360ms 7.1174ms 140.4998 Ops/s 136.9798 Ops/s $\color{#35bf28}+2.57\%$
test_ppo_speed[True-None] 1.5108ms 1.3947ms 717.0212 Ops/s 708.0938 Ops/s $\color{#35bf28}+1.26\%$
test_ppo_speed[True-backward] 3.4577ms 3.2258ms 310.0008 Ops/s 311.1484 Ops/s $\color{#d91a1a}-0.37\%$
test_ppo_speed[reduce-overhead-None] 1.0690ms 1.0190ms 981.3572 Ops/s 939.1581 Ops/s $\color{#35bf28}+4.49\%$
test_reinforce_speed[False-None] 2.4398ms 2.3062ms 433.6064 Ops/s 433.5622 Ops/s $\color{#35bf28}+0.01\%$
test_reinforce_speed[False-backward] 3.8857ms 3.4632ms 288.7510 Ops/s 292.5300 Ops/s $\color{#d91a1a}-1.29\%$
test_reinforce_speed[True-None] 1.3786ms 1.2357ms 809.2624 Ops/s 802.6682 Ops/s $\color{#35bf28}+0.82\%$
test_reinforce_speed[True-backward] 3.2091ms 3.0087ms 332.3725 Ops/s 330.3640 Ops/s $\color{#35bf28}+0.61\%$
test_reinforce_speed[reduce-overhead-None] 0.4605s 10.0793ms 99.2131 Ops/s 97.3842 Ops/s $\color{#35bf28}+1.88\%$
test_iql_speed[False-None] 10.3143ms 9.5632ms 104.5671 Ops/s 104.3476 Ops/s $\color{#35bf28}+0.21\%$
test_iql_speed[False-backward] 13.9695ms 13.6510ms 73.2549 Ops/s 74.5890 Ops/s $\color{#d91a1a}-1.79\%$
test_iql_speed[True-None] 2.2335ms 2.1317ms 469.1003 Ops/s 455.4741 Ops/s $\color{#35bf28}+2.99\%$
test_iql_speed[True-backward] 5.0304ms 4.7756ms 209.3985 Ops/s 206.1856 Ops/s $\color{#35bf28}+1.56\%$
test_iql_speed[reduce-overhead-None] 17.1663ms 10.1141ms 98.8716 Ops/s 75.8651 Ops/s $\textbf{\color{#35bf28}+30.33\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 6.1018ms 5.9631ms 167.6993 Ops/s 166.7937 Ops/s $\color{#35bf28}+0.54\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.0389ms 0.2863ms 3.4924 KOps/s 2.6329 KOps/s $\textbf{\color{#35bf28}+32.64\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.4942ms 0.2676ms 3.7367 KOps/s 2.7674 KOps/s $\textbf{\color{#35bf28}+35.03\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.1087ms 5.8324ms 171.4563 Ops/s 171.6447 Ops/s $\color{#d91a1a}-0.11\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.8104ms 0.2971ms 3.3655 KOps/s 3.2708 KOps/s $\color{#35bf28}+2.89\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.5452ms 0.3209ms 3.1166 KOps/s 3.3887 KOps/s $\textbf{\color{#d91a1a}-8.03\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 1.6114ms 1.2961ms 771.5567 Ops/s 677.7946 Ops/s $\textbf{\color{#35bf28}+13.83\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 1.4330ms 1.1982ms 834.5827 Ops/s 734.3900 Ops/s $\textbf{\color{#35bf28}+13.64\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 6.1206ms 5.9922ms 166.8849 Ops/s 165.0214 Ops/s $\color{#35bf28}+1.13\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.0175ms 0.4862ms 2.0567 KOps/s 2.2582 KOps/s $\textbf{\color{#d91a1a}-8.92\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.6728ms 0.4403ms 2.2712 KOps/s 2.4033 KOps/s $\textbf{\color{#d91a1a}-5.50\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 5.8858ms 5.8252ms 171.6668 Ops/s 170.3478 Ops/s $\color{#35bf28}+0.77\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2.0753ms 0.3064ms 3.2642 KOps/s 3.5346 KOps/s $\textbf{\color{#d91a1a}-7.65\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.5815ms 0.3457ms 2.8927 KOps/s 3.7881 KOps/s $\textbf{\color{#d91a1a}-23.64\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.1597ms 5.8363ms 171.3420 Ops/s 170.1916 Ops/s $\color{#35bf28}+0.68\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.7918ms 0.3296ms 3.0339 KOps/s 2.6724 KOps/s $\textbf{\color{#35bf28}+13.53\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6375ms 0.3735ms 2.6773 KOps/s 2.9338 KOps/s $\textbf{\color{#d91a1a}-8.74\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 6.0856ms 5.9825ms 167.1530 Ops/s 165.3306 Ops/s $\color{#35bf28}+1.10\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.0410ms 0.5271ms 1.8972 KOps/s 2.0614 KOps/s $\textbf{\color{#d91a1a}-7.97\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.7141ms 0.5182ms 1.9296 KOps/s 1.9712 KOps/s $\color{#d91a1a}-2.11\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.6077s 17.0982ms 58.4855 Ops/s 50.4679 Ops/s $\textbf{\color{#35bf28}+15.89\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 11.0781ms 1.9695ms 507.7401 Ops/s 495.9556 Ops/s $\color{#35bf28}+2.38\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 7.4696ms 1.2436ms 804.1114 Ops/s 760.6846 Ops/s $\textbf{\color{#35bf28}+5.71\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 6.7244ms 5.0534ms 197.8848 Ops/s 193.4261 Ops/s $\color{#35bf28}+2.31\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 11.9215ms 1.9399ms 515.4810 Ops/s 515.6271 Ops/s $\color{#d91a1a}-0.03\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 1.0189ms 0.9027ms 1.1078 KOps/s 1.0781 KOps/s $\color{#35bf28}+2.76\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.5524s 16.2051ms 61.7091 Ops/s 51.0400 Ops/s $\textbf{\color{#35bf28}+20.90\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 12.6603ms 2.2790ms 438.7925 Ops/s 505.4726 Ops/s $\textbf{\color{#d91a1a}-13.19\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 2.2941ms 1.2696ms 787.6629 Ops/s 914.1660 Ops/s $\textbf{\color{#d91a1a}-13.84\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] 37.9769ms 36.1145ms 27.6897 Ops/s 27.3931 Ops/s $\color{#35bf28}+1.08\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] 20.1514ms 18.5201ms 53.9954 Ops/s 52.4007 Ops/s $\color{#35bf28}+3.04\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] 41.8062ms 37.0673ms 26.9780 Ops/s 26.6808 Ops/s $\color{#35bf28}+1.11\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] 20.2933ms 18.6503ms 53.6184 Ops/s 52.0161 Ops/s $\color{#35bf28}+3.08\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 41.1255ms 38.8959ms 25.7096 Ops/s 25.6485 Ops/s $\color{#35bf28}+0.24\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False] 21.6429ms 20.2773ms 49.3163 Ops/s 49.6543 Ops/s $\color{#d91a1a}-0.68\%$

@github-actions
Copy link
Contributor

github-actions bot commented Jan 29, 2026

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 153. Improved: $\large\color{#35bf28}8$. Worsened: $\large\color{#d91a1a}18$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_tensor_to_bytestream_speed[pickle] 83.1898μs 81.3893μs 12.2866 KOps/s 11.7650 KOps/s $\color{#35bf28}+4.43\%$
test_tensor_to_bytestream_speed[torch.save] 0.1461ms 0.1453ms 6.8830 KOps/s 7.0583 KOps/s $\color{#d91a1a}-2.48\%$
test_tensor_to_bytestream_speed[untyped_storage] 0.1378s 0.1373s 7.2834 Ops/s 9.2769 Ops/s $\textbf{\color{#d91a1a}-21.49\%}$
test_tensor_to_bytestream_speed[numpy] 2.8823μs 2.8746μs 347.8793 KOps/s 384.5755 KOps/s $\textbf{\color{#d91a1a}-9.54\%}$
test_tensor_to_bytestream_speed[safetensors] 38.9953μs 38.7923μs 25.7783 KOps/s 25.7764 KOps/s $+0.01\%$
test_simple 0.6745s 0.5817s 1.7192 Ops/s 1.7243 Ops/s $\color{#d91a1a}-0.30\%$
test_transformed 1.2753s 1.1814s 0.8465 Ops/s 0.8598 Ops/s $\color{#d91a1a}-1.55\%$
test_serial 1.8278s 1.7318s 0.5774 Ops/s 0.5946 Ops/s $\color{#d91a1a}-2.89\%$
test_parallel 1.2160s 1.1475s 0.8715 Ops/s 0.9070 Ops/s $\color{#d91a1a}-3.92\%$
test_step_mdp_speed[True-True-True-True-True] 0.3178ms 46.0666μs 21.7077 KOps/s 21.8994 KOps/s $\color{#d91a1a}-0.88\%$
test_step_mdp_speed[True-True-True-True-False] 59.8910μs 25.5657μs 39.1149 KOps/s 38.9215 KOps/s $\color{#35bf28}+0.50\%$
test_step_mdp_speed[True-True-True-False-True] 54.3710μs 25.5108μs 39.1991 KOps/s 39.1482 KOps/s $\color{#35bf28}+0.13\%$
test_step_mdp_speed[True-True-True-False-False] 47.6410μs 14.3615μs 69.6305 KOps/s 70.6652 KOps/s $\color{#d91a1a}-1.46\%$
test_step_mdp_speed[True-True-False-True-True] 89.6420μs 49.1795μs 20.3337 KOps/s 20.5663 KOps/s $\color{#d91a1a}-1.13\%$
test_step_mdp_speed[True-True-False-True-False] 56.6110μs 28.2918μs 35.3459 KOps/s 35.1131 KOps/s $\color{#35bf28}+0.66\%$
test_step_mdp_speed[True-True-False-False-True] 59.2910μs 28.8371μs 34.6776 KOps/s 34.4240 KOps/s $\color{#35bf28}+0.74\%$
test_step_mdp_speed[True-True-False-False-False] 45.5400μs 16.8927μs 59.1971 KOps/s 58.7889 KOps/s $\color{#35bf28}+0.69\%$
test_step_mdp_speed[True-False-True-True-True] 92.4020μs 51.9457μs 19.2509 KOps/s 19.2071 KOps/s $\color{#35bf28}+0.23\%$
test_step_mdp_speed[True-False-True-True-False] 65.0620μs 31.4404μs 31.8063 KOps/s 31.9379 KOps/s $\color{#d91a1a}-0.41\%$
test_step_mdp_speed[True-False-True-False-True] 62.5410μs 28.3417μs 35.2837 KOps/s 34.5968 KOps/s $\color{#35bf28}+1.99\%$
test_step_mdp_speed[True-False-True-False-False] 44.0610μs 17.0586μs 58.6216 KOps/s 58.3781 KOps/s $\color{#35bf28}+0.42\%$
test_step_mdp_speed[True-False-False-True-True] 96.4020μs 54.4556μs 18.3636 KOps/s 18.4650 KOps/s $\color{#d91a1a}-0.55\%$
test_step_mdp_speed[True-False-False-True-False] 62.7310μs 34.0283μs 29.3873 KOps/s 29.2871 KOps/s $\color{#35bf28}+0.34\%$
test_step_mdp_speed[True-False-False-False-True] 68.3220μs 30.7455μs 32.5251 KOps/s 32.2452 KOps/s $\color{#35bf28}+0.87\%$
test_step_mdp_speed[True-False-False-False-False] 51.6610μs 19.7267μs 50.6927 KOps/s 50.6241 KOps/s $\color{#35bf28}+0.14\%$
test_step_mdp_speed[False-True-True-True-True] 0.1102ms 50.9947μs 19.6099 KOps/s 19.3561 KOps/s $\color{#35bf28}+1.31\%$
test_step_mdp_speed[False-True-True-True-False] 58.4610μs 30.7999μs 32.4676 KOps/s 32.1991 KOps/s $\color{#35bf28}+0.83\%$
test_step_mdp_speed[False-True-True-False-True] 71.4310μs 32.2536μs 31.0043 KOps/s 31.2581 KOps/s $\color{#d91a1a}-0.81\%$
test_step_mdp_speed[False-True-True-False-False] 53.0010μs 18.8374μs 53.0859 KOps/s 53.3203 KOps/s $\color{#d91a1a}-0.44\%$
test_step_mdp_speed[False-True-False-True-True] 2.6616ms 55.2962μs 18.0844 KOps/s 18.3305 KOps/s $\color{#d91a1a}-1.34\%$
test_step_mdp_speed[False-True-False-True-False] 89.8020μs 33.4960μs 29.8543 KOps/s 29.3272 KOps/s $\color{#35bf28}+1.80\%$
test_step_mdp_speed[False-True-False-False-True] 67.5120μs 35.4657μs 28.1962 KOps/s 28.2927 KOps/s $\color{#d91a1a}-0.34\%$
test_step_mdp_speed[False-True-False-False-False] 63.1220μs 21.4996μs 46.5125 KOps/s 46.0529 KOps/s $\color{#35bf28}+1.00\%$
test_step_mdp_speed[False-False-True-True-True] 96.0620μs 58.2866μs 17.1566 KOps/s 17.2686 KOps/s $\color{#d91a1a}-0.65\%$
test_step_mdp_speed[False-False-True-True-False] 66.0310μs 37.1102μs 26.9468 KOps/s 26.9600 KOps/s $\color{#d91a1a}-0.05\%$
test_step_mdp_speed[False-False-True-False-True] 89.1820μs 35.8194μs 27.9178 KOps/s 28.6537 KOps/s $\color{#d91a1a}-2.57\%$
test_step_mdp_speed[False-False-True-False-False] 50.4210μs 21.6544μs 46.1801 KOps/s 46.5775 KOps/s $\color{#d91a1a}-0.85\%$
test_step_mdp_speed[False-False-False-True-True] 91.4920μs 59.7543μs 16.7352 KOps/s 16.7266 KOps/s $\color{#35bf28}+0.05\%$
test_step_mdp_speed[False-False-False-True-False] 84.9010μs 39.6587μs 25.2151 KOps/s 25.1288 KOps/s $\color{#35bf28}+0.34\%$
test_step_mdp_speed[False-False-False-False-True] 69.1620μs 38.1868μs 26.1870 KOps/s 26.5058 KOps/s $\color{#d91a1a}-1.20\%$
test_step_mdp_speed[False-False-False-False-False] 47.8910μs 24.3358μs 41.0917 KOps/s 41.3255 KOps/s $\color{#d91a1a}-0.57\%$
test_non_tensor_env_rollout_speed[1000-single-True] 0.7641s 0.7619s 1.3125 Ops/s 1.2746 Ops/s $\color{#35bf28}+2.97\%$
test_non_tensor_env_rollout_speed[1000-single-False] 0.7356s 0.6430s 1.5551 Ops/s 1.5567 Ops/s $\color{#d91a1a}-0.10\%$
test_non_tensor_env_rollout_speed[1000-serial-no-buffers-True] 1.7943s 1.7154s 0.5830 Ops/s 0.5885 Ops/s $\color{#d91a1a}-0.95\%$
test_non_tensor_env_rollout_speed[1000-serial-no-buffers-False] 1.5641s 1.4888s 0.6717 Ops/s 0.6809 Ops/s $\color{#d91a1a}-1.35\%$
test_non_tensor_env_rollout_speed[1000-serial-buffers-True] 2.0494s 1.9797s 0.5051 Ops/s 0.5154 Ops/s $\color{#d91a1a}-2.00\%$
test_non_tensor_env_rollout_speed[1000-serial-buffers-False] 1.8222s 1.7482s 0.5720 Ops/s 0.5814 Ops/s $\color{#d91a1a}-1.61\%$
test_non_tensor_env_rollout_speed[1000-parallel-no-buffers-True] 4.9667s 4.7789s 0.2093 Ops/s 0.2156 Ops/s $\color{#d91a1a}-2.92\%$
test_non_tensor_env_rollout_speed[1000-parallel-no-buffers-False] 4.5272s 4.4710s 0.2237 Ops/s 0.2235 Ops/s $\color{#35bf28}+0.05\%$
test_non_tensor_env_rollout_speed[1000-parallel-buffers-True] 2.0469s 1.9634s 0.5093 Ops/s 0.4991 Ops/s $\color{#35bf28}+2.04\%$
test_non_tensor_env_rollout_speed[1000-parallel-buffers-False] 1.8123s 1.7040s 0.5868 Ops/s 0.5920 Ops/s $\color{#d91a1a}-0.87\%$
test_values[generalized_advantage_estimate-True-True] 11.8339ms 11.4935ms 87.0054 Ops/s 93.5412 Ops/s $\textbf{\color{#d91a1a}-6.99\%}$
test_values[vec_generalized_advantage_estimate-True-True] 19.9795ms 17.8154ms 56.1313 Ops/s 56.5254 Ops/s $\color{#d91a1a}-0.70\%$
test_values[td0_return_estimate-False-False] 0.2403ms 0.1322ms 7.5641 KOps/s 7.4372 KOps/s $\color{#35bf28}+1.71\%$
test_values[td1_return_estimate-False-False] 31.6020ms 31.2223ms 32.0284 Ops/s 33.7950 Ops/s $\textbf{\color{#d91a1a}-5.23\%}$
test_values[vec_td1_return_estimate-False-False] 18.5949ms 17.9089ms 55.8381 Ops/s 56.1035 Ops/s $\color{#d91a1a}-0.47\%$
test_values[td_lambda_return_estimate-True-False] 47.5940ms 46.4095ms 21.5473 Ops/s 22.6688 Ops/s $\color{#d91a1a}-4.95\%$
test_values[vec_td_lambda_return_estimate-True-False] 19.5035ms 17.9229ms 55.7945 Ops/s 56.5089 Ops/s $\color{#d91a1a}-1.26\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 10.5194ms 10.4338ms 95.8421 Ops/s 105.8189 Ops/s $\textbf{\color{#d91a1a}-9.43\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 1.7361ms 1.5490ms 645.5886 Ops/s 678.7062 Ops/s $\color{#d91a1a}-4.88\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.7532ms 0.4400ms 2.2730 KOps/s 2.2684 KOps/s $\color{#35bf28}+0.20\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 34.7649ms 34.5707ms 28.9262 Ops/s 28.5989 Ops/s $\color{#35bf28}+1.14\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 1.9092ms 1.7772ms 562.6681 Ops/s 577.6576 Ops/s $\color{#d91a1a}-2.59\%$
test_dqn_speed[False-None] 1.7874ms 1.4412ms 693.8855 Ops/s 700.2137 Ops/s $\color{#d91a1a}-0.90\%$
test_dqn_speed[False-backward] 2.3083ms 1.9708ms 507.3985 Ops/s 512.8849 Ops/s $\color{#d91a1a}-1.07\%$
test_dqn_speed[True-None] 0.7976ms 0.5494ms 1.8201 KOps/s 1.7594 KOps/s $\color{#35bf28}+3.45\%$
test_dqn_speed[True-backward] 1.0620ms 1.0196ms 980.7848 Ops/s 889.4819 Ops/s $\textbf{\color{#35bf28}+10.26\%}$
test_dqn_speed[reduce-overhead-None] 0.9011ms 0.5372ms 1.8614 KOps/s 1.7703 KOps/s $\textbf{\color{#35bf28}+5.14\%}$
test_ddpg_speed[False-None] 0.2011s 3.5545ms 281.3310 Ops/s 346.6818 Ops/s $\textbf{\color{#d91a1a}-18.85\%}$
test_ddpg_speed[False-backward] 4.3245ms 4.1785ms 239.3197 Ops/s 241.7055 Ops/s $\color{#d91a1a}-0.99\%$
test_ddpg_speed[True-None] 1.8180ms 1.4405ms 694.2192 Ops/s 701.0369 Ops/s $\color{#d91a1a}-0.97\%$
test_ddpg_speed[True-backward] 2.5194ms 2.4543ms 407.4478 Ops/s 382.3040 Ops/s $\textbf{\color{#35bf28}+6.58\%}$
test_ddpg_speed[reduce-overhead-None] 1.8031ms 1.4316ms 698.5217 Ops/s 701.1952 Ops/s $\color{#d91a1a}-0.38\%$
test_sac_speed[False-None] 9.0300ms 8.2536ms 121.1585 Ops/s 122.1525 Ops/s $\color{#d91a1a}-0.81\%$
test_sac_speed[False-backward] 11.9572ms 11.5292ms 86.7362 Ops/s 86.0901 Ops/s $\color{#35bf28}+0.75\%$
test_sac_speed[True-None] 2.5917ms 2.2035ms 453.8172 Ops/s 442.9531 Ops/s $\color{#35bf28}+2.45\%$
test_sac_speed[True-backward] 4.2083ms 4.1083ms 243.4122 Ops/s 236.0096 Ops/s $\color{#35bf28}+3.14\%$
test_sac_speed[reduce-overhead-None] 2.6260ms 2.1823ms 458.2416 Ops/s 446.2314 Ops/s $\color{#35bf28}+2.69\%$
test_redq_speed[False-None] 15.4652ms 10.8427ms 92.2280 Ops/s 94.4915 Ops/s $\color{#d91a1a}-2.40\%$
test_redq_speed[False-backward] 19.1643ms 18.2563ms 54.7756 Ops/s 56.3582 Ops/s $\color{#d91a1a}-2.81\%$
test_redq_speed[True-None] 4.8675ms 4.5182ms 221.3255 Ops/s 227.9170 Ops/s $\color{#d91a1a}-2.89\%$
test_redq_speed[True-backward] 10.1868ms 9.8685ms 101.3328 Ops/s 99.7165 Ops/s $\color{#35bf28}+1.62\%$
test_redq_speed[reduce-overhead-None] 4.9203ms 4.4844ms 222.9956 Ops/s 220.0864 Ops/s $\color{#35bf28}+1.32\%$
test_redq_deprec_speed[False-None] 11.8792ms 11.3151ms 88.3775 Ops/s 88.8342 Ops/s $\color{#d91a1a}-0.51\%$
test_redq_deprec_speed[False-backward] 17.4086ms 16.2322ms 61.6059 Ops/s 62.0047 Ops/s $\color{#d91a1a}-0.64\%$
test_redq_deprec_speed[True-None] 4.0276ms 3.7812ms 264.4662 Ops/s 266.9431 Ops/s $\color{#d91a1a}-0.93\%$
test_redq_deprec_speed[True-backward] 8.0133ms 7.7218ms 129.5030 Ops/s 127.9252 Ops/s $\color{#35bf28}+1.23\%$
test_redq_deprec_speed[reduce-overhead-None] 4.0930ms 3.7181ms 268.9515 Ops/s 265.6008 Ops/s $\color{#35bf28}+1.26\%$
test_td3_speed[False-None] 8.4097ms 8.2773ms 120.8126 Ops/s 122.5465 Ops/s $\color{#d91a1a}-1.41\%$
test_td3_speed[False-backward] 11.6552ms 11.2120ms 89.1899 Ops/s 90.4823 Ops/s $\color{#d91a1a}-1.43\%$
test_td3_speed[True-None] 1.9437ms 1.8857ms 530.3211 Ops/s 519.5591 Ops/s $\color{#35bf28}+2.07\%$
test_td3_speed[True-backward] 3.8699ms 3.7430ms 267.1632 Ops/s 269.6074 Ops/s $\color{#d91a1a}-0.91\%$
test_td3_speed[reduce-overhead-None] 1.8683ms 1.8352ms 544.9120 Ops/s 539.6428 Ops/s $\color{#35bf28}+0.98\%$
test_cql_speed[False-None] 31.4244ms 26.8968ms 37.1791 Ops/s 38.2002 Ops/s $\color{#d91a1a}-2.67\%$
test_cql_speed[False-backward] 39.9347ms 36.3246ms 27.5296 Ops/s 28.2302 Ops/s $\color{#d91a1a}-2.48\%$
test_cql_speed[True-None] 12.8047ms 12.3660ms 80.8671 Ops/s 80.6485 Ops/s $\color{#35bf28}+0.27\%$
test_cql_speed[True-backward] 18.8007ms 18.5253ms 53.9802 Ops/s 52.5995 Ops/s $\color{#35bf28}+2.62\%$
test_cql_speed[reduce-overhead-None] 15.7257ms 12.6648ms 78.9590 Ops/s 77.9687 Ops/s $\color{#35bf28}+1.27\%$
test_a2c_speed[False-None] 5.7304ms 5.4374ms 183.9111 Ops/s 185.0585 Ops/s $\color{#d91a1a}-0.62\%$
test_a2c_speed[False-backward] 12.3755ms 11.9487ms 83.6912 Ops/s 83.9828 Ops/s $\color{#d91a1a}-0.35\%$
test_a2c_speed[True-None] 4.0293ms 3.7714ms 265.1540 Ops/s 268.0954 Ops/s $\color{#d91a1a}-1.10\%$
test_a2c_speed[True-backward] 9.0277ms 8.7999ms 113.6382 Ops/s 114.8261 Ops/s $\color{#d91a1a}-1.03\%$
test_a2c_speed[reduce-overhead-None] 4.5284ms 3.7420ms 267.2397 Ops/s 263.7929 Ops/s $\color{#35bf28}+1.31\%$
test_ppo_speed[False-None] 6.4731ms 6.0890ms 164.2299 Ops/s 167.5200 Ops/s $\color{#d91a1a}-1.96\%$
test_ppo_speed[False-backward] 13.4447ms 12.8333ms 77.9222 Ops/s 78.3427 Ops/s $\color{#d91a1a}-0.54\%$
test_ppo_speed[True-None] 4.7149ms 3.6667ms 272.7215 Ops/s 271.7547 Ops/s $\color{#35bf28}+0.36\%$
test_ppo_speed[True-backward] 8.7329ms 8.5286ms 117.2526 Ops/s 114.5024 Ops/s $\color{#35bf28}+2.40\%$
test_ppo_speed[reduce-overhead-None] 3.7718ms 3.6082ms 277.1460 Ops/s 294.9494 Ops/s $\textbf{\color{#d91a1a}-6.04\%}$
test_reinforce_speed[False-None] 4.8396ms 4.5837ms 218.1666 Ops/s 228.2628 Ops/s $\color{#d91a1a}-4.42\%$
test_reinforce_speed[False-backward] 7.6565ms 7.4628ms 133.9978 Ops/s 135.8397 Ops/s $\color{#d91a1a}-1.36\%$
test_reinforce_speed[True-None] 3.7263ms 2.9531ms 338.6304 Ops/s 373.0998 Ops/s $\textbf{\color{#d91a1a}-9.24\%}$
test_reinforce_speed[True-backward] 8.1751ms 7.9123ms 126.3853 Ops/s 133.6620 Ops/s $\textbf{\color{#d91a1a}-5.44\%}$
test_reinforce_speed[reduce-overhead-None] 3.3188ms 2.9062ms 344.0894 Ops/s 376.9169 Ops/s $\textbf{\color{#d91a1a}-8.71\%}$
test_iql_speed[False-None] 23.4425ms 20.2544ms 49.3719 Ops/s 51.8607 Ops/s $\color{#d91a1a}-4.80\%$
test_iql_speed[False-backward] 31.6068ms 30.8798ms 32.3836 Ops/s 33.3651 Ops/s $\color{#d91a1a}-2.94\%$
test_iql_speed[True-None] 9.1551ms 8.5876ms 116.4470 Ops/s 122.2275 Ops/s $\color{#d91a1a}-4.73\%$
test_iql_speed[True-backward] 17.2623ms 16.8152ms 59.4699 Ops/s 63.7395 Ops/s $\textbf{\color{#d91a1a}-6.70\%}$
test_iql_speed[reduce-overhead-None] 9.0374ms 8.6095ms 116.1507 Ops/s 124.8511 Ops/s $\textbf{\color{#d91a1a}-6.97\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 6.3423ms 6.1184ms 163.4405 Ops/s 162.1200 Ops/s $\color{#35bf28}+0.81\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2.2473ms 0.3595ms 2.7817 KOps/s 2.7749 KOps/s $\color{#35bf28}+0.24\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6779ms 0.4167ms 2.3996 KOps/s 2.7374 KOps/s $\textbf{\color{#d91a1a}-12.34\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.1321ms 5.9082ms 169.2553 Ops/s 168.5526 Ops/s $\color{#35bf28}+0.42\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 2.3209ms 0.4252ms 2.3519 KOps/s 2.7868 KOps/s $\textbf{\color{#d91a1a}-15.60\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6098ms 0.4080ms 2.4510 KOps/s 2.6777 KOps/s $\textbf{\color{#d91a1a}-8.47\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 1.7779ms 1.5019ms 665.8134 Ops/s 665.8500 Ops/s $-0.01\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 1.7903ms 1.4255ms 701.5060 Ops/s 708.8661 Ops/s $\color{#d91a1a}-1.04\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 9.4885ms 6.2101ms 161.0284 Ops/s 163.5923 Ops/s $\color{#d91a1a}-1.57\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.8207ms 0.5058ms 1.9772 KOps/s 2.1569 KOps/s $\textbf{\color{#d91a1a}-8.33\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.6406ms 0.4517ms 2.2140 KOps/s 2.1788 KOps/s $\color{#35bf28}+1.61\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 6.0723ms 5.9563ms 167.8902 Ops/s 166.6730 Ops/s $\color{#35bf28}+0.73\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2.0937ms 0.3391ms 2.9494 KOps/s 2.8412 KOps/s $\color{#35bf28}+3.81\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.5672ms 0.3760ms 2.6598 KOps/s 3.0835 KOps/s $\textbf{\color{#d91a1a}-13.74\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.1717ms 5.9022ms 169.4297 Ops/s 167.9968 Ops/s $\color{#35bf28}+0.85\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 2.2276ms 0.2907ms 3.4396 KOps/s 3.5399 KOps/s $\color{#d91a1a}-2.83\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6284ms 0.2713ms 3.6861 KOps/s 2.4498 KOps/s $\textbf{\color{#35bf28}+50.46\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 6.2375ms 6.1121ms 163.6087 Ops/s 164.2291 Ops/s $\color{#d91a1a}-0.38\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.1918ms 0.5061ms 1.9759 KOps/s 2.0496 KOps/s $\color{#d91a1a}-3.59\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.6998ms 0.4772ms 2.0956 KOps/s 2.1719 KOps/s $\color{#d91a1a}-3.51\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 6.6678ms 5.1993ms 192.3319 Ops/s 50.5925 Ops/s $\textbf{\color{#35bf28}+280.16\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 10.9237ms 1.8929ms 528.2786 Ops/s 545.6110 Ops/s $\color{#d91a1a}-3.18\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 1.1014ms 0.8989ms 1.1125 KOps/s 929.9040 Ops/s $\textbf{\color{#35bf28}+19.63\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 6.8023ms 5.1860ms 192.8281 Ops/s 191.1438 Ops/s $\color{#35bf28}+0.88\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 3.9159ms 1.7927ms 557.8080 Ops/s 560.7438 Ops/s $\color{#d91a1a}-0.52\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 6.4265ms 1.2270ms 815.0030 Ops/s 763.8170 Ops/s $\textbf{\color{#35bf28}+6.70\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.5478s 16.2291ms 61.6179 Ops/s 58.0089 Ops/s $\textbf{\color{#35bf28}+6.22\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 4.1508ms 1.9783ms 505.4849 Ops/s 527.5452 Ops/s $\color{#d91a1a}-4.18\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 2.5297ms 1.1256ms 888.3866 Ops/s 959.7763 Ops/s $\textbf{\color{#d91a1a}-7.44\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] 39.4080ms 37.0057ms 27.0229 Ops/s 27.2050 Ops/s $\color{#d91a1a}-0.67\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] 20.6244ms 18.8268ms 53.1158 Ops/s 55.1341 Ops/s $\color{#d91a1a}-3.66\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] 42.1929ms 38.0056ms 26.3119 Ops/s 26.7791 Ops/s $\color{#d91a1a}-1.74\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] 20.6111ms 19.0211ms 52.5731 Ops/s 54.6901 Ops/s $\color{#d91a1a}-3.87\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 41.1948ms 39.5799ms 25.2653 Ops/s 25.2580 Ops/s $\color{#35bf28}+0.03\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False] 21.7429ms 20.5137ms 48.7479 Ops/s 50.3334 Ops/s $\color{#d91a1a}-3.15\%$

vmoens added a commit that referenced this pull request Jan 29, 2026
@vmoens
[CI] Add flaky test tracking system (#3408)
975065d
@vmoens vmoens mentioned this pull request Jan 29, 2026
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CI Has to do with CI setup (e.g. wheels & builds, tests...) CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@vmoens