Skip to content

feat: Add benchmark chart toggles for y-axis starting at 0.0, outlier clipping#875

Merged
RobertJacobsonCDC merged 2 commits into
mainfrom
RobertJacobsonCDC_benchmark_chart_tweaks
May 6, 2026
Merged

feat: Add benchmark chart toggles for y-axis starting at 0.0, outlier clipping#875
RobertJacobsonCDC merged 2 commits into
mainfrom
RobertJacobsonCDC_benchmark_chart_tweaks

Conversation

@RobertJacobsonCDC
Copy link
Copy Markdown
Collaborator

Added two quality of life toggles to the benchmark history page:

  1. start the y-axis at 0.0, making absolute benchmark scale easier to compare across runs. Turning it off preserves Chart.js’s existing automatic axis behavior.
  2. MAD clipping heuristic for extreme high outliers in time-series charts.

Both are toggled on by default.

@RobertJacobsonCDC RobertJacobsonCDC requested a review from cdc-as81 May 5, 2026 03:57
github-actions Bot added a commit that referenced this pull request May 5, 2026
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 5, 2026

Benchmark Results

Hyperfine

Command Mean [ms] Min [ms] Max [ms] Relative
large_sir::baseline 2.3 ± 0.0 2.2 2.5 1.00
large_sir::entities 4.6 ± 0.1 4.5 4.9 1.98 ± 0.05

Criterion

Regressions (slower)
Group Bench Param Change CI Lower CI Upper
algorithm_benches algorithm_sampling_multiple_known_length 6.039% 5.630% 6.442%
algorithm_benches algorithm_sampling_multiple_l_reservoir 4.586% 4.136% 5.030%
large_dataset bench_query_population_derived_property_entities 3.263% 2.928% 3.588%
indexing query_people_multiple_individually_indexed_properties_entities 3.213% 2.608% 3.698%
sampling sampling_multiple_l_reservoir_entities 1.994% 1.904% 2.088%
counts multi_property_indexed_entities 1.339% 1.140% 1.525%
Improvements (faster)
Group Bench Param Change CI Lower CI Upper
sample_entity sample_entity_single_property_indexed 1000 -14.955% -15.536% -14.299%
large_dataset bench_query_population_multi_unindexed_entities -12.446% -13.677% -11.025%
sample_entity sample_entity_single_property_indexed 100000 -12.368% -12.585% -12.150%
sample_entity sample_entity_single_property_indexed 10000 -11.017% -11.202% -10.853%
sample_entity sample_entity_multi_property_indexed 100000 -10.646% -10.917% -10.454%
sample_entity sample_entity_multi_property_indexed 10000 -10.429% -10.729% -10.113%
sample_entity sample_entity_whole_population 1000 -10.227% -12.791% -8.105%
sample_entity sample_entity_multi_property_indexed 1000 -9.264% -9.523% -8.987%
large_dataset bench_filter_unindexed_entity -5.058% -9.225% -1.054%
indexing with_query_results_single_indexed_property_entities -2.936% -3.526% -2.426%
counts single_property_indexed_entities -2.399% -2.748% -1.885%
large_dataset bench_query_population_multi_indexed_entities -1.924% -2.284% -1.528%
Unchanged / inconclusive (CI crosses 0%)
Group Bench Param Change CI Lower CI Upper
examples example-basic-infection -2.363% -6.869% 4.611%
large_dataset bench_match_entity 1.188% 0.063% 2.303%
indexing query_people_count_multiple_individually_indexed_properties_enti 1.008% 0.637% 1.313%
algorithm_benches algorithm_sampling_single_l_reservoir 0.772% 0.594% 1.007%
indexing with_query_results_indexed_multi-property_entities -0.758% -1.276% -0.172%
sampling sampling_single_l_reservoir_entities 0.564% 0.159% 1.134%
sampling sampling_single_known_length_entities 0.554% -0.206% 1.145%
sample_entity sample_entity_single_property_unindexed 1000 -0.540% -1.158% -0.051%
indexing query_people_count_indexed_multi-property_entities -0.530% -0.869% -0.140%
sample_entity sample_entity_whole_population 100000 0.516% 0.209% 0.892%
indexing query_people_count_single_indexed_property_entities 0.487% -0.212% 1.974%
sampling count_and_sampling_single_known_length_entities 0.468% -0.243% 1.039%
examples example-births-deaths 0.424% -0.004% 0.874%
algorithm_benches algorithm_sampling_single_rand_reservoir 0.398% -0.228% 1.120%
counts index_after_adding_entities 0.386% 0.061% 0.734%
large_dataset bench_query_population_property_entities 0.348% -0.157% 1.057%
algorithm_benches algorithm_sampling_single_known_length -0.306% -0.788% 0.194%
sample_entity sample_entity_single_property_unindexed 100000 -0.242% -0.802% 0.297%
sampling sampling_multiple_known_length_entities 0.233% -0.349% 0.791%
sample_entity sample_entity_whole_population 10000 -0.231% -0.572% 0.094%
counts single_property_unindexed_entities -0.188% -0.762% 0.420%
counts concrete_plus_derived_unindexed_entities -0.156% -0.451% 0.138%
indexing with_query_results_multiple_individually_indexed_properties_enti 0.137% -0.346% 0.509%
indexing query_people_single_indexed_property_entities 0.112% 0.026% 0.202%
sampling count_and_sampling_single_unindexed_concrete_plus_derived_entiti -0.111% -0.180% -0.064%
indexing query_people_indexed_multi-property_entities -0.110% -0.285% 0.051%
counts multi_property_unindexed_entities 0.109% -0.249% 0.546%
sampling sampling_single_unindexed_concrete_plus_derived_entities -0.076% -0.137% -0.031%
large_dataset bench_filter_indexed_entity 0.070% -11.826% 13.751%
sample_entity sample_entity_single_property_unindexed 10000 -0.052% -0.429% 0.207%
sampling sampling_multiple_unindexed_entities -0.040% -0.166% 0.076%
counts reindex_after_adding_more_entities 0.039% -0.431% 0.431%
sampling sampling_single_unindexed_entities 0.034% -0.069% 0.135%
large_dataset bench_query_population_indexed_property_entities -0.003% -0.371% 0.493%

Copy link
Copy Markdown
Collaborator

@cdc-as81 cdc-as81 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@RobertJacobsonCDC RobertJacobsonCDC merged commit 24dc123 into main May 6, 2026
22 checks passed
@RobertJacobsonCDC RobertJacobsonCDC deleted the RobertJacobsonCDC_benchmark_chart_tweaks branch May 6, 2026 12:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants