Skip to content

Conversation

@cscheid
Copy link
Collaborator

@cscheid cscheid commented Jan 21, 2025

With some A/B testing, we find evidence that the code in the PR is faster than that of main (I can share the script later)

$ N_TRIALS=3000 QUARTO_NO_TYPECHECK=1 ./run_experiment_2.py coalesce-raw.csv main tmp-2025-01-21-coalesce-raw quarto render --quiet docs/references/formats/presentations/revealjs.qmd
┏━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┓
┃ Variant ┃ Count ┃ Mean    ┃ Std Dev   ┃
┡━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━┩
│ A       │ 10    │ 2.51012 │ 0.0106346 │
│ B       │ 10    │ 2.49222 │ 0.0101653 │
└─────────┴───────┴─────────┴───────────┘
{'outcome': 'significant', 'winner': 'B', 'p_value': np.float64(0.0018407964064409569), 'total_samples': 20, 'alpha_spent': 0.05, 'statistics': {'counts': {'A': 10, 'B': 10}, 'means': {'A': 2.510115218162537, 'B': 2.4922211170196533}, 'variances': {'A': 0.00011309452203267512, 'B': 0.00010333396089645674}}}
Statistical Summary:
===================
Variant     Count         Mean      Std Dev
-------- -------- ------------ ------------
A              10     2.510115     0.010635
B              10     2.492221     0.010165

I used docs/references/formats/presentations/revealjs.qmd from quarto-web because I measured that file to be the one to take the longest time on coalesce_raw in a site render.

@cscheid cscheid merged commit 634916c into main Jan 22, 2025
47 checks passed
@cscheid cscheid deleted the tmp-2025-01-21-coalesce-raw branch January 22, 2025 01:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants