Skip to content

Conversation

@cscheid
Copy link
Collaborator

@cscheid cscheid commented Jan 23, 2025

It seems hard to avoid the overhead of coalesce_raw, but we can write by hand a filter that combines its behavior with one of its neighbors on the list. This appears to work well on A/B testing.

$ run_experiment.py main 2025-01-23-finalize-combined deno run --allow-all ~/repos/github/quarto-dev/quarto-cli/tools/run-pandoc-capture.ts render-command.json
┏━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Variant ┃ Count ┃ Mean     ┃ Std Dev    ┃
┡━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━┩
│ A       │ 10    │ 0.591234 │ 0.00634839 │
│ B       │ 9     │ 0.534588 │ 0.00587546 │
└─────────┴───────┴──────────┴────────────┘
{'outcome': 'significant', 'winner': 'B', 'p_value': np.float64(5.109983792608639e-13), 'total_samples': 20, 'alpha_spent': 0.05, 'statistics': {'counts': {'A': 10, 'B': 10}, 'means': {'A': 0.5912335634231567, 'B': 0.5357627630233764}, 'variances': {'A': 4.030205561839084e-05, 'B': 4.3499259748558394e-05}}}
Statistical Summary:
===================
Variant     Count         Mean      Std Dev
-------- -------- ------------ ------------
A              10     0.591234     0.006348
B              10     0.535763     0.006595

@cscheid
Copy link
Collaborator Author

cscheid commented Jan 23, 2025

So this one is kind of a bigger win - 10% of the entire Pandoc runtime on this document.

@cscheid cscheid merged commit 53da9da into main Jan 23, 2025
47 checks passed
@cscheid cscheid deleted the perf/2023-01-25-finalize-combined branch January 23, 2025 22:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants