Skip to content

Fix flaky randomString bias test - #993

Merged
theoephraim merged 1 commit into
mainfrom
fix-flaky-randomstring-bias-test
Aug 11, 2026
Merged

Fix flaky randomString bias test#993
theoephraim merged 1 commit into
mainfrom
fix-flaky-randomstring-bias-test

Conversation

@theoephraim

Copy link
Copy Markdown
Member

The unbiased-output test for randomString() checked every charset char against a +/-30% bound. At n=10000 that bound sits at ~3.8 sigma, and with 62 chars checked on both tails it fails by pure chance roughly once every 130 runs (seen in CI: 210 vs 209.68). The old assertion also could not catch the modulo bias it guards against: a naive byte % 62 puts over-represented chars at ~195 counts, under the old 209.7 ceiling.

Replaced it with an aggregate count of the first 8 charset chars, exactly the ones modulo bias inflates. Unbiased sampling lands around 1290 (stddev ~33.5), biased around 1562, so the 1425 threshold is ~4 sigma from both: false failures drop to ~1 in 30k runs while a simulated biased sampler trips it in 20/20 trials. Kept a loose sanity check that every charset char appears at least once.

No resolver changes; the implementation's rejection sampling was already correct.

@github-actions

Copy link
Copy Markdown
Contributor

bumpy-frog

The changes in this PR will be included in the next version bump.

Bump files in this PR

Click here if you want to add another bump file to this PR


This comment is maintained by bumpy.

@github-actions

Copy link
Copy Markdown
Contributor

📦 Bundle size

no change to bundle size

Metric main This PR Δ
Total dist 5010.6 KB 5010.6 KB
JS 1717.9 KB 1717.9 KB
Sourcemaps 3216.2 KB 3216.2 KB
Type defs 76.6 KB 76.6 KB

dist/ only; native binaries are versioned separately and not counted here.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes across the test-only update to randomString() bias coverage and its release metadata.

  • Targeted bias assertion: Replaces noisy per-character bounds with an aggregate check over the eight default-charset indices that naive modulo sampling overrepresents.
  • Sanity coverage: Retains a low-risk assertion that every default-charset character appears in the 10,000-character sample.
  • Release metadata: Records the test-only change as a no-bump entry for varlock.

Pullfrog  | View workflow run | Using azure/gpt-5.6-sol𝕏

@theoephraim
theoephraim merged commit dfcfccf into main Aug 11, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant