Add more random generator test coverage and use elm-pages for random generator#134
Merged
mdgriffith merged 16 commits intoApr 3, 2026
Merged
Conversation
The generator can now be run via elm-pages:
cd test/property-test/script
elm-pages run src/GenerateProgram.elm -- --seed 42 --count 10
With coverage instrumentation:
elm-pages run --coverage src/GenerateProgram.elm -- --seed 42 --count 10
This produces an LCOV report showing which elm-codegen code paths
the generated programs exercise.
Note: requires elm-pages 3.3.3+ (global) and the local elm-pages
Elm package (12.1.1, not yet published). The Platform.worker
approach (generate.mjs) still works as a fallback.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Guided by elm-pages --coverage output, added generators that exercise: - Elm.Let.unpack with tuple destructuring (+9% Let coverage) - Elm.fn2/fn3 expression-level lambdas - Elm.expose and Elm.exposeConstructor - Elm.withDocumentation - Elm.comment Key coverage improvements: Elm.Let: 32.7% -> 41.8% Internal.Compiler: 38.6% -> 42.6% Elm: 42.9% -> 45.8% Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ybe/result, Elm.group/docs, Elm.apply with annotations. Coverage improvements: TOTAL: 48.4% -> 51.0% Elm.Annotation: 51.6% -> 62.5% (+10.9%) Elm.Let: 41.8% -> 56.4% (+14.6%) Internal.Render: 60.0% -> 76.8% (+16.8%) Internal.Arg: 45.0% -> 53.5% (+8.5%) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Coverage improvements: Elm.Declare: 19.3% -> 33.7% (+14.4%) TOTAL: 48.4% -> 50.9% Note: elm-pages --coverage stack overflows with count=20 programs (printCoverageReportSync in coverage.js). Works fine with count=10. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Targets Internal.Compiler coverage gaps: inferRecordField, getField, unifiableFields, isAppendable, and Elm.Arg.varWith. Coverage improvements: TOTAL: 52.7% -> 53.5% Internal.Compiler: 41.7% -> 43.5% Internal.Arg: 53.5% -> 57.4% Elm.Arg: 50.0% -> 55.6% No new bugs surfaced — all failures remain the known operator indentation (finding mdgriffith#3) and comparable leak (finding mdgriffith#4) issues. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…Record. Found 2 new bugs (finding mdgriffith#5): - Elm.functionReduced with record accessor (\r -> r.name) generates type annotation `r -> name` instead of `{ b | name : a } -> a` - Elm.functionReduced with arithmetic (\x -> x + 1) generates type annotation `x -> Int` instead of `Int -> Int` Both are betaReduce annotation bugs — the expression is correctly reduced but the type annotation doesn't reflect the body's constraints. Coverage improvements: TOTAL: 53.5% -> 57.4% Elm: 47.0% -> 57.6% (+10.6%) Internal.Compiler: 43.5% -> 48.9% (+5.4%) Internal.Clean: 84.6% -> 92.3% Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ors. Found 1 new bug (finding mdgriffith#6): - Elm.Arg.aliasAs generates fresh type variable for the alias instead of the actual record type from the underlying pattern. Coverage improvements: TOTAL: 57.4% -> 60.9% Elm.Declare: 33.7% -> 54.2% (+20.5%) Elm.Arg: 55.6% -> 77.8% (+22.2%) Internal.Arg: 57.4% -> 69.8% (+12.4%) Internal.Write: 66.1% -> 71.0% 6 bugs total found by property testing: mdgriffith#1 Number type variable collision (FIXED) mdgriffith#2 Pipe + lambda parens (FIXED) mdgriffith#3 Operator multi-line indentation mdgriffith#4 Comparable type variable leak mdgriffith#5 functionReduced wrong type annotations mdgriffith#6 aliasAs wrong type for alias variable Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Found 2 more bugs: - Finding mdgriffith#5 update: Elm.unwrapper also generates wrong type annotations (val -> unwrapped instead of Wrapper -> Int) - Finding mdgriffith#7: Elm.Op.pipeLeft with lambda absorbs <| into lambda body (\x -> x <| "hello" instead of (\x -> x) <| "hello") Coverage: 60.9% -> 62.5% Elm: 60.5% -> 64.6% Elm.Declare: 54.2% -> 63.9% (+9.7%) Elm.Op: 75.6% -> 80.5% Total bugs found: 7 mdgriffith#1 Number type variable collision (FIXED) mdgriffith#2 Pipe + lambda parens (FIXED) mdgriffith#3 Operator multi-line indentation mdgriffith#4 Comparable type variable leak mdgriffith#5 functionReduced/unwrapper wrong type annotations mdgriffith#6 aliasAs wrong type for alias variable mdgriffith#7 pipeLeft lambda not parenthesized Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Found 1 new class of bugs (finding mdgriffith#8): - 19 missing type annotations across 6 different API patterns (Let.unpack, Let.fn, unwrap, Declare.value, Elm.get through alias) - elm-codegen's type inference silently fails for these patterns, producing declarations without type annotations Coverage: 62.5% -> 63.2% Total bugs found: 8 mdgriffith#1 Number type variable collision (FIXED) mdgriffith#2 Pipe + lambda parens (FIXED) mdgriffith#3 Operator multi-line indentation mdgriffith#4 Comparable type variable leak mdgriffith#5 functionReduced/unwrapper wrong type annotations mdgriffith#6 aliasAs wrong type for alias variable mdgriffith#7 pipeLeft lambda not parenthesized mdgriffith#8 Missing type annotations for multiple APIs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ators. Confirmed that Elm.Op.pipeTo hits the same lambda parens bug as finding mdgriffith#2/mdgriffith#7 (internally calls functionReduced). Extensible record annotations (Type.extensible) generate correctly — `{ a | name : String }` works for both aliases and function args. List pattern matching (Elm.Arg.list, items, listRemaining) generates correct code with proper head :: tail and [first, second] patterns. Note: elm-pages --coverage stopped working after recent changes (Coverage.track not found in compiled output). Filed as DX bug. Coverage at last measurement: 63.2% total. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Elm.parse successfully parses raw Elm source and includes declarations in generated files. variant3 (3-arg constructor) and fn5 (5-arg function) both generate correctly. Coverage: 66.1% -> 67.0% Elm: 66.0% -> 70.0% (+4.0%, from parse + unsafe) Elm.Declare: 68.7% -> 73.5% (+4.8%, from variant3 + fn5) No new bugs from this round — these code paths are solid. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove generate.mjs (old Node + vm approach) - Update run.sh to use elm-pages run - Upgrade elm-pages to 12.1.2 (published with --coverage support) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
dillonkearns
changed the title
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hey @mdgriffith! This is just a continuation of the random program generator bug finder. This PR doubles the code coverage, and it also uses
elm-pagesto run so we can run the script with a coverage report.