Skip to content

lowering: capture check by free variables, and say when it rejects - #41

Merged
simontreanor merged 1 commit into
mainfrom
fix/tail-call-capture-precision
Jul 31, 2026
Merged

lowering: capture check by free variables, and say when it rejects#41
simontreanor merged 1 commit into
mainfrom
fix/tail-call-capture-precision

Conversation

@simontreanor

Copy link
Copy Markdown
Owner

Follow-up to #39, the option you picked: free variables plus a note.

The capture precondition compared every name mentioned inside a nested function against the names the frame binds. Two safe shapes were rejected by that; one is now fixed and the other explains itself.

Fixed — shadowing is not capture. List.map (fun n -> n + 1) xs inside a function whose parameter is also n used to block the loop, though the lambda binds its own n and shares nothing. The check now collects true free variables: subtract the nested function's parameters, locals and match captures, and add back anything it declares nonlocal or global, since those name the enclosing cell by definition and are captures even though the body assigns them.

Still rejected — capture that cannot escape. List.fold (fun acc x -> acc + x * n) 0 xs captures n but is consumed within the iteration. Proving that needs per-callee escape analysis, which is not worth the machinery.

And the silence is gone. A rejection previously left the function recursing with nothing said, so a program could keep hitting the recursion limit with only the emitted Python to find out from:

note: `collect` calls itself in tail position but keeps its recursive form: a closure in it captures `n`
note: `shadowed` calls itself in tail position but keeps its recursive form: the body rebinds `shadowed`

has_self_tail_call runs before any precondition, so a function with no self tail call is never mentioned — which is almost all of them. There are tests for both halves of that: a rejected call notes, a successful one and an ordinary function say nothing.

Plumbing: notes ride out through lowering::lower_collecting / LoweredModule.noteslib::compile_collecting / project::CompiledProject.notes → the CLI's report_notes, printed to stderr by compile and run. lower and compile keep their existing signatures by delegating and dropping the notes, so no caller had to change.

Tests: 4 new in tests/compile.rs (shadowing no longer blocks; a rejection notes with the reason; a successful rewrite is silent; an ordinary function is silent), on top of #39's guard tests which all still pass. Full suite, clippy and fmt clean.

The self-tail-call capture precondition compared *every name mentioned*
inside a nested function against the names the frame binds, which rejected
two shapes that are in fact safe. One is now fixed and the other is
explained.

Fixed: a nested function that binds the name itself shares nothing with us,
so `List.map (fun n -> n + 1) xs` inside a function whose parameter is also
`n` no longer blocks the loop. The check now collects true free variables,
subtracting the nested function's own parameters, locals and match
captures, and adding back anything it declares `nonlocal` or `global`,
since those name the enclosing cell by definition and so are captures even
though the body assigns them.

Still rejected: a closure that genuinely captures but cannot escape, such
as one consumed by `List.fold` within the iteration. Proving that needs
escape analysis per callee, which is not worth the machinery for the
payoff.

The bigger cost was silence. A rejection left the function recursing with
nothing said, so a program could keep hitting the recursion limit with only
the emitted Python to find out from. A function that *does* call itself in
tail position but kept its recursive form now emits a note naming the
reason; `has_self_tail_call` runs first, so the overwhelming majority of
functions, which have no self tail call at all, are never mentioned.

Notes ride out through `lower_collecting` / `LoweredModule.notes` to
`compile_collecting` / `CompiledProject.notes` and are printed by the CLI
for `compile` and `run`. `lower` and `compile` keep their signatures by
delegating and dropping the notes.
@simontreanor
simontreanor merged commit 98e044d into main Jul 31, 2026
11 checks passed
@simontreanor
simontreanor deleted the fix/tail-call-capture-precision branch July 31, 2026 13:23
@simontreanor simontreanor mentioned this pull request Jul 31, 2026
simontreanor added a commit that referenced this pull request Jul 31, 2026
Two dogfooding reports from real programs, and the standard-library sweep
they triggered.

Language:

* a `type` declaration can name an imported type, bare or module-qualified
  (#36) — the one gap that changed a program's architecture rather than its
  phrasing, forcing two modules into one file
* field access resolves from the base's type when it is known, so two
  records may share a field name without prefixes (#37)
* parameters destructure: tuples (#38), records (#40), and `_`
* a direct self tail call lowers to a loop, so an interactive turn loop no
  longer walks the stack (#39, #41)

Standard library — about 115 new members, taking every module to the F#
core set: List (#42), Seq (#44), Set and Map (#46), String (#47), Option
and Result (#48), then a member-by-member FSharp.Core audit (#51). Every
built-in member now carries a one-line description and its complexity in
hover and completion (#43, #49), enforced by tests.

Fixes:

* `pyfun run` on a single file gives the program its own stdin, so an
  interactive program is runnable by the command whose job is running
  programs (#35)
* a partially applied lambda closes over its argument instead of being
  wrapped, so `List.map ((+) 2)` emits `lambda b: 2 + b` (#52)
* every multi-argument callback's scheme put the effect variable on the
  wrong arrows, so `List.fold` could never accept an effectful folder (#51)
* `Seq.empty` lowered to a bare `iter()`, a TypeError (#51)

One source-incompatible change, which is why this is 0.4.0 and not 0.3.1:
a dotted `extern` target whose module prefix cannot be decided from the
text is now a compile error naming the `extern import` to add (#50).
`sys.stdout.flush` used to emit `import sys.stdout` and fail at runtime;
declaring `extern import sys` fixes it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant