Skip to content

Replace runtime-type quotation with splice-into-source typechecking#706

Merged
eb8680 merged 15 commits into
masterfrom
dn-696-splice-typecheck
Jul 11, 2026
Merged

Replace runtime-type quotation with splice-into-source typechecking#706
eb8680 merged 15 commits into
masterfrom
dn-696-splice-typecheck

Conversation

@datvo06

@datvo06 datvo06 commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Closes #696. Addresses #577, #576.

Currently, for synthesis, we are reconstructing a Template's lexical context as synthetic type stubs. This is brittle and likely unsound.

This PR splices the generated code into the Template module, and type checks the whole file. This come with a few changes

  1. mypy_type_check now parameterized with generated (code) and anchor (the Template). The anchor is used to recover its module source,

Note, in practice there is a few more tweaks:

  • Mypy runs on the whole module, but only diagnostics inside the spliced span are raised, so an unrelated pre-existing error elsewhere in the module can't fail synthesis.
  • Non-nestable constructs (from x import *, __future__) are caught by an explicit AST pre-scan, since mypy silently accepts a nested star-import; a fatal mypy error (exit ≥ 2) is surfaced rather than silently passed; unrecoverable source (no file, no linecache) is skipped with a logged reason.
  1. Name collisions (Synthesized Callable Name shouldn't matter #542) are now handled structurally: the generated def is a function-body local that shadows a colliding context name rather than redefining it at module level, so the AST-rename pass is gone. The whole quotation layer is deleted — type_to_ast, signature_to_ast, collect_variable_declarations, collect_runtime_type_stubs, collect_imports, and _RenameTransformer/_generate_unique_name (net +510 / −1991).

  2. Behavior change. Runtime-only / local types, and names with no lexical presence at the splice point, now raise instead of being (partially, fakily) stubbed. The check is sound on annotated generated code; unannotated bodies are checked at their signature only (--check-untyped-defs is deliberately off — it rejects runnable heterogeneous-container code).

Tests.

  1. Deleted the quotation-internals unit tests; kept the security/REPL tests
  2. Added behavior tests asserting raise-or-not (not mypy text): Synthesized Callable Name shouldn't matter #542 collision shadowing, the gate (unrelated module error doesn't block a correct synthesis), closures (enclosing locals in scope), method/static/class templates, Enum + dataclass context (Decoding runtime-only Pydantic models in Callables fails #576), injected-name, non-nestable, linecache-backed recovery (the REPL tool code should be typechecked when possible #690 path), annotation-collision, and end-to-end decode through Encodable[Callable]. mypy effectful/handlers/llm/ clean, ruff clean, full LLM suite green.

@datvo06
datvo06 force-pushed the dn-696-splice-typecheck branch 2 times, most recently from ba6761e to c4059f2 Compare June 30, 2026 14:24
@datvo06

datvo06 commented Jun 30, 2026

Copy link
Copy Markdown
Contributor Author

My intuition is mypy failed with stress test. Checking..

@datvo06
datvo06 force-pushed the dn-696-splice-typecheck branch from c4059f2 to 0375180 Compare June 30, 2026 14:38
@datvo06
datvo06 marked this pull request as ready for review June 30, 2026 15:14
@datvo06
datvo06 requested a review from eb8680 June 30, 2026 15:14

@eb8680 eb8680 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this seems like a big improvement. There are several heuristics and silent failures that I worry will undermine its correctness, though.

Comment thread effectful/handlers/llm/evaluation.py Outdated
Comment thread effectful/handlers/llm/evaluation.py Outdated
Comment thread effectful/handlers/llm/evaluation.py Outdated
Comment thread effectful/handlers/llm/evaluation.py Outdated
Comment thread effectful/handlers/llm/evaluation.py Outdated
Comment thread effectful/handlers/llm/evaluation.py Outdated
Comment thread effectful/handlers/llm/evaluation.py Outdated
Comment thread effectful/handlers/llm/evaluation.py Outdated
Comment thread effectful/handlers/llm/completions.py Outdated
Comment thread effectful/handlers/llm/completions.py Outdated
@datvo06
datvo06 force-pushed the dn-696-splice-typecheck branch 2 times, most recently from 0a36f8a to 9dd1252 Compare July 1, 2026 01:31
Type-check LLM-generated Callables by splicing them into the Template's real
module source -- in place at the template def's position -- and running mypy on
the whole module, raising only on diagnostics inside the spliced region. This
removes the brittle quotation layer (type_to_ast, signature_to_ast, collect_*,
_RenameTransformer) and the runtime/local-type stubbing it depended on, which
mishandled common types such as Enum and runtime-created Pydantic models.

- mypy_type_check now takes (generated, anchor); the anchor (Template.__default__)
  is provided by the type_check_anchor effect op, bound per call by
  Template.__apply__ (so nested synthesis sees the right one) and rebound to None
  around tool-argument decoding, so the argument path skips the check.
- Region-scoped diagnostics: unrelated pre-existing errors elsewhere in the module
  never block synthesis (no spurious retry burn).
- Leaves the template def's decorators untouched (mypy checks the body against its
  declared return type regardless of decorators); explicit pre-scan for non-nestable
  constructs (star-import, __future__); source recovery via getsourcelines/file/
  linecache (REPL-defined templates); fatal mypy errors are surfaced not swallowed.
- Name collisions (#542) are handled structurally by nesting/shadowing, so the
  AST-rename pass is deleted.
- Delete the quotation emitters + their unit tests; add behavior tests covering
  collision shadowing, the gate, closures, method/static/class templates,
  Enum/dataclass context, injected-name, non-nestable, and linecache recovery.
@datvo06
datvo06 force-pushed the dn-696-splice-typecheck branch from 9dd1252 to b3ec9c3 Compare July 1, 2026 17:11
datvo06 added 12 commits July 1, 2026 14:11
The scan is provider-independent AST validation, not type-checking; lift it
into the decode as a precondition. Keep it gated on the anchor -- the
constructs it rejects are illegal only once nested by the splice, and legal
on the argument path's module-level exec.
mypy_type_check scraped mypy's human-readable output with a hand-rolled regex (_MYPY_LINE_RE) to pull line numbers and filter to the spliced region -- an undocumented format where a real error line that failed to match was silently dropped, letting bad code pass the gate. Switch to --output=json (documented, one object per diagnostic) and filter on mypy's own severity/line fields. A fatal exit (status >= 2) emits text not JSON, so it's handled before parsing.
…py failure

The mypy region was [def-line, end], so it flagged the Template's own signature -- pre-existing source the LLM never generated. For notebook/REPL templates whose recovered source is a single cell missing other cells' imports, the signature's annotations (Literal, Callable) look undefined to mypy, spuriously failing synthesis. Narrow the region to the spliced body only. Also: a mypy exit status >= 2 is a tool failure, not a type error -- raise RuntimeError, and check status before deriving a type-error TypeError from diagnostics.
Drop the redundant os.path.isfile + open branch: linecache.getlines reads real files from disk too (as inspect does), so one path covers real modules and linecache-registered REPL/exec/notebook sources alike. The function stays -- inspect.getsource(inspect.getmodule(fn)) can't replace it, since getmodule/getsource fail for exec- and __main__-defined templates that _recover_module_source handles via the function's co_filename.
Both single-use helpers. _recover_module_source folds into its one caller, collapsing its three return-None paths into the existing skip check. _unwrap is replaced by inspect.unwrap(anchor): staticmethod/classmethod implement __wrapped__ (== __func__), so the stdlib call returns the underlying function identically and drops the hand-rolled isinstance heuristic.
Split mypy_type_check into _splice_into_source (construct the modified module source) and _mypy_check_region (run mypy on a given source), with mypy_type_check as a thin orchestrator -- so the mypy step is decoupled from splicing. Also correct the tool-argument anchor-suppression comments (completions.py, encoding.py): the reason isn't 'no source anchor' (the Template's source is available) but that a tool-arg Callable's contract is the tool parameter's type, not the Template's return type, so the Template is the wrong splice target.
When the anchor's def can't be located in the recovered source, that's source drift (the file changed since the anchor was compiled), not a normal skip condition. Raise RuntimeError with a clear message instead of silently skipping the type check.

@eb8680 eb8680 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nearly there, just a few small things

Comment thread effectful/handlers/llm/encoding.py Outdated
Comment thread effectful/handlers/llm/evaluation.py Outdated
Comment thread effectful/handlers/llm/evaluation.py Outdated
Comment thread effectful/handlers/llm/evaluation.py Outdated
env, template.__signature__.return_annotation, **self.config
env,
template.__signature__.return_annotation,
anchor=template.__default__,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure template.__default__ will work as anchor for bound-method Templates. You should make sure this case is covered in a test, which you'll probably then need to fix.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added an integration test, and it works ok after multiple retries (due to LLM flakiness), also added the replay record.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The relay buffer gives me this:

┌───────┬────────────────────────────────────────────────────┬────────┐
│ Round │                   Generated code                   │ Valid? │
├───────┼────────────────────────────────────────────────────┼────────┤
│ 0     │ def add(a, b) -> int:\n    return a + b<br>\n\nadd │ ✗      │
├───────┼────────────────────────────────────────────────────┼────────┤
│ 1     │ def add(...): ...<br>\n\n# comment\n\nadd          │ ✗      │
├───────┼────────────────────────────────────────────────────┼────────┤
│ 2     │ def add(...): ...<br>\n\nadd                       │ ✗      │
├───────┼────────────────────────────────────────────────────┼────────┤
│ 3     │ def add(a: int, b: int) -> int:\n    return a + b  │ ✓      │
└───────┴────────────────────────────────────────────────────┴────────┘

The function body is correct every time — that's the part that "looks correct." But rounds 0–2 append a trailing bare add (the function name on its own line, as a "here's the result" hint). That makes the module's last statement a bare Expr, and decode requires it to be a FunctionDef:

Value error, decode() requires the last statement to be a function definition, got Expr

I think this is more of the mypy thing but not sure we should try adding fix in this PR cause
I'm not sure how sound/complete the solution could be, and it still works after multiple retries.

Comment thread effectful/handlers/llm/encoding.py Outdated
datvo06 added 2 commits July 11, 2026 15:36
…od integration test

type_check(source, lo=None, hi=None) now just runs mypy on a given source, reporting only diagnostics in [lo, hi] (whole file if omitted); splicing is a separate public splice_into_source(generated, anchor), and decode does the two steps explicitly. Removes the mypy_type_check convenience (tests splice + call the op). Error report uses json.dumps(e) per diagnostic and drops the (large) source from the message. Adds a bound-method Template synthesis integration test (replayed) covering the template.__default__ anchor for a bound method.
@eb8680
eb8680 merged commit fdd4bfd into master Jul 11, 2026
29 checks passed
@eb8680
eb8680 deleted the dn-696-splice-typecheck branch July 11, 2026 07:01
@datvo06 datvo06 mentioned this pull request Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Round-tripping through quotation in handlers.llm is brittle and incomplete

2 participants