Skip to content

Python: regenerate stale type stubs and fix typing imports in stub generator#7927

Merged
knutwannheden merged 1 commit into
mainfrom
zippy-mule
Jun 6, 2026
Merged

Python: regenerate stale type stubs and fix typing imports in stub generator#7927
knutwannheden merged 1 commit into
mainfrom
zippy-mule

Conversation

@knutwannheden
Copy link
Copy Markdown
Contributor

Motivation

The auto-generated .pyi type stubs in rewrite-python had drifted from their source modules. The most visible symptom: markers.pyi declared SearchResult but omitted all Markup* classes (Markup, MarkupWarn, MarkupError, MarkupInfo, MarkupDebug), which were added to markers.py two days after the stubs were last regenerated. Regenerating showed 8 stub files were stale (nothing checks stub freshness, which is how they drifted).

Regenerating also surfaced a pre-existing bug in scripts/generate_stubs.py: it hardcoded from typing import Any, ClassVar, List, Optional (plus optional TypeVar, Generic) and never imported other typing names actually used in annotations — Dict, Callable, Type, Iterable, IO, or Generic used as a base class — so the generated stubs referenced names that were never imported.

Summary

  • Generator: import the typing names a stub actually uses. Added collect_typing_names, which scans annotation contexts (fields, parameters, return types) and class base expressions (for Generic[...]/Protocol[...]) for names in typing.__all__, and appends them to the header import — keeping Any, ClassVar, List, Optional first and TypeVar, Generic last to minimize diff churn.
  • Generator: avoid false imports from local-name collisions. Added collect_defined_names so a name shadowed by a local class/function/import (e.g. J.Literal, Py.TypeAlias, JavaType.Union — all collide with typing.__all__) is not wrongly imported from typing. Names imported from typing in the source are preserved (they are exactly what the stub must re-import).
  • Regenerated all stub files from current sources: markers.pyi, parser.pyi, tree.pyi, java/tree.pyi, java/support_types.pyi, python/markers.pyi, python/support_types.pyi, python/tree.pyi.

Test plan

  • All regenerated stubs parse as valid Python (ast.parse).
  • Every typing name referenced (in Load context) in each stub is resolvable — either imported or defined locally as a class — verified via an AST check across all stubs.
  • Generator is idempotent: a second python scripts/generate_stubs.py run produces zero further changes.

… stub generator

The `.pyi` stubs had drifted from their source modules — most visibly,
`markers.pyi` declared `SearchResult` but omitted the `Markup*` classes added
to `markers.py` two days after the stubs were last generated.

Regenerating also surfaced a generator bug: `generate_stubs.py` hardcoded
`from typing import Any, ClassVar, List, Optional` (plus optional
`TypeVar, Generic`) and never imported other typing names (`Dict`, `Callable`,
`Type`, `Iterable`, `IO`, or `Generic` used as a base class) that appear in
annotations, producing stubs with unresolved names.

- Detect typing names referenced in annotations and class bases, and import them
- Exclude names shadowed by local definitions so collisions like `J.Literal`,
  `Py.TypeAlias`, and `JavaType.Union` are not wrongly imported from `typing`
- Regenerate all stub files from current sources
@github-project-automation github-project-automation Bot moved this to In Progress in OpenRewrite Jun 6, 2026
@knutwannheden knutwannheden changed the title rewrite-python: regenerate stale type stubs and fix typing imports in stub generator Python: regenerate stale type stubs and fix typing imports in stub generator Jun 6, 2026
@knutwannheden knutwannheden merged commit 81d5217 into main Jun 6, 2026
1 check passed
@knutwannheden knutwannheden deleted the zippy-mule branch June 6, 2026 10:30
@github-project-automation github-project-automation Bot moved this from In Progress to Done in OpenRewrite Jun 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant