Skip to content

fix(python): stop shipping build artifacts and ship the PEP 561 marker - #4857

Merged
dbrattli merged 1 commit into
mainfrom
fix/python-wheel-packaging
Jul 28, 2026
Merged

fix(python): stop shipping build artifacts and ship the PEP 561 marker#4857
dbrattli merged 1 commit into
mainfrom
fix/python-wheel-packaging

Conversation

@dbrattli

Copy link
Copy Markdown
Collaborator

Two packaging defects in src/fable-library-py, pulling in opposite directions: the wheel built in the source tree shipped far too much and was uninstallable, while the wheel published to PyPI shipped too little.

Neither shows up in git status.gitignore has [Oo]bj/, so the offending files are invisible to git but still swept up by maturin.

1. The dev wheel was uninstallable

Fable.Library.Python.fsproj lives inside the package directory, so dotnet build drops MSBuild output into fable_library/obj/. With python-source = "." maturin packaged it wholesale — and one of those paths contains a comma:

fable_library/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.fs

RECORD is CSV, so that is four fields where three are expected:

Caused by: RECORD file is invalid
Caused by: CSV error: record 76 (line: 77, byte: 6817):
           found record with 4 fields, but the previous record has 3 fields

pip is lenient enough to survive this; uv pip install is not — which makes it look intermittent. It blocked building a local fable-library to benchmark against a released version.

The same wheel also carried 13 F# sources and 22 __pycache__ entries, including another interpreter's bytecode (array_.cpython-312.pyc inside a cp314 wheel) — a stale-cache hazard inside an artifact whose whole purpose is to be a clean build.

Fixed with an explicit exclude in [tool.maturin], which holds regardless of build order. Deleting obj/ in the build script would not: it regenerates on the next dotnet build and breaks again for the next person.

entries build artifacts installs
before 120 52
after 68 0

The sdist is covered by the same config (92 entries, 0 junk).

2. The published wheel's type stubs were inert

Under PEP 561 the marker must sit inside the package directory. CopyStage copied GetFiles(SourceDir, "*.py"), so fable_library/py.typed was dropped, while the distribution-root copy was staged to a location where PEP 561 does not look. Every released version has therefore shipped core/*.pyi stubs that mypy would not read.

Verified against an installed wheel, with and without the marker:

with:    error: Incompatible types in assignment
         (expression has type "Int32", variable has type "str")

without: error: Skipping analyzing "fable_library.core": module is installed,
         but missing library stubs or py.typed marker

Narrower than it first appears: pyright reads the stubs either way, since useLibraryCodeForTypes defaults to true. It is mypy that gets nothing; py.typed is what makes the stubs authoritative rather than inferred.

The inert distribution-root py.typed is removed rather than left to suggest the marker was handled.

3. Vestigial build config

Dropped [tool.hatch.build.targets.*] and hatchling from build-system.requires. build-backend is maturin, so those settings did nothing — while looking exactly like they governed what ships. Two build backends configured in one file is a good way for a packaging bug to go unnoticed.

On CI coverage

Added one line to build-python, which already stages the library: assert py.typed survives. That guards the regression class that actually occurred — a CopyStage extension filter silently dropping a file — and is also the code this PR changes, so it is the most likely thing to break again. It costs nothing in a job that already runs the staging step.

Deliberately not guarded: wheel contents. A clean checkout has no obj/ or __pycache__, so any guard must dirty the tree first. I tried to make it cheap via maturin sdist (0.3s, no Rust compile) and measured that it does not work — maturin honours gitignore for sdists:

entries junk
sdist, exclude present 92 0
sdist, exclude = [] 105 13 — all .fs, zero obj/

A cheap guard would pass while the actual breakage went through. Only a full wheel build reproduces it: minutes of CI per PR to protect a static config block that now carries its own explanation, against a break that never reached the published artifact.

Not addressed

Moving the .fsproj out of fable_library/ would remove the cause rather than filter it. But Fable is invoked on the directory (CmdLine.appendRaw sourceDir), and 12 F# sources plus 8 linked files from fable-library-ts live there — so sourceDir conflates "where the F# project is" with "where the Python package is". That is a restructure, not a packaging fix, and belongs on its own.

Verification

# reproduce, then confirm
cd src/fable-library-py
dotnet build fable_library/Fable.Library.Python.fsproj
python -c "import compileall; compileall.compile_dir('fable_library')"
uv run --frozen maturin build --release --out dist
uv pip install --no-cache --target /tmp/check dist/*.whl   # succeeds

./build.sh fable-library --python
test -f temp/fable-library-py/fable_library/py.typed
  • 2418 Python tests, 270 fable-library-py tests
  • wheel and sdist clean and installable; marker present in the staged layout
  • PEP 561 behaviour confirmed with mypy against a real venv install, both with and without the marker

🤖 Generated with Claude Code

Two packaging defects in `src/fable-library-py`, pulling in opposite directions:
the wheel built in the source tree shipped far too much and was uninstallable,
while the wheel published to PyPI shipped too little.

Neither shows up in `git status` -- `.gitignore` has `[Oo]bj/`, so the offending
files are ignored by git but still swept up by maturin.

## The dev wheel was uninstallable

`Fable.Library.Python.fsproj` lives inside the package directory, so `dotnet build`
drops MSBuild output into `fable_library/obj/`. With `python-source = "."` maturin
packaged it wholesale, and one of those paths contains a comma:

    fable_library/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.fs

RECORD is CSV, so that is four fields where three are expected, and the wheel is
rejected:

    Caused by: RECORD file is invalid
    Caused by: CSV error: record 76: found record with 4 fields, but the previous
               record has 3 fields

`pip` is lenient enough to survive this; `uv pip install` is not, which makes the
failure look intermittent. It blocked building a local `fable-library` to benchmark
against a released version.

The same wheel also carried 13 F# sources and 22 `__pycache__` entries -- including
another interpreter's bytecode (`array_.cpython-312.pyc` inside a cp314 wheel), a
stale-cache hazard inside an artifact whose purpose is to be a clean build.

Fixed with an explicit `exclude` in `[tool.maturin]`, which works regardless of
build order. Deleting `obj/` in the build script would not: it regenerates on the
next `dotnet build` and breaks again for the next person.

120 -> 68 entries, 0 build artifacts, installs cleanly. The sdist is covered too.

## The published wheel's type stubs were inert

Under PEP 561 the marker must sit *inside* the package directory. `CopyStage` copied
`GetFiles(SourceDir, "*.py")`, so `fable_library/py.typed` was dropped, while the
distribution-root copy was staged to a location where PEP 561 does not look. Every
released version therefore shipped `core/*.pyi` that mypy would not read.

Verified against an installed wheel, with and without the marker:

    with:    error: Incompatible types in assignment (expression has type "Int32",
             variable has type "str")
    without: error: Skipping analyzing "fable_library.core": module is installed,
             but missing library stubs or py.typed marker

Note this is narrower than it first appears: pyright reads the stubs either way,
since `useLibraryCodeForTypes` defaults to true. It is mypy that gets nothing, and
`py.typed` is what makes the stubs authoritative rather than inferred.

The inert distribution-root `py.typed` is removed rather than left to suggest the
marker was handled.

## Also

- Dropped the vestigial `[tool.hatch.build.targets.*]` config and `hatchling` from
  `build-system.requires`. `build-backend` is maturin, so those settings did
  nothing -- while looking exactly like they governed what ships, which is a good
  way for a packaging bug to go unnoticed.
- Added a one-line assertion to `build-python` that the marker survives staging.
  `CopyStage` filters by extension, so this is easy to drop again, and it costs
  nothing in a job that already stages the library.

Deliberately not guarded in CI: wheel contents. A clean checkout has no `obj/` or
`__pycache__`, so a guard would have to dirty the tree first, and the sdist does not
exercise those exclusions at all -- maturin honours gitignore for sdists, so an
sdist built with `exclude = []` still shows 0 obj/ entries. Only a full wheel build
reproduces the failure, which is minutes of CI per PR to protect a static config
block that carries its own explanation, against a break that never reached the
published artifact.

Not addressed: moving the fsproj out of `fable_library/` would remove the cause
rather than filter it, but Fable is invoked on the directory and 12 F# sources plus
8 linked files live there, so it is a restructure rather than a packaging fix.

Verified: 2418 Python tests, 270 fable-library-py tests, wheel and sdist clean and
installable, marker present in the staged layout.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Python Type Checking Results (Pyright)

Metric Value
Total errors 34
Files with errors 4
Excluded files 4
New errors ✅ No
Excluded files with errors (4 files)

These files have known type errors and are excluded from CI. Remove from pyrightconfig.ci.json as errors are fixed.

File Errors Status
temp/tests/Python/test_hash_set.py 18 Excluded
temp/tests/Python/test_applicative.py 12 Excluded
temp/tests/Python/test_nested_and_recursive_pattern.py 2 Excluded
temp/tests/Python/fable_modules/thoth_json_python/encode.py 2 Excluded

@dbrattli
dbrattli merged commit 03d40c9 into main Jul 28, 2026
42 checks passed
@dbrattli
dbrattli deleted the fix/python-wheel-packaging branch July 28, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant