fix(python): PEP 517 builds from sdist produce wheels with missing proto stubs

[andre-motta]

Agent Diagnostic

• No OpenShell-specific skill applies (this is a build-system packaging issue, not a runtime bug)
• Investigated pyproject.toml build-system configuration, .gitignore rules, mise task graph, Dockerfile.python-wheels, and upstream CI workflows
• Built sdist with maturin sdist and confirmed _proto/ contains only __init__.py
• Built wheel from git checkout with maturin 1.7.8 and 1.11.5; both include proto stubs (because stubs were pre-generated on disk)
• Confirmed [build-system].requires lists only maturin, not grpcio-tools
• Confirmed no maturin build hook or custom build script invokes protoc
• Root cause: proto generation is wired only through the mise task graph (python:proto dependency on build:python:wheel:linux), not through PEP 517 build metadata
• Searched existing issues: Linux wheels for 0.0.54 are missing openshell/_proto/*_pb2.py files present in macOS wheel #1705 (closed by fix(python): include generated proto stubs in Linux wheels #2029) addressed gitignore-vs-include, but not the missing generation step
• Tested on current main (v0.0.96)

Description

The Python wheel ships protobuf-generated stubs (*_pb2.py, *_pb2_grpc.py, *.pyi) under openshell/_proto/. These stubs are:

1. Generated by mise run python:proto (invokes grpc_tools.protoc)
2. Gitignored (.gitignore lines 100-101: python/openshell/_proto/* with !__init__.py)
3. Listed in [tool.maturin].include globs so maturin force-includes them despite gitignore

This works when building from a git checkout where python:proto has already run (all upstream CI paths). It breaks when building from the source tarball (sdist), because:

• The sdist does not contain the generated stubs (they are gitignored and not committed)
• [build-system].requires lists only maturin>=1.5,<2.0, not grpcio-tools
• No maturin build hook or build script triggers proto generation
• The include globs silently match nothing
• The resulting wheel has _proto/__init__.py only

__init__.py does from . import datamodel_pb2, openshell_pb2, so importing the package hard-fails with ImportError.

This affects any PEP 517 build frontend (pip, build, fromager) building from the sdist, and any downstream rebuild pipeline that does not replicate the mise task graph.

Reproduction Steps

1. Generate sdist:

   maturin sdist --out /tmp/sdist-test

2. Inspect sdist contents:

   tar tzf /tmp/sdist-test/openshell-*.tar.gz | grep _proto
   # Only __init__.py present

3. Extract and build wheel from sdist:

   mkdir /tmp/sdist-repro && cd /tmp/sdist-repro
   tar xzf /tmp/sdist-test/openshell-*.tar.gz
   cd openshell-*/
   maturin build --release --out /tmp/wheels-from-sdist

4. Inspect wheel:

   unzip -l /tmp/wheels-from-sdist/*.whl | grep _proto
   # Only __init__.py — no *_pb2.py files

5. Verify import fails:

   pip install /tmp/wheels-from-sdist/*.whl
   python -c "import openshell._proto"
   # ImportError: cannot import name 'datamodel_pb2'

Environment

• OS: Fedora 44 (kernel 7.1.3)
• maturin: tested 1.7.8 and 1.11.5
• OpenShell: v0.0.96 (current main)
• Latest release checked: yes, issue present on main
• Possible duplicates checked: Linux wheels for 0.0.54 are missing openshell/_proto/*_pb2.py files present in macOS wheel #1705 (closed by fix(python): include generated proto stubs in Linux wheels #2029) addressed include paths but not the missing generation step

Possible Fixes

Option A: Add grpcio-tools to [build-system].requires and wire a build script that runs protoc before maturin collects files.

Option B: Stop gitignoring the generated stubs and commit them. They would then ship in the sdist and be found by the include globs. Requires regeneration discipline on proto changes.

Option C: Generate stubs in a maturin pre-build hook (if maturin supports one).