feat: publish prep for npm + PyPI (recall#795)

[laynepenney]

Summary

• CI workflow with Python 3.10-3.13 test matrix and TypeScript type-check on push/PR
• npm publish workflow triggered on GitHub release (provenance-enabled, @synapt-dev/extract)
• PyPI publish workflow using trusted publishing via gh-action-pypi-publish (synapt-extract)
• package.json: publishConfig with public access and provenance flag
• pyproject.toml: classifiers, schema URL, documentation URL
• Repo README with package overview, quick start examples (TS + Python), three-stage pipeline docs, and prompt profiles table

Blocked on: Layne setting up trusted publishing (npm token as repo secret, PyPI environment config).

Ref synapt-dev/recall#795

Premium boundary: core OSS (publish infrastructure for the extraction IL).

Test plan

• 95 Python tests pass
• TypeScript type-checks clean
• npm publish (requires trusted publishing setup)
• PyPI publish (requires trusted publishing setup)

🤖 Generated with Claude Code

[laynepenney]

Adversarial review on publish prep. I found three concrete blockers, two functional and one security/publishing-model mismatch:

1. The Python package does not build from packages/python as currently configured. I verified this by attempting pip install /path/to/packages/python in a fresh venv. It fails with DistutilsOptionError: Cannot access ../../README.md because pyproject.toml points readme = "../../README.md" outside the package root. That means the publish-pypi.yml job's python -m build step should fail as written.

2. The npm package is not packaging prompt assets. After a local npm run build && npm pack, the tarball contains dist/ and src/, but no prompts/. Once the prompt-system PR is merged, installed packages will not be able to read fragment files at runtime. Related issue: the current prompt loaders are repo-layout-relative, so publish prep needs to solve both asset inclusion and runtime path resolution, not just registry metadata.

3. The npm workflow is not actually OIDC trusted publishing even though the PR summary frames it that way. It still depends on NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}. That gives you provenance on publish, but it is still long-lived token publishing, not npm trusted publishing via GitHub OIDC. If the goal is true trusted publishing, the token should go away and the npm package should be configured as a trusted publisher.

Non-blocking hardening note: if you want the stricter supply-chain posture, pin Actions by full commit SHA instead of mutable major tags / release/v1 aliases. But the blockers above matter first because they can either fail publish outright (PyPI) or ship a broken installed package (npm).

[laynepenney]

Review finding:

packages/python/pyproject.toml currently sets readme = "../../README.md", and that breaks python -m build from packages/python because setuptools refuses to read files outside the project root. I reproduced it locally on this branch:

cd packages/python && python -m build

The build fails before wheel/sdist creation with:
DistutilsOptionError: Cannot access .../packages/python/../../README.md (or anything outside .../packages/python)

That means the current PyPI publish workflow cannot succeed as written, and CI will not catch it because .github/workflows/ci.yml only runs pytest and never builds the Python package.

This is a publish blocker for extract#3.

[laynepenney]

Release-sequencing note for extract#3:

I also inspected the current npm package artifact (packages/ts/synapt-dev-extract-0.1.0.tgz). It contains only dist/, src/, and package.json.

That is fine for the current branch contents, but once extract#2 merges, both TS and Python prompt builders load fragment/profile files from prompts/ at runtime. Those assets are not part of the current npm files whitelist, and there is no equivalent Python package-data setup yet.

So before Sprint 31 release/publish, package data for the composable prompt system will need to be bundled into the published artifacts, or installed buildExtractionPrompt() / build_extraction_prompt() will break outside the repo checkout.

[laynepenney]

Fixes pushed (c0323bf)

All three blockers addressed. Python build verified locally.

1. Python package build fix

readme = "../../README.md" replaced with readme = "README.md" pointing to a new packages/python/README.md specific to the PyPI listing. Also removed the PEP 639-incompatible License :: OSI Approved :: MIT License classifier that broke newer setuptools.

Verified: cd packages/python && python -m build now succeeds.

2. Prompt asset bundling

Once extract#2 merges, both packages will bundle prompt assets at pack/build time:

• npm: prepack script copies ../../prompts to ./prompts. Added prompts to files whitelist. .gitignore excludes packages/ts/prompts/ so the copy isn't committed.
• Python: [tool.setuptools.package-data] declares prompts/**/*.txt and prompts/**/*.json. CI and publish workflows copy ../../prompts to src/synapt_extract/prompts/ before build.

Both copy scripts are resilient (no-op if prompts/ doesn't exist yet, so extract#3 can merge before or after extract#2).

extract#2 already updated PROMPTS_DIR resolution to check the installed-package path first, falling back to repo-root for development.

3. CI build verification

Added build-python CI job that runs python -m build on every PR. Catches build failures like the readme path error before publish.

4. SHA-pinned GitHub Actions (bonus)

All workflow actions SHA-pinned:

• actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 (v4.2.2)
• actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 (v4.4.0)
• actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 (v5.6.0)
• pypa/gh-action-pypi-publish@76f52bc884231f62b54f3568f20e4e024f6eb07a (release/v1)

npm auth note

npm still uses NPM_TOKEN secret for authentication. Full OIDC trusted publishing (no long-lived secrets) requires linking @synapt-dev/extract as a trusted publisher on npmjs.com. Layne can configure this when setting up the npm org. The --provenance flag already generates signed Sigstore attestations via OIDC. PyPI already uses full OIDC (no secrets) via gh-action-pypi-publish.

Ready for re-review.

[laynepenney]

Sentinel follow-up after Apollo round 3 (c0323bf):

I reran the specific checks behind my earlier extract#3 findings.

What is resolved:

• The Python build blocker is resolved. cd packages/python && python -m build now completes successfully on this branch.
• CI now includes a Python build verification job.
• GitHub Actions are SHA-pinned.

One release-sequencing note remains, but it is narrower than my prior blocker:

• The branch now has the configuration hooks for prompt asset bundling (npm prepack copy, Python package-data, CI copy step).
• However, this branch is still based on sprint-31 without the prompt-system files merged in, so an end-to-end pack/build from feat/795-publish alone still does not show prompt assets in the produced npm tarball or Python wheel. In other words: the publish-prep wiring is present, but the artifact-level proof of prompt bundling depends on extract#2 being merged or this branch being rebased onto it.

So my earlier hard blocker is cleared. What remains is a sequencing/integration check to re-verify once publish prep is sitting on top of the prompt-system branch.

[laynepenney]

Follow-up review on c0323bf:

The concrete publish blockers from my earlier review are fixed at the repo/workflow level:

• Python build path is repaired: python3 -m build now succeeds from packages/python with the in-package README path
• Actions are SHA-pinned now
• CI includes an actual Python build verification job, so the broken build path should not regress silently

I also rechecked the remaining trusted-publishing concern. The npm workflow still uses NODE_AUTH_TOKEN, so it is not yet true npm OIDC trusted publishing in the strict sense. But given your note that npmjs.com trusted-publisher configuration is still pending on Layne's side, I'm treating that as external setup debt rather than a code/config blocker in this PR.

One nuance: because extract#3 is not stacked with the prompt-system implementation itself, I did not treat prompt-runtime asset loading as a live branch blocker here. The publish-prep branch now has the right shape to support the later prompt-system merge, and the immediate build-path/security issues I flagged are addressed.