Skip to content

Adopt pip-compile for Python dependency management - #380

Merged
jonasbn merged 5 commits into
masterfrom
chore/pip-compile-dependency-management
Jul 30, 2026
Merged

Adopt pip-compile for Python dependency management#380
jonasbn merged 5 commits into
masterfrom
chore/pip-compile-dependency-management

Conversation

@jonasbn

@jonasbn jonasbn commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Follow-up to Bump bracex from 2.5.post1 to 3.0.1 to fix glob source matching #379 (kept separate per request). Where Bump bracex from 2.5.post1 to 3.0.1 to fix glob source matching #379 is the immediate patch for Strange problem with source matching with 0.63.0 #378, this PR is the structural fix so this class of bug (a hand-edited transitive pin silently falling out of sync with what its own dependency tree needs) can't recur.
  • Adds requirements.in as the source of truth for genuinely direct dependencies (pyspelling, pymdown-extensions — the latter is used directly via pymdownx.superfences in this repo's own spellcheck.yaml and examples/markdown.yml).
  • requirements.txt is now generated from requirements.in via pip-compile (pip-tools) instead of hand-maintained. It's still a fully pinned file installed as-is by the Dockerfile — no Dockerfile changes needed.
  • This run is a minimal-diff regeneration: only bracex/wcmatch move (to the same compatible pair as Bump bracex from 2.5.post1 to 3.0.1 to fix glob source matching #379bracex==3.0.1, wcmatch==11.0); every other package stays at its currently pinned version. A full --upgrade resolve was available but intentionally not used here to avoid bundling unrelated version bumps (beautifulsoup4, lxml, Markdown, PyYAML, six, soupsieve all have newer releases available) into a tooling-adoption PR — those stay as their own deliberate, individually-reviewed PRs as before.
  • backrefs and zipp are dropped. Neither shows up anywhere in the resolved dependency graph for pyspelling + pymdown-extensions on Python 3.14 (verified via each package's own declared metadata) — they were stale manual pins (the zipp comment even said "not directly required, pinned by Snyk to avoid a vulnerability"), not active dependencies, so removing them is not a security regression.
  • .github/dependabot.yml's pip entry already sets versioning-strategy: lockfile-only, which is specifically designed for a requirements.in (manifest) + generated requirements.txt (lockfile) pair. No change needed there — this PR just makes that existing setting apply the way it was presumably intended, so Dependabot should now be able to bump transitive pins like bracex on its own going forward.
  • Documents the new workflow in CLAUDE.md (Development Commands + Key Files + Dependabot Gotchas).

Does removing zipp reintroduce a vulnerability?

No. zipp>=3.19.1 was originally pinned in PR #204 (changelog 0.39.0) to patch CVE-2024-5569, because at the time pyspelling pulled in importlib-metadata, which depends on zipp.

That chain no longer exists:

  • pyspelling==2.12.1's own declared metadata (Requires-Dist) lists only beautifulsoup4, html5lib, lxml, markdown, pyyaml, soupsieve, wcmatch — no importlib-metadata, conditional or otherwise.
  • The full pip3 list inside the rebuilt image (13 packages total) has neither importlib-metadata nor zipp.

So zipp isn't installed in the image at all, pinned or not — a package that's absent can't be exploited through this image. The old pin was a vestigial defensive measure for a dependency edge that pyspelling has since dropped (likely once its minimum Python requirement rose past the point of needing the importlib-metadata backport). Same reasoning applies to backrefs, which was never a declared dependency of pymdown-extensions at the pinned version.

Verification

Test plan

  • Regenerate requirements.txt via pip-compile with a minimal diff
  • Confirm backrefs/zipp are genuinely unused (checked each package's declared metadata)
  • Rebuild locally and rerun the Strange problem with source matching with 0.63.0 #378 repro against the reporter's real config
  • Smoke test against this repo's own spellcheck.yaml
  • pre-commit hooks pass
  • CLAUDE.md documents the new pip-compile workflow

🤖 Generated with Claude Code

Manually hand-editing transitive pins in requirements.txt is what
caused #378: bracex was pinned below the version wcmatch itself
requires, and nothing caught the mismatch. Introduce requirements.in
as the source of truth for direct dependencies (pyspelling,
pymdown-extensions) and generate requirements.txt via `pip-compile`,
so pip's own resolver enforces compatible transitive versions instead
of relying on manual curation.

This is a minimal-diff regeneration: only bracex/wcmatch move (to the
same compatible versions as #379), everything else stays pinned as
before. `backrefs` and `zipp` are dropped since pip-compile's resolver
confirms neither is actually part of the dependency graph for
pyspelling + pymdown-extensions on Python 3.14 — they were stale
manual additions, not active mitigations, so removing them is not a
security regression.

.github/dependabot.yml already sets `versioning-strategy: lockfile-only`
for the pip ecosystem, which is designed for exactly this
requirements.in/requirements.txt split. No change needed there; this
commit's requirements.in makes that existing setting apply correctly,
and CLAUDE.md documents the new workflow.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 30, 2026 14:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR formalizes Python dependency management for the Docker image by introducing a pip-tools (pip-compile) workflow: requirements.in becomes the direct-dependency manifest, and requirements.txt is treated as the generated, fully pinned lockfile to prevent transitive pin drift (the root cause class behind #378/#379).

Changes:

  • Add requirements.in to explicitly declare direct dependencies (pyspelling, pymdown-extensions).
  • Regenerate requirements.txt via pip-compile (minimal diff) to pin a compatible bracex==3.0.1 / wcmatch==11.0 pair and remove stale pins (backrefs, zipp).
  • Document the new dependency update workflow and Dependabot expectations in CLAUDE.md.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated no comments.

File Description
requirements.txt Switch to a pip-compile-generated, fully pinned lockfile (including bracex/wcmatch correction and removal of stale pins).
requirements.in Introduce a manifest for direct Python dependencies used to generate requirements.txt.
CLAUDE.md Document the pip-compile workflow and clarify how Dependabot’s lockfile-only strategy aligns with requirements.in/requirements.txt.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…mage contents

Both techniques were used to investigate and verify the #378 fix in
this PR: overriding INPUT_CONFIG_PATH to reproduce the reporter's
real config, and pulling the published image directly to see what
pip actually installed versus what requirements.txt says.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jonasbn

jonasbn commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator Author

@copilot resolve the merge conflicts in this pull request

Copilot AI commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

@copilot resolve the merge conflicts in this pull request

Resolved the merge conflicts by merging origin/master into this branch and fixing requirements.txt. Addressed in commit 6f4f174.

jonasbn and others added 2 commits July 30, 2026 22:21
Documents the pip-compile dependency management migration (PR #380),
including the backrefs/zipp removal and why it doesn't reintroduce
CVE-2024-5569.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

@jonasbn jonasbn left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@jonasbn
jonasbn merged commit d4fd16d into master Jul 30, 2026
2 checks passed
@jonasbn jonasbn added this to the 0.64.0 milestone Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants