Skip to content

Adds support for pulling packages from PyPI#273

Merged
arcanis merged 4 commits intomainfrom
mael/pypi
Apr 7, 2026
Merged

Adds support for pulling packages from PyPI#273
arcanis merged 4 commits intomainfrom
mael/pypi

Conversation

@arcanis
Copy link
Copy Markdown
Member

@arcanis arcanis commented Apr 7, 2026

This PR adds experimental support for pulling packages from the PyPI repository. Such dependencies must be annotated with the pypi: prefix and have no special support in the linker phase.


Note

Medium Risk
Adds a new network-backed package source and resolution logic (PEP 440 parsing, metadata-driven version selection, dependency inference), which can affect install determinism and compatibility across registries.

Overview
Adds experimental pypi: support end-to-end: new Range/Reference variants plus PEP 440 parsing (PypiVersion/PypiSpecifierSet) via pep440_rs, and serialization coverage in Descriptor/Locator.

zpm now resolves pypi: ranges/tags by querying PyPI JSON metadata, selecting a wheel artifact, and deriving dependencies from requires_dist (ignoring marker-based entries), then fetches and caches the wheel bytes as a .zip artifact. Content flag extraction is skipped for PyPI archives.

Acceptance tests add a mock PyPI API to the test server and new protocol tests validating byte-for-byte caching, range resolution, and requires_dist handling.

Reviewed by Cursor Bugbot for commit f29e955. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 7, 2026

⏱️ Benchmark Results

gatsby install-full-cold

Metric Base Head Difference
Mean 2.413s 2.441s +1.16% ⚠️
Median 2.413s 2.422s +0.37% ⚠️
Min 2.366s 2.361s
Max 2.457s 2.665s
Std Dev 0.022s 0.072s
📊 Raw benchmark data (gatsby install-full-cold)

Base times: 2.398s, 2.388s, 2.431s, 2.457s, 2.423s, 2.442s, 2.377s, 2.416s, 2.413s, 2.416s, 2.398s, 2.441s, 2.410s, 2.423s, 2.397s, 2.419s, 2.407s, 2.423s, 2.413s, 2.407s, 2.367s, 2.421s, 2.402s, 2.400s, 2.427s, 2.409s, 2.451s, 2.366s, 2.410s, 2.424s

Head times: 2.369s, 2.412s, 2.412s, 2.393s, 2.361s, 2.402s, 2.406s, 2.402s, 2.436s, 2.430s, 2.373s, 2.475s, 2.588s, 2.652s, 2.665s, 2.452s, 2.407s, 2.382s, 2.417s, 2.420s, 2.441s, 2.407s, 2.417s, 2.431s, 2.435s, 2.457s, 2.442s, 2.457s, 2.424s, 2.455s


gatsby install-cache-and-lock (warm, with lockfile)

Metric Base Head Difference
Mean 0.362s 0.368s +1.82% ⚠️
Median 0.359s 0.366s +1.95% ⚠️
Min 0.351s 0.355s
Max 0.421s 0.428s
Std Dev 0.013s 0.014s
📊 Raw benchmark data (gatsby install-cache-and-lock (warm, with lockfile))

Base times: 0.358s, 0.351s, 0.353s, 0.356s, 0.353s, 0.355s, 0.352s, 0.370s, 0.361s, 0.360s, 0.358s, 0.363s, 0.366s, 0.359s, 0.362s, 0.365s, 0.365s, 0.421s, 0.357s, 0.358s, 0.356s, 0.355s, 0.380s, 0.364s, 0.362s, 0.364s, 0.359s, 0.359s, 0.358s, 0.357s

Head times: 0.365s, 0.356s, 0.366s, 0.362s, 0.357s, 0.358s, 0.360s, 0.366s, 0.363s, 0.366s, 0.366s, 0.378s, 0.371s, 0.375s, 0.379s, 0.373s, 0.369s, 0.371s, 0.366s, 0.428s, 0.393s, 0.357s, 0.360s, 0.357s, 0.361s, 0.363s, 0.381s, 0.368s, 0.360s, 0.355s

@arcanis
Copy link
Copy Markdown
Member Author

arcanis commented Apr 7, 2026

@cursor review


if let Some(post) = parsed.post() {
prerelease_segments.push("post".to_string());
prerelease_segments.push(post.to_string());
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Post-release versions mapped to semver pre-releases inverts ordering

Medium Severity

to_lossy_semver places PEP 440 post releases into the semver prerelease segment, which inverts their ordering. In PEP 440, 1.0.0.post1 is NEWER than 1.0.0, but the projected semver 1.0.0-post.1 is OLDER than 1.0.0 per semver ordering rules. This projected version is stored in Resolution via project_pep440_to_semver and could cause incorrect behavior in any downstream code that compares resolution versions using semver semantics.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 5f0b38e. Configure here.

pub fn encode_path_segment(segment: &str) -> String {
url::form_urlencoded::byte_serialize(segment.as_bytes())
.collect::<String>()
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Form URL encoding used for URL path segments

Low Severity

encode_path_segment uses url::form_urlencoded::byte_serialize, which follows application/x-www-form-urlencoded rules (encoding spaces as + instead of %20). This is the wrong encoding for URL path segments. While it works for typical PyPI names and versions, it would produce incorrect encoding for any input containing spaces or other characters that differ between form and percent encoding schemes.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 5f0b38e. Configure here.

Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit f29e955. Configure here.

}

Ok(best)
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Version selection ignores wheel availability causing resolution failures

Medium Severity

select_version_for_specifier and select_latest_version pick the highest matching version without checking whether it has any wheel distributions. If the highest version only ships an sdist (no wheels), select_best_wheel returns None and resolution fails with an error — even when an older matching version has perfectly valid wheels available. This two-phase approach (pick version first, then find wheel) doesn't fall back to try the next-best version.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit f29e955. Configure here.

@arcanis arcanis merged commit ddc6a46 into main Apr 7, 2026
14 checks passed
@arcanis arcanis deleted the mael/pypi branch April 7, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant