New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add CLI command to py-compile wheels #3253
Merged
Merged
Changes from all commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
95125d4
Initial CLI implementation
rth d123875
Initial implementation
rth a99d928
Fix core tests
rth d26bf4f
More test fixes
rth e7da92b
Merge remote-tracking branch 'upstream/main' into py-compile-wheels
rth a63eb3f
Rework tag updating
rth cc604bb
Xfail numpy test and path invalid syntax in RobotRaconteur
rth fe424e4
Merge remote-tracking branch 'upstream/main' into py-compile-wheels
rth 84029ca
Merge remote-tracking branch 'upstream/main' into py-compile-wheels
rth c1c8dc5
Revert changes to build, add test and changelog
rth 4a50ef8
Test tracebacks
rth c9487e9
Merge branch 'main' into py-compile-wheels
rth a9987fe
Merge branch 'pyodide:main' into py-compile-wheels
ryanking13 476ae85
Update changelog.md
ryanking13 14ed612
Apply suggestions from code review
rth 8e5e67f
Merge remote-tracking branch 'upstream/main' into py-compile-wheels
rth 99c1360
Update comment
rth File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
54 changes: 54 additions & 0 deletions
54
packages/packaging/patches/allow-cpp310-none-any-tag.patch
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
From b8b2f8aa47c0676628f336487a31f602bea0c857 Mon Sep 17 00:00:00 2001 | ||
From: Thimo <thimo.kraemer@joonis.de> | ||
Date: Sat, 20 Aug 2022 00:10:48 +0200 | ||
Subject: [PATCH] Add a "cpNNN-none-any" tag (#541) | ||
|
||
Note: Can be removed with >=22.1 packaging release | ||
|
||
Closes #511 | ||
|
||
See https://github.com/pypa/pip/issues/10923 for motivation. | ||
|
||
Co-authored-by: Brett Cannon <brett@python.org> | ||
--- | ||
packaging/tags.py | 7 +++++-- | ||
tests/test_tags.py | 13 +++++++++++++ | ||
2 files changed, 18 insertions(+), 2 deletions(-) | ||
|
||
diff --git a/packaging/tags.py b/packaging/tags.py | ||
index 5b6c5ffd..a0e1ea23 100644 | ||
--- a/packaging/tags.py | ||
+++ b/packaging/tags.py | ||
@@ -499,6 +499,9 @@ def sys_tags(*, warn: bool = False) -> Iterator[Tag]: | ||
yield from generic_tags() | ||
|
||
if interp_name == "pp": | ||
- yield from compatible_tags(interpreter="pp3") | ||
+ interp = "pp3" | ||
+ elif interp_name == "cp": | ||
+ interp = "cp" + interpreter_version(warn=warn) | ||
else: | ||
- yield from compatible_tags() | ||
+ interp = None | ||
+ yield from compatible_tags(interpreter=interp) | ||
diff --git a/tests/test_tags.py b/tests/test_tags.py | ||
index 06cd3b4a..39515e8d 100644 | ||
--- a/tests/test_tags.py | ||
+++ b/tests/test_tags.py | ||
@@ -1226,3 +1226,16 @@ def test_pypy_first_none_any_tag(self, monkeypatch): | ||
break | ||
|
||
assert tag == tags.Tag("pp3", "none", "any") | ||
+ | ||
+ def test_cpython_first_none_any_tag(self, monkeypatch): | ||
+ # When building the complete list of cpython tags, make sure the first | ||
+ # <interpreter>-none-any one is cpxx-none-any | ||
+ monkeypatch.setattr(tags, "interpreter_name", lambda: "cp") | ||
+ | ||
+ # Find the first tag that is ABI- and platform-agnostic. | ||
+ for tag in tags.sys_tags(): | ||
+ if tag.abi == "none" and tag.platform == "any": | ||
+ break | ||
+ | ||
+ interpreter = f"cp{tags.interpreter_version()}" | ||
+ assert tag == tags.Tag(interpreter, "none", "any") |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,169 @@ | ||
import py_compile | ||
import shutil | ||
import sys | ||
import zipfile | ||
from pathlib import Path | ||
from tempfile import TemporaryDirectory | ||
|
||
from packaging.tags import Tag | ||
from packaging.utils import parse_wheel_filename | ||
|
||
|
||
def _specialize_convert_tags(tags: set[Tag] | frozenset[Tag], wheel_name: str) -> Tag: | ||
"""Convert a sequence of wheel tags to a single tag corresponding | ||
to the current interpreter and compatible with the py -> pyc compilation. | ||
|
||
Having more than one output tag is not supported. | ||
|
||
Examples | ||
-------- | ||
>>> from packaging.tags import parse_tag | ||
>>> tags = parse_tag("py2.py3-none-any") | ||
>>> str(_specialize_convert_tags(set(tags), "")) | ||
'cp310-none-any' | ||
>>> tags = parse_tag("cp310-cp310-emscripten_3_1_24_wasm32") | ||
>>> str(_specialize_convert_tags(set(tags), "")) | ||
'cp310-cp310-emscripten_3_1_24_wasm32' | ||
>>> tags = parse_tag("py310.py311-any-none") | ||
>>> str(_specialize_convert_tags(set(tags), "")) | ||
'cp310-any-none' | ||
>>> tags = parse_tag("py36-abi3-none") | ||
>>> str(_specialize_convert_tags(set(tags), "")) | ||
'cp310-abi3-none' | ||
""" | ||
if len(tags) == 0: | ||
raise ValueError("Failed to parse tags from the wheel file name: {wheel_name}!") | ||
|
||
output_tags = set() | ||
interpreter = "cp" + "".join(str(el) for el in sys.version_info[:2]) | ||
for tag in tags: | ||
output_tags.add( | ||
Tag(interpreter=interpreter, abi=tag.abi, platform=tag.platform) | ||
) | ||
|
||
if len(output_tags) > 1: | ||
# See https://github.com/pypa/packaging/issues/616 | ||
raise NotImplementedError( | ||
"Found more than one output tag after py-compilation: " | ||
f"{[str(tag) for tag in output_tags]} in {wheel_name}" | ||
) | ||
|
||
return list(output_tags)[0] | ||
|
||
|
||
def _py_compile_wheel_name(wheel_name: str) -> str: | ||
"""Return the name of the py-compiled wheel | ||
|
||
See https://peps.python.org/pep-0427/ for more information. | ||
|
||
Examples | ||
-------- | ||
>>> _py_compile_wheel_name('micropip-0.1.0-py3-none-any.whl') | ||
'micropip-0.1.0-cp310-none-any.whl' | ||
>>> _py_compile_wheel_name("numpy-1.22.4-cp310-cp310-emscripten_3_1_24_wasm32.whl") | ||
'numpy-1.22.4-cp310-cp310-emscripten_3_1_24_wasm32.whl' | ||
>>> # names with '_' are preserved (instead of using '-') | ||
>>> _py_compile_wheel_name("a_b-0.0.0-cp310-cp310-emscripten_3_1_24_wasm32.whl") | ||
'a_b-0.0.0-cp310-cp310-emscripten_3_1_24_wasm32.whl' | ||
>>> # if there are multiple tags (e.g. py2 & py3), we only keep the relevant one | ||
>>> _py_compile_wheel_name('attrs-21.4.0-py2.py3-none-any.whl') | ||
'attrs-21.4.0-cp310-none-any.whl' | ||
|
||
|
||
# >>> msg = "Processing more than one tag is not implemented" | ||
# >>> with pytest.rases(NotImplementedError, match=msg): | ||
# ... _py_compile_wheel_name("numpy-1.23.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl") | ||
""" | ||
name, version, build, tags = parse_wheel_filename(wheel_name) | ||
if build: | ||
# TODO: not sure what to do here, but we never have such files in Pyodide | ||
# Opened https://github.com/pypa/packaging/issues/616 about it. | ||
raise NotImplementedError(f"build tag {build} not implemented") | ||
output_name = f"{name.replace('-', '_')}-{version}-" | ||
output_name += str(_specialize_convert_tags(tags, wheel_name=wheel_name)) | ||
return output_name + ".whl" | ||
|
||
|
||
def _py_compile_wheel( | ||
wheel_path: Path, | ||
keep: bool = True, | ||
verbose: bool = True, | ||
) -> Path: | ||
"""Compile .py files to .pyc in a wheel | ||
|
||
All non Python files are kept unchanged. | ||
|
||
Parameters | ||
---------- | ||
wheel_path | ||
input wheel path | ||
keep | ||
if False, delete the input file. Otherwise, it will be either kept or | ||
renamed with a suffix .whl.old (if the input path == computed output | ||
path) | ||
verbose | ||
print logging information | ||
|
||
Returns | ||
------- | ||
wheel_path_out | ||
processed wheel with .pyc files. | ||
|
||
|
||
""" | ||
if wheel_path.suffix != ".whl": | ||
raise ValueError(f"Error: only .whl files are supported, got {wheel_path.name}") | ||
|
||
if not wheel_path.exists(): | ||
raise FileNotFoundError(f"{wheel_path} does not exist!") | ||
|
||
wheel_name_out = _py_compile_wheel_name(wheel_path.name) | ||
wheel_path_out = wheel_path.parent / wheel_name_out | ||
if verbose: | ||
print(f" - Running py-compile on {wheel_path} -> ", end="", flush=True) | ||
|
||
with zipfile.ZipFile(wheel_path) as fh_zip_in, TemporaryDirectory() as temp_dir_str: | ||
temp_dir = Path(temp_dir_str) | ||
wheel_path_tmp = temp_dir / wheel_name_out | ||
with zipfile.ZipFile( | ||
wheel_path_tmp, mode="w", compression=zipfile.ZIP_DEFLATED | ||
) as fh_zip_out: | ||
for name in fh_zip_in.namelist(): | ||
if name.endswith(".pyc"): | ||
# We are going to re-compile all .pyc files | ||
continue | ||
|
||
stream = fh_zip_in.read(name) | ||
if not name.endswith(".py"): | ||
# Write file without changes | ||
fh_zip_out.writestr(name, stream) | ||
continue | ||
|
||
# Otherwise write file to disk and run py_compile | ||
# Unfortunately py_compile doesn't support bytes input/output, it has to be real files | ||
tmp_path_py = temp_dir / name.replace("/", "_") | ||
tmp_path_py.write_bytes(stream) | ||
|
||
tmp_path_pyc = temp_dir / (tmp_path_py.name + "c") | ||
py_compile.compile( | ||
str(tmp_path_py), cfile=str(tmp_path_pyc), dfile=name, doraise=True | ||
) | ||
|
||
fh_zip_out.writestr(name + "c", tmp_path_pyc.read_bytes()) | ||
if wheel_name_out == wheel_path.name: | ||
if keep: | ||
if verbose: | ||
print( | ||
" (adding .old prefix to avoid overwriting input file) ->", | ||
end="", | ||
flush=True, | ||
) | ||
wheel_path.rename(wheel_path.with_suffix(".whl.old")) | ||
elif not keep: | ||
# Remove input file | ||
wheel_path.unlink() | ||
|
||
shutil.copyfile(wheel_path_tmp, wheel_path_out) | ||
if verbose: | ||
print(f" {wheel_path_out.name}") | ||
return wheel_path_out |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
import sys | ||
from pathlib import Path | ||
|
||
import typer # type: ignore[import] | ||
|
||
from pyodide_build._py_compile import _py_compile_wheel | ||
|
||
|
||
def main( | ||
wheel_path: Path = typer.Argument(..., help="Path to the input wheel"), | ||
) -> None: | ||
"""Compile .py files to .pyc in a wheel""" | ||
if wheel_path.suffix != ".whl": | ||
typer.echo(f"Error: only .whl files are supported, got {wheel_path.name}") | ||
sys.exit(1) | ||
_py_compile_wheel(wheel_path, verbose=False) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Packaging 0.22.0 is released now, so probably we can go ahead and update.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For now, we are still blocked by pypa/packaging#638 due to pyodide/micropip#36 so we would need to wait for 0.22.1.
I'll update the comment in the patch.