The anatomy of the wheel

[techtonik]

Here is my research on how to build wheels manually:
https://github.com/techtonik/leanbook/blob/master/manuscript/07wheel-anatomy.md

This was used to pack RHVoice project:
https://github.com/techtonik/RHVoice/blob/d08c7244c4fd47daa6ff9576f8c9ca4491c87d6e/src/nvda-synthDriver/PACKING.md

[njsmith]

Wow that's... really terrible.

Just filed a bug on pip so it will hopefully start rejecting your invalid wheels: pypa/pip#3513

why not just create a valid METADATA and RECORD file?

(the reason they have both METADATA and WHEEL is that WHEEL contains metadata about the format of the wheel package itself, while METADATA contains metadata about the package contained in the wheel. All versions of the wheel format will necessarily have a WHEEL file, because that's what tells you what version you have so you can interpret everything else; but new versions of the format could potentially replace METADATA with some other way of storing the same information.)

[techtonik]

@njsmith why not to have one packaging information file contain all the necessary stuff and make it common format instead of inventing yet another wheel? Having JSON or YAML (because JSON doesn't have comments) will make it possible to process the stuff easier - like inspecting wheels from catalogue application and gathering stats about dependencies.

Every requirements complicates implementation of both client and server side, and the original goal of inventing wheels was to be simple in comparison to EGGs.

[njsmith]

The METADATA format long predates wheels -- it's the same as the PKG-INFO format used in sdists and in .egg-info / .dist-info directories used to describe what packages are installed in a given environment, and its defined in several PEPs. PEP 426 has a proposal for replacing it with JSON, but in practice it's a big mess because you can't just stop supporting the old format (that would mean throwing away every existing version of every existing package, like deleting PyPI and starting over), so adding JSON as an additional format you have to support would just mean that now you have two problems.

If you want to process the stuff then there are packages maintained by the core folks that you can just import and will read these things for you (including all the necessary backcompat hacks etc.). I think packaging is the name of the main package for doing stuff like this.

These questions would be better asked on distutils-sig... people have been trying to incrementally clean this stuff up for years, and it's extremely difficult given all the weird broken old stuff that has to keep working. You're suggesting adding weird broken new stuff, and if packages like you suggest start being distributed in the wild then it'll just make it that much harder to ever clean any of this stuff up.

[techtonik]

@njsmith the argument that METADATA long predates wheels and that's why it should be in wheels is bogus. The argument that METADATA has PEPs and that's why it should be in wheels is bogus as well.

The argument that METADATA support should be abandoned if wheels use different format is bogus, because it is installer that decides how to record information about installed package. If wheels needs to be shipped with precompiled chunks of repository database - it is a a problem on the distutils side. But distutils don't install wheels, so there is no problem here.

JSON or YAML is needed for cross-platform compatibility. Your packaging advice require me to use Python, but my server stack is based on node.js, tools for Linux are written in Perl and dependency analysis software uses Java. If wheels used some extensible metadata format from the start, chances are that we won't need PyPI for Linux at all, because packaging tools on this platform could just automatically map the fields.

I am not sure why we need to move to distutils-sig@ - I can't reply to this list from the web and you've already created specific issue here. So let's try to reach a conclusion - why a wheel with single WHEEL metadata file in a format like https://github.com/toml-lang/toml is bad?

[njsmith]

You say that "JSON or YAML is needed for cross-platform compatibility", but your plan is to unilaterally invent a new format that no-one else uses and with no plan for how to convince them to start using it. If you want your new format to be compatible with, like, pip and PyPI, then you have to talk to distutils-sig and get their buy-in. If you don't care about that, then cool, but at that point you're not... really talking about wheels or python packages at all.

I'm not going to sit here and help you design a new file format that no-one cares about or will use, sorry...

[ogrisel]

I don't understand the aim of opening such an issue on the wheel-builders organizational repo. The wheel format is already well specified by a PEP that has been approved by the Python core developers:

https://www.python.org/dev/peps/pep-0427/

along with a draft PEP for an evolution of the format here:

https://www.python.org/dev/peps/pep-0491/

There is a reference implementation maintained at:

https://bitbucket.org/pypa/wheel

with the documentation published at:

http://wheel.readthedocs.org/en/latest/

If you face specific limitations with the current state of the specification this should be discussed on the distutils-sig mailing list.

If you find bugs in the reference implementation they should be reported on the issue tracker of the wheel project it-self.