Get metadata for unbuilt package

[jaraco]

There are many use-cases where a packager / installer may need to know something about a project prior to building/installing it. Maybe they need to inspect the dependencies or find the name of the package being installed or perform some checks on the version that would be produced.

In the distutils-only world, the challenge would rest on distutils to supply interfaces to expose that information. Distutils adds support for setup.py --name and --version and --requires (although that last one doesn't work because nobody is using metadata 1.1).

In the PEP 517 world, since arbitrary builders might exist, distutils/setuptools cannot be responsible for defining the interface for such use-cases, as a caller can no longer assume the project can be built by setuptools.

For now, projects are limping along by relying on distutils and will find themselves unable to wean themselves off of setup.py unless these use-cases have a solution in the PEP 517 world.

[dstufft]

Isn't this just prepare_metadata_for_build_wheel. from PEP 517?

[jaraco]

Yes, maybe. And the fallback for when that optional method isn't supplied is for the build tool to actually build a wheel and query the metadata from the result. Now all that's needed is a tool that (a) orchestrates this process and (b) extracts the metadata reliably from the result.

Even with this tool, in this comment, @Jonast expresses some concerns that may arise. Having prepare_metadata_for_build_wheel would likely address those concerns.

[pfmoore]

Now all that's needed is a tool that (a) orchestrates this process and (b) extracts the metadata reliably from the result.

This should be covered by https://github.com/pypa/pep517

It probably needs some more real world use to iron out the details and improve the interface, but certainly the intention is that it provides a standard interface to PEP 517, for tools to use. Please feel free to try it out and help out with ensuring it meets real world needs.

[jaraco]

I'm looking forward to checking it out.

[ghost]

(ignore this, I was confused)

@jaraco as I explained in the other ticket, I don't think anything that requires build_ext to work (which I assume will be the case for building a wheel) is of much use. It's just too likely to cause trouble with the Cython setup. For a pure Android-targeted package, there is no guarantee build_ext will work on a regular x64 Linux host, so relying on this shouldn't be necessary

Edit: it may also require installing dependencies for .pxd files, as one of some other caveats. It's just not a good idea to require the package to build just to be able to read out the dependency metadata

[dstufft]

The idea here is that build backends that support something like build_ext or with complex build requirements will implement prepare_metadata_for_build_wheel, which allows them to produce metadata without the rest of their machinery, but that build backends which can guarantee that building a wheel is both fast and will always work, can opt not to do that.

[ghost]

(ignore this as well, was still confused)

Will setuptools/distutils do that? (I use setuptools.setup + distutils.command.build_ext in a package where you can not just build a wheel without installing dependencies first, due to external .pxd cimports - and installing the deps to get the deps list is really a bit silly) Because otherwise this just seems like a theoretical fix

Edit: oh right. I misunderstood, so if prepare_metadata_for_build_wheel is present I essentially just don't need to build it. makes sense 👍

[ghost]

@jaraco I've looked into some of the pep517 example code and played around with it, but I ran into multiple showstoppers:

1. Pep517HookCaller now has a new parameter (which btw is not properly passed in the README example) which is the build_backend. However, requiring me to specify this is entirely pointless, since the whole goal is to make it work for any backend/package type, so why would I need to specify this? How am I expected to obtain it without manually hardcoding all different setup.py, pyproject.toml... file types and opening them all up? Doesn't seem practical at all

2. hooks.build_sys_requires throws an AttributeError

3. hooks.get_requires_for_build_wheel simply returns nothing without wheel installed. The package I'm looking at doesn't need wheel though, and I can definitely install it without wheel, so surely I should be able to get the requirements?

4. hooks.get_requires_for_build_wheel throws a very weird subprocess.CalledProcessError with no wheel installed

In the end, the best result with hardcoded "setuptools" backend I was able to get was []. This result is wrong / useless.

So how can I get the requirements for an uninstalled package? Because with nothing in the pep517 README I was actually able to do so... maybe with wheel around, but should that be reasonably a requirement? If it is, why does pep517 not depend on it & ensure it's installed? What's the point of all this functionality if it doesn't actually work in this practical case?

[ghost]

Now with wheel installed, I'm getting this:

AttributeError: module 'setuptools' has no attribute 'build_wheel'

I haven't gotten a single method to work properly... 😢 and I'm not saying they're broken, but the README example of pep517 right now seems outdated/broken/useless for anyone who doesn't know in-depth how the systems work

[ghost]

Okay, with setuptools.build_meta as build_backend it worked. But again, why do I need to know/specify this? Isn't the entire point of pep517 that it figures out the dependencies/info for a package from a folder on its own, rather than requiring me to hardcode knowledge for specific build systems?

[pfmoore]

pep517 is a helper library for people building install frontends (as per PEP 517). The frontend should include the necessary code to read pyproject.toml, and the correct build_backend should either be specified by the project in its pyproject.toml, or the frontend can supply a default (pip defaults to setuptools.build_meta, for example, to provide backward compatibility).

It sounds like you're trying to use pep517 directly, rather than building a frontend, so that may be why you're finding things confusing.

Having said that, yes the documentation is weak at the moment. Help in improving it is welcome, of course...

[ghost]

@pfmoore but I was redirected here to use it to get dependencies of an installed package. If that's not what pep517 is for, what should I be using then? (see also the name of this ticket)

The frontend should include

Well I don't have that. I really just want to get the dependencies, and all I have is a folder path

Edit: wait I'm not sure I understand the pip tidbit. You're saying pip doesn't parse pyproject.toml either? I never used such alternatives

[pfmoore]

Oh wait, sorry. I hadn't read the full thread here (I'd seen a number of pings on different issues and got confused).

In my view (and it's only my view!) "the intention is that it (pep517) provides a standard interface to PEP 517, for tools to use" as I said before. The key here is "for tools to use". It sounds like you want a tool that gets the dependencies of a package. pep517 isn't that tool, it's a library that could be used to build that tool.

what should I be using then?

I'm not aware of any tool that does that right now.

wait I'm not sure I understand the pip tidbit

Sorry, maybe I wasn't clear. Pip does parse pyproject.toml.

[ghost]

@jaraco got any input on this? specifically on what I should be using if not pep517

[ghost]

@jaraco I implemented this now, but the big question seems to be where it should go: pypa/pyproject-hooks#44

A ticket for the discussion of having a basic package examination library for unbuilt/not installed packages: #247

Just to reiterate this, pep517, as it is now, is IMHO a bad match for this task because it's not universal/highlevel enough. To be precise, pep517 cannot easily spit out absolute basics like name or dependencies without quite some additional code, and it cannot deal with remote references which is needed for dependency analysis & only focuses on spilling out sdist and/or wheel of a local source tree

[jaraco]

@Jonast, In pypa/pyproject-hooks#48, I have a working implementation for preparing the metadata as an in-memory zip-file and in importlib_metadata!67, I've demonstrated using that functionality to load the metadata. I'm still working out a few kinks, but feel free to review/comment.

[jaraco]

This issue was mostly solved in pep517.meta and later ported to build.util.project_wheel_metadata and used by pytest-checkdocs and jaraco.packaging.