Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Project metadata #1036

Open
hynek opened this issue Apr 13, 2019 · 17 comments

Comments

Projects
None yet
5 participants
@hynek
Copy link

commented Apr 13, 2019

  • I have searched the issues of this repo and believe that this is not a duplicate.
  • I have searched the documentation and believe that my question is not covered.

Question

Since poetry stores its metadata like version, homepage etc in pyproject.toml, is there a way for project to introspect themselves after being installed?

There is the "standard" of putting meta data into dunders into the main __init__.py such that they can be introspected at runtime.

My setup.py-based approach is based on parsing those __init__.py s but I could live with inverting that relationship, so is there a way to fill __init__.py data based on the data in pyproject.toml?

FWIW, flit seems to at least extract the __version__ from there.

@pmav99

This comment has been minimized.

Copy link

commented Apr 13, 2019

For the record to do that, you need to ship the *.toml file with your wheels

@hynek

This comment has been minimized.

Copy link
Author

commented Apr 13, 2019

That would require to depend on both setuptools (for pkg_resources) and toml (to parse it) – no?

@pmav99

This comment has been minimized.

Copy link

commented Apr 13, 2019

Toml is not really required for simple key-values: #144 (comment)

@hynek

This comment has been minimized.

Copy link
Author

commented Apr 27, 2019

IIRC there were some issues when parsing toml using an ini parser.

Regardless, I really don't think that loading and parsing a file in your __init__.py is a good idea. There should be a better way.

I'd personally prefer if poetry could actually get its metadata from the __init__.py like flit does it for __version__.

@drunkwcodes

This comment has been minimized.

Copy link

commented Apr 27, 2019

It's recommended to use pyproject.toml as the source of the version number.
There are a lot of srcs to place version number, including in git tags, which are not in any file.

I think this should not be handled by poetry, which prefers semantic versioning.
The lack of official PEP440 implementation is another cost to complete this feature.

A wiki to gather solutions of this kind of puzzles is probably what poetry needs.

@chrahunt

This comment has been minimized.

Copy link

commented May 7, 2019

This use case should be covered by importlib_metadata. In your __init__.py:

from importlib_metadata import version

__version__ = version(__package__)

This will use the package data that is persisted in METADATA or PKG-INFO by pip on installation from wheel or sdist.

@drunkwcodes

This comment has been minimized.

Copy link

commented May 7, 2019

@chrahunt It is off-topic.

@chrahunt

This comment has been minimized.

Copy link

commented May 7, 2019

The specific question here is

is there a way to fill __init__.py data based on the data in pyproject.toml?

The approach I mention above answers this question, since the METADATA/PKG-INFO which are present in the installed package are generated by poetry from the pyproject.toml.

I should also mention that importlib_metadata is not a random package, but a backport of importlib.metadata which will be available in Python 3.8. What I posted will, I believe, be the conventional way for Python packages to provide this metadata moving forward (regardless of what build tool they use).

@drunkwcodes

This comment has been minimized.

Copy link

commented May 7, 2019

@chrahunt It is still different from parsing version number from pyproject.toml.

And the path introduces another coupling to pip by a new, distant route which should be prevented.

@chrahunt

This comment has been minimized.

Copy link

commented May 7, 2019

If we take the question as-stated: "based on the data in pyproject.toml", then I think it is OK to derive it from the metadata which is based on the data in pyproject.toml. Can we think of any use cases that are not covered by using importlib_metadata?

And the path introduces another coupling to pip by a new, distant route which should be prevented.

There is nothing pip-specific here. The mentioned behavior is true for any Python packages as specified in PEP 345 (PKG-INFO for sdist) and PEP 427 (METADATA for wheels). Poetry already implements these and in fact it must in order for the generated packages to be considered valid.

@chrahunt

This comment has been minimized.

Copy link

commented May 7, 2019

Just to be clear, I'm not suggesting any change in poetry for this, I think that an approach like I mentioned above (or a better one if we can come up with it) needs to be documented somewhere since it will be a common problem poetry users will have.

@drunkwcodes

This comment has been minimized.

Copy link

commented May 7, 2019

@chrahunt But pyproject.toml will be also in the package. And the metadata will be in either place. The issue still needs to be open after you posted a even not a workaround.

@chrahunt

This comment has been minimized.

Copy link

commented May 7, 2019

But pyproject.toml is also in the package.

pyproject.toml is not in the package - to check please run poetry build on a plain poetry-based project and extract the wheel. We would need to include it explicitly as mentioned by @pmav99, but there is not really a need if the required data is in <project>-<version>.dist-info/METADATA, as generated by poetry (and accessible in a generic way using importlib_metadata).

@nickpresta

This comment has been minimized.

Copy link

commented May 8, 2019

I'd like to add my +1 to the original request and my experience with a confusing bug due to this:

I have a package with a pyproject.toml defined as such (most things omitted for brevity):

version = "1.5.0"

include = ["pyproject.toml", "mypackage/**/*.py"]

In mypackage/version.py, I have the following:

import pkgutil
import re

version_regexp = re.compile(r'''^version = "([^"]*)"''', re.M)

data = pkgutil.get_data(__package__, "../pyproject.toml")
match = version_regexp.search(data.decode("utf-8"))
if match:
    __version__ = match.group(1)
else:  # pragma: no cover
    raise RuntimeError("Unable to find version string")

This seems to work and doesn't depend on a toml package for parsing (at the risk of the regexp failing).

When installing this package, my site-packages looks like:

$ ls venv/lib/python3.7/site-packages/mypackage*
venv/lib/python3.7/site-packages/chunnelx:
version.py

venv/lib/python3.7/site-packages/mypackage-1.5.0.dist-info:
INSTALLER  METADATA   RECORD     WHEEL

In addition to:

$ cat venv/lib/python3.7/site-packages/pyproject.toml
[tool.poetry]
name = "mypackage"
version = "1.5.0"

# rest omitted

This works "fine" until I installed another package that is doing the same thing (e.g. mypackage2). Now, the pyproject.tomlfile insite-packagesgets overriden during installation, potentially in a non-deterministic way depending on package installation order, withmypackage2's pyproject.tomland the code inmypackage` is using the wrong version.

It seems like an ideal solution would be some way to use data_files or additional meta-data functionality.

Both this ticket and #890 seem to cover the related functionality to make something like this easier/possible.

There is an open PR: #901 -- is there something I can do to help get this merged in (reviews, testing, etc)?

@drunkwcodes

This comment has been minimized.

Copy link

commented May 8, 2019

The override can be avoided by not getting the version number in runtime.
But the metadata should be checked if in sync with __init__.py manually in development.

I don't know if it is a fasion to get version number like this thread describes but I hate it (without effective verification.) You are exploiting poetry version.

@chrahunt

This comment has been minimized.

Copy link

commented May 8, 2019

In #144 (also linked above) it was mentioned that bumping version in both pyproject.toml and __init__.py is planned. Maybe it just needs an implementation?

Something like poetry version --check would also be good, to verify in CI.

@drunkwcodes

This comment has been minimized.

Copy link

commented May 9, 2019

@chrahunt I mean the checksum in metadata in the case of getting version in runtime that you mentioned.

By the way, how is it going about your pyenv maintenance?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.