Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Project metadata #1036

Closed
2 tasks done
hynek opened this issue Apr 13, 2019 · 19 comments
Closed
2 tasks done

Project metadata #1036

hynek opened this issue Apr 13, 2019 · 19 comments

Comments

@hynek
Copy link

@hynek hynek commented Apr 13, 2019

  • I have searched the issues of this repo and believe that this is not a duplicate.
  • I have searched the documentation and believe that my question is not covered.

Question

Since poetry stores its metadata like version, homepage etc in pyproject.toml, is there a way for project to introspect themselves after being installed?

There is the "standard" of putting meta data into dunders into the main __init__.py such that they can be introspected at runtime.

My setup.py-based approach is based on parsing those __init__.py s but I could live with inverting that relationship, so is there a way to fill __init__.py data based on the data in pyproject.toml?

FWIW, flit seems to at least extract the __version__ from there.

@pmav99
Copy link

@pmav99 pmav99 commented Apr 13, 2019

For the record to do that, you need to ship the *.toml file with your wheels

@hynek
Copy link
Author

@hynek hynek commented Apr 13, 2019

That would require to depend on both setuptools (for pkg_resources) and toml (to parse it) – no?

@pmav99
Copy link

@pmav99 pmav99 commented Apr 13, 2019

Toml is not really required for simple key-values: #144 (comment)

@hynek
Copy link
Author

@hynek hynek commented Apr 27, 2019

IIRC there were some issues when parsing toml using an ini parser.

Regardless, I really don't think that loading and parsing a file in your __init__.py is a good idea. There should be a better way.

I'd personally prefer if poetry could actually get its metadata from the __init__.py like flit does it for __version__.

@drunkwcodes
Copy link

@drunkwcodes drunkwcodes commented Apr 27, 2019

It's recommended to use pyproject.toml as the source of the version number.
There are a lot of srcs to place version number, including in git tags, which are not in any file.

I think this should not be handled by poetry, which prefers semantic versioning.
The lack of official PEP440 implementation is another cost to complete this feature.

A wiki to gather solutions of this kind of puzzles is probably what poetry needs.

@chrahunt
Copy link

@chrahunt chrahunt commented May 7, 2019

This use case should be covered by importlib_metadata. In your __init__.py:

from importlib_metadata import version

__version__ = version(__package__)

This will use the package data that is persisted in METADATA or PKG-INFO by pip on installation from wheel or sdist.

@chrahunt
Copy link

@chrahunt chrahunt commented May 7, 2019

The specific question here is

is there a way to fill __init__.py data based on the data in pyproject.toml?

The approach I mention above answers this question, since the METADATA/PKG-INFO which are present in the installed package are generated by poetry from the pyproject.toml.

I should also mention that importlib_metadata is not a random package, but a backport of importlib.metadata which will be available in Python 3.8. What I posted will, I believe, be the conventional way for Python packages to provide this metadata moving forward (regardless of what build tool they use).

@drunkwcodes
Copy link

@drunkwcodes drunkwcodes commented May 7, 2019

@chrahunt It is still different from parsing version number from pyproject.toml.

And the path introduces another coupling to pip by a new, distant route which should be prevented.

@chrahunt
Copy link

@chrahunt chrahunt commented May 7, 2019

If we take the question as-stated: "based on the data in pyproject.toml", then I think it is OK to derive it from the metadata which is based on the data in pyproject.toml. Can we think of any use cases that are not covered by using importlib_metadata?

And the path introduces another coupling to pip by a new, distant route which should be prevented.

There is nothing pip-specific here. The mentioned behavior is true for any Python packages as specified in PEP 345 (PKG-INFO for sdist) and PEP 427 (METADATA for wheels). Poetry already implements these and in fact it must in order for the generated packages to be considered valid.

@chrahunt
Copy link

@chrahunt chrahunt commented May 7, 2019

Just to be clear, I'm not suggesting any change in poetry for this, I think that an approach like I mentioned above (or a better one if we can come up with it) needs to be documented somewhere since it will be a common problem poetry users will have.

@drunkwcodes
Copy link

@drunkwcodes drunkwcodes commented May 7, 2019

@chrahunt But pyproject.toml will be also in the package. And the metadata will be in either place. The issue still needs to be open after you posted a even not a workaround.

@chrahunt
Copy link

@chrahunt chrahunt commented May 7, 2019

But pyproject.toml is also in the package.

pyproject.toml is not in the package - to check please run poetry build on a plain poetry-based project and extract the wheel. We would need to include it explicitly as mentioned by @pmav99, but there is not really a need if the required data is in <project>-<version>.dist-info/METADATA, as generated by poetry (and accessible in a generic way using importlib_metadata).

@nickpresta
Copy link

@nickpresta nickpresta commented May 8, 2019

I'd like to add my +1 to the original request and my experience with a confusing bug due to this:

I have a package with a pyproject.toml defined as such (most things omitted for brevity):

version = "1.5.0"

include = ["pyproject.toml", "mypackage/**/*.py"]

In mypackage/version.py, I have the following:

import pkgutil
import re

version_regexp = re.compile(r'''^version = "([^"]*)"''', re.M)

data = pkgutil.get_data(__package__, "../pyproject.toml")
match = version_regexp.search(data.decode("utf-8"))
if match:
    __version__ = match.group(1)
else:  # pragma: no cover
    raise RuntimeError("Unable to find version string")

This seems to work and doesn't depend on a toml package for parsing (at the risk of the regexp failing).

When installing this package, my site-packages looks like:

$ ls venv/lib/python3.7/site-packages/mypackage*
venv/lib/python3.7/site-packages/chunnelx:
version.py

venv/lib/python3.7/site-packages/mypackage-1.5.0.dist-info:
INSTALLER  METADATA   RECORD     WHEEL

In addition to:

$ cat venv/lib/python3.7/site-packages/pyproject.toml
[tool.poetry]
name = "mypackage"
version = "1.5.0"

# rest omitted

This works "fine" until I installed another package that is doing the same thing (e.g. mypackage2). Now, the pyproject.tomlfile insite-packagesgets overriden during installation, potentially in a non-deterministic way depending on package installation order, withmypackage2's pyproject.tomland the code inmypackage` is using the wrong version.

It seems like an ideal solution would be some way to use data_files or additional meta-data functionality.

Both this ticket and #890 seem to cover the related functionality to make something like this easier/possible.

There is an open PR: #901 -- is there something I can do to help get this merged in (reviews, testing, etc)?

@drunkwcodes
Copy link

@drunkwcodes drunkwcodes commented May 8, 2019

The override can be avoided by not getting the version number in runtime.
But the metadata should be checked if in sync with __init__.py manually in development.

I don't know if it is a fasion to get version number like this thread describes but I hate it (without effective verification.) You are exploiting poetry version.

@chrahunt
Copy link

@chrahunt chrahunt commented May 8, 2019

In #144 (also linked above) it was mentioned that bumping version in both pyproject.toml and __init__.py is planned. Maybe it just needs an implementation?

Something like poetry version --check would also be good, to verify in CI.

@drunkwcodes
Copy link

@drunkwcodes drunkwcodes commented May 9, 2019

@chrahunt I mean the checksum in metadata in the case of getting version in runtime that you mentioned.

By the way, how is it going about your pyenv maintenance?

@finswimmer finswimmer mentioned this issue Sep 18, 2019
2 tasks
lainiwa added a commit to lainiwa/my-ip that referenced this issue Oct 12, 2019
From now on version will only be declared in pyproject.toml.
Comment on importlib use with poetry:
python-poetry/poetry#1036 (comment).
@stale
Copy link

@stale stale bot commented Nov 13, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Nov 13, 2019
@merwok
Copy link

@merwok merwok commented Nov 13, 2019

This is still relevant 🤖 Poetry could get a way to extract metadata from code (like flit and setuptools), call an arbitrary function, or (if it takes that stance) document how to use importlib.metadata to get version.

@stale stale bot removed the stale label Nov 13, 2019
@finswimmer
Copy link
Member

@finswimmer finswimmer commented Feb 8, 2020

The solution given by @chrahunt using importlib_metadata IS the solution to the issue. It is not a workaround .

@finswimmer finswimmer closed this Feb 8, 2020
br3ndonland added a commit to br3ndonland/test3 that referenced this issue Jun 20, 2020
During `poetry install`, Poetry installs the project as if it were a
package itself. You can then pull the version from the package metadata.
It's a little tricky to get `importlib_metadata` to work with Docker.
You have to remove `--no-root` from the _Dockerfile_ and add
`importlib-metadata` to prod dependencies, and you still might get
errors after that.

python-poetry/poetry#144
python-poetry/poetry#1036
br3ndonland added a commit to br3ndonland/algorithms that referenced this issue Jul 20, 2020
br3ndonland/test3@227ab28

During `poetry install`, Poetry installs the project as if it were a
package itself. You can then pull the version from the package metadata.
It's a little tricky to get `importlib_metadata` to work with Docker.
You have to remove `--no-root` from the _Dockerfile_ and add
`importlib-metadata` to prod dependencies, and you still might get
errors after that.

python-poetry/poetry#144
python-poetry/poetry#1036
br3ndonland added a commit to br3ndonland/inboard that referenced this issue Oct 18, 2020
So far, testing has been limited to the Python modules. However, the
pyproject.toml is an important source of metadata for the project. This
commit will use toml (pytest dependency) to load pyproject.toml, and
Pydantic (FastAPI dependency) will be used for data validation. Unit
tests will verify the presence of select fields in the pyproject.toml.

It is also possible to use `importlib.metadata` to read other package
metadata, as explained in python-poetry/poetry#1036. The metadata
originate in pyproject.toml, and becomes available after Poetry
installs the root project package. The `importlib.metadata` approach
works in a virtualenv, in which the Poetry root project is installed,
but is error prone in Docker, because the root project is usually not
installed. Parsing the TOML is a more durable alternative, because it
works even if the root project is not installed.

The `importlib.metadata` version calculation in `inboard/__init__.py`
was not necessary, so it will be removed.
dobraczka added a commit to dobraczka/kiez that referenced this issue Jun 17, 2021
- Relying on importlib as per python-poetry/poetry#1036 (comment)
haasad added a commit to haasad/commodore that referenced this issue Jun 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants