Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metadata and options are now could be set in configuration files #862

Merged
merged 21 commits into from Dec 8, 2016
Merged

Metadata and options are now could be set in configuration files #862

merged 21 commits into from Dec 8, 2016

Conversation

@idlesign
Copy link
Member

@idlesign idlesign commented Nov 26, 2016

Hi,

It is useful to be able to have metadata and other options declared in configuration files (e.g. setup.cfg). That not only allows some automation, but also reduces bolerplate code in some cases.
See also #394.

This chage allows us to have something like this in our config files:

[metadata]
name = some_name
version = attr: some_package.VERSION
description = Some description
long_description = file: README
keywords = one, two
license = BSD 3-Clause License

[metadata.classifiers]
Framework :: Django
Programming Language :: Python :: 3.5

[options]
zip_safe = False
include_package_data = True
packages = find:
scripts = 
  bin/one.py
  bin/two.py

[options.package_data]
* = *.txt, *.rst
hello = *.msg

[options.extras_require]
pdf = ReportLab>=1.2; RXP
rest = docutils>=0.3; pack ==1.1, ==1.3

Currently every metadata and other options are supported (or I'd like to believe that). More examples in test_config.py.
What I'm concerned with though is whether we need to sandbox file: directive which is supported by some options for security considerations (shouldn't it be impossible to read data from outside).

If overall design and implementation is fine I could update the documentation.

Below are supported ways of value definition for various options.

Terms used

  • list-comma: dangling list or comma-separated values string
  • list-semi: dangling list or semicolon-separated values string
  • bool: True is 1, yes, true
  • dict: list-comma where keys from values are separated by =
  • attr: value could be read from module attibute
  • file: value could be read from a file
  • section: values could be read from a dedicated section

Metadata

  • version: string or attr (iterables, callables are supported; others - through str())
  • platforms (alias: platform): list-comma
  • keywords: list-comma
  • provides: list-comma
  • requires: list-comma
  • obsoletes: list-comma
  • classifiers (alias: classifier): file, section, or list-comma
  • license: file or string
  • description (alias: summary): file or string
  • long_description: file or string
  • author_email (alias: author-email): string
  • url (alias: home-page): string

Options

  • zip_safe: bool
  • setup_requires: list-semi
  • install_requires: list-semi
  • extras_require: section
  • entry_points: file, section
  • use_2to3: bool
  • use_2to3_fixers: list-comma
  • use_2to3_exclude_fixers: list-comma
  • convert_2to3_doctests: list-comma
  • scripts: list-comma
  • eager_resources: list-comma
  • dependency_links: list-comma
  • tests_require: list-semi
  • include_package_data: bool
  • packages: list-comma or find:
  • package_dir: dict
  • package_data: section
  • exclude_package_data: section
  • namespace_packages: list-comma
@idlesign
Copy link
Member Author

@idlesign idlesign commented Nov 26, 2016

Fails on unrelated test_easy_install.py, and test_sandbox.py.

@idlesign
Copy link
Member Author

@idlesign idlesign commented Dec 1, 2016

@jaraco What I can do for this issue to be reviewed?

@jaraco
Copy link
Member

@jaraco jaraco commented Dec 3, 2016

I'm trying to re-start the Travis build now that the test failures have been fixed in master, but it's not restarting readily. I restarted the appveyor tests, and the Python 2.7 build is now passing, but not the Python 3.

@jaraco
Copy link
Member

@jaraco jaraco commented Dec 3, 2016

The Travis builds are running now and are showing the same - test failures on Python 3.

@jaraco
Copy link
Member

@jaraco jaraco commented Dec 3, 2016

I like this a lot. The implementation is well-factored and well-thought-out.

I'd like @ncoghlan's thoughts - does this feature fit well into the plans for packaging metadata? Does it conflict or align with the goals of PEP 426 or similar PEPs?

A few aspects I'll want to consider before acceptance:

  • Fix test failures on Python 3.
  • Update docs (as you've offered to do), probably following additional concurrence on the implementation. An important question to answer in the docs: What is the precedence when a value is defined both in setup.cfg and setup.py imperatively? I'd expect setup.py values to take precedence.
  • Use from __future__ import absolute_import, unicode_literals and maybe others in the new modules for better future compatibility.
  • Assess this behavior in light of pbr. IIRC, pbr also uses setup.cfg to store declarative metadata. Does this feature interfere with that functionality?
  • Run the integration tests. There's a tests/test_pypi.py which can be run manually. Let's run those tests as well (which may answer the question about pbr).
@idlesign
Copy link
Member Author

@idlesign idlesign commented Dec 3, 2016

Py3 failures seem to be connected with pbr indeed, since those 'author_email': 'openstack-dev@lists.openstack.org must have come from there.

I'll peep into pbr to deside how to handle those cases. Yet most probably the only option is not to raise Unknown distribution option exception and to skip unknown keys silenlty.

I'd expect setup.py values to take precedence.

Yes, that's how it's currently done.

@idlesign
Copy link
Member Author

@idlesign idlesign commented Dec 3, 2016

Skimmed through pbr docs. There seems to be another option: we can handle pbr notation of metadata section. And it's easy enough.

Yet I'm not sure whether it'll the right thing to do, since keys used there do not fully correspond with setup() argument names (classifier vs classifiers; author-email vs author_email; etc.) that we are all used to.

So I propose that we skip unknown keys, and handle both forms for those clashing (it seems to be keywords only).

What do you think?

@idlesign
Copy link
Member Author

@idlesign idlesign commented Dec 4, 2016

Items from #862 (comment) seem to be covered except docs update, which I'll do after @ncoghlan review.

There's one more thing I'm concerned with: it's seems that it'll the the right thing to expose even minimal API, so that automation tools could easily get already parsed and preprocessed data from configs without ado.

@ncoghlan
Copy link
Member

@ncoghlan ncoghlan commented Dec 4, 2016

@jaraco When we decided to go in the pyproject.toml direction for the build system independent toolchain bootstrapping, that left setup.cfg free for setuptools/d2to1/pbr to use as you wish.

@idlesign
Copy link
Member Author

@idlesign idlesign commented Dec 4, 2016

Now with PEP-518, pbr and d2to1 in mind, I think it will be wise not only to handle metadata keys introduced by the latter two, but also to keep closer to .toml. It should simplify the task of transition for users.

@idlesign
Copy link
Member Author

@idlesign idlesign commented Dec 5, 2016

@jaraco Seems to be quite complete. Docs require proof reading, I guess.

@jaraco jaraco merged commit 3b50de3 into pypa:master Dec 8, 2016
2 checks passed
2 checks passed
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
jaraco added a commit that referenced this pull request Dec 8, 2016
@anthrotype
Copy link

@anthrotype anthrotype commented Dec 8, 2016

Thanks for working on this! 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants