Add pyproject.toml by default

[FlorianWilhelm]

It seems that pyproject.toml is here to stay. Some tools like black already use it to store its configurations. Thus every new project should have one with the content:

[build-system]
requires = ["wheel", "setuptools", "setuptools-scm"]
build-backend = "setuptools.build_meta"

[ssbarnea]

TBH, when I found this ticket I decided that pyscaffold is not for me. I wonder if there is a better maintained project layout generator for python, one that already adopted stuff like this.

[FlorianWilhelm]

Currently, I have not much time working on PyScaffold in my spare time. Contributors and even maintainers are always welcome.

[abravalheri]

Hi @FlorianWilhelm, I found out one of these days that adding pyproject.toml to a project is not exactly backwards compatible... It changes the way setuptools and pip think about the project because it is supposed to be built in isolation...

Other then breaking some builds, it also might affect us a bit in terms of development. Particularly we have to avoid circular dependencies in the

[build-system]
requires = ["setuptools >= 40.6.0", "wheel"]  # <-- I assume pyscaffold and it's extensions would go here.

section.

Do you think that this can affect the way we list the extensions as extra optional dependencies for PyScaffold? Should we remove the formal dependency in the extensions (leaving them implicit and only try catching ImportError) in order to keep supporting PyScaffold[all] and avoid the cycles?

I also saw recently that setuptools is implementing a new entrypoint (setuptools.finalize_distribution_options), to allow plugins to register via pyproject.toml. When the key in this entrypoint appears in a [tool:<key of the entrypoint>] section header inside pyproject.toml, it is equivalent to using a setup_keyword=...

I was wondering, if there is any way to configure setuptools_scm entirely from the pyproject.toml file with its primitives... Now that we are deprecating pytest-runner integration, it is just Sphinx and setuptools_scm that are preventing PyScaffold to not be required during the build... Sphinx can be easily integrated into tox... and we could include the configuration for setuptools_scm into the pyproject.toml template... This way, PyScaffold would not be needed during the build, just when generating and updating projects.

This simplifies a lot, speeds up the build and avoid the self-bootstrapping thing when building PyScaffold itself...

Of course, we could go completely the other direction as well... Now that the API of setuptools is standardised by a PEP we could wrap it and simplify all the configuration by abstracting...

My final doubt is, should we also move PyScaffold configs from setup.cfg to pyproject.toml? It seems that lot's of projects are doing it... It just is unfortunate that you have spent a lot of energy on the configupdater. The good side is that tomlkit can also keep comments and format unchanged...

[FlorianWilhelm]

Hi @abravalheri, thanks for looking into this. My gut feeling is that we should start simplifying things for setup.py, i.e. deprecating pytest-runner integration, Sphinx integration as well as checking out if setuptools_scm can be somehow included with the help of pyproject.toml. In this case, PyScaffold would no longer be needed as a built dependency just as you described it. This would also solve problems people had with PyScaffold when checking out some repo from some internal git repo and trying to build a package without internet access.
This would also allow us to use more dependencies in PyScaffold itself, e.g. click to reduce our code around cli parsing and introducing some interactive option would be easier. Right now that would be hard since all those dependencies would be needed for building someone's project.

Do you think with the new setuptools entrypoint, you mentioned, it would be possible to completely switch to pyproject.toml and get rid of setuptools.cfg completely in a standardized way (i.e. without any hacks, only using setuptools new functionality?)? I would support that and rather make things simpler by not using configupdater any more.

In general, I also wonder if the update feature of PyScaffold is really used that often. A lot of complexity in our code comes from it and I also guess it still has quite some bugs as we tested only a few most common combinations. For version 4.0 I would favor concentrating on the core functionality of PyScaffold, i.e. providing some project template with everything you need for some serious coding in a standard way.

It would really be a major change and reorg of the current codebase. No more wrapping of setuptools_scm, getting rid of all vendorized dependencies. It feels like losing a lot of ballast. What do you think?

[abravalheri]

Hi @FlorianWilhelm, I have question regarding setuptools_scm: how does the custom formatting you encoded inside PyScaffold differs from setuptools_scm default (guess-next-dev + node-and-date)? Is it too bad to just use the defaults or use one of the pre-sets (that can be written as plain strings in setup.cfg/pyproject.toml)?

Right now on my terminal in PyScaffold, if I do:

python setup.py --version  # => 3.2.3.post0.dev37+ga37efb2
python -m setuptools_scm  # => Guessed Version 3.2.4.dev37+ga37efb2

And if I create an arbitrary tag:

git tag -a v3.3a1-dev0 -m "Test"
python setup.py --version  # => v3.3a1.dev0
python -m setuptools_scm  # => Guessed Version 3.3a1.dev0

So I see that PyScaffold keeps the old version and adds a .post0 while the default of setuptools_scm is to use the next version + .dev... Apart from preferences, does this affect PyPI in anyway? Will PyPI offer the next.dev version to install instead of the latest stable? (I have the impression that it won't...)
If we find that the pre-sets of setuptools_scm are not good enough one alternative would also try to contribute directly to the project with a new pre-set...

Even if we adopt pyproject.toml by default we still will need to maintain both setup.py and setup.cfg. Editable installs require a setup.py and setuptools still do not support listing dependencies and entrypoints in the toml file.

A few other comments 😝

• click is cool, specially for interactive stuff. But I do like how argparse enforces the separation of the application logic and the terminal interface.
• what are the parts of the update feature in PyScaffold that you don't like? Maybe I can give a look and try some refactoring... I understand that is a pain migrating from one version to the other, but I am very excited with the perspective of "adding extensions" on the fly. E.g.: "decided to start using pre-commit in an existing PyScaffold project?" => "no problem, just run putup -u --pre-commit .". "Already generated a project, but forgot to add CI?" => "putup -u --travis .". "Was using travis, but now need to test on windows?" => "why not give Cirrus CI a go? putup -u --cirrus .". This, together with being able to save a default configuration for PyScaffold and avoid typing the same options always, would be the killer features in v4 for me...

[abravalheri]

btw, this post is quite nice: https://www.scivision.dev/pyproject-toml-vs-setup-py/

[FlorianWilhelm]

Hi Anderson,

what I don't like about the default version guessing of setuptools_scm is that the successor of version 3.2.3 doesn't need to be 3.2.4. It could also be version 3.3. In this case, you have a lot of dev versions for 3.2.4 and none for 3.3 in your artifact store. We always set up our CI/CD that each commit gets built and pushed to some artifact store. Then it's pulled and installed and only then unit tests are run. This helps to find errors that would not have been found if unit tests only ran before the build. It gets worse if I tag a version with 1.0. After some commit, setuptools_scm will call it 1.1.dev0+g9e..., not 1.0.1.dev0 or something. This means that my artifact store will have dev versions of 1.1 and later if I tag manually 1.0.1 I will get versions 1.0.2.dev.... In total, I have now newer dev versions with a lower version number! If you use pip install --pre to get the latest dev version from the artifact store, you end up getting old versions. This is pretty bad and inconsistent.

So, in general, I find the post0 versioning better but could live to go with the defaults if it simplifies things when usingpyproject.toml. For years it seems I am the only one having problems with the version guessing of setuptools_scm :-) But maybe we can convince @RonnyPfannschmidt to include the post0 versioning as an additional option directly.

Regarding your statement

Editable installs require a setup.py and setuptools still do not support listing dependencies and entrypoints in the toml file.

Will this change in the near future? I am not sure if having a mixture of setup.cfg and pyproject.toml is really helpful for end-users but it definitely is more error-prone. Maybe things will resolve soon and we just wait a little more to then fully adopt pyproject.toml? What do you think?

I agree with argparse and click. Do you think that click can still be used to cleanly separate application logic and the terminal interface? It's a really powerful package and I guess we could save a lot of code, especially when providing an interaction feature, by using it.

What I don't like about the update feature is the fact that we are checking what the old PyScaffold version was and then we have those migration rules that for instance change bits and pieces in setup.cfg. In principle, they work but the number of tests for them in the future will grow a lot and I am not sure many PyScaffold users are actually making use of it. The usage you describe to add more features or activate an option is totally fine.

[RonnyPfannschmidt]

Setuptools_scm already ships both support for pyproject.toml and post release version numbers, im happy to help with the details

[abravalheri]

Hi @FlorianWilhelm, as @RonnyPfannschmidt pointed out we could also try the post-release scheme:

$ python -c 'from setuptools_scm import get_version; print(get_version(version_scheme="post-release"))'
3.2.3.post39+g7d412e9
$ python setup.py --version  # PyScaffold
3.2.3.post0.dev39+g7d412e9

Indeed it is not exactly the same thing. I suppose the way PyScaffold works right now all the non tagged versions are considered dev-only, while setuptools_scm post-release strategy would result in valid 'sub-minor' releases...

We could settle for this middle ground, unless @RonnyPfannschmidt is open for a contribution of another strategy (something like post-dev?)

A doubt: @FlorianWilhelm, is there any difference for PyScaffold's local scheme and setuptools_scm's node-and-date? I went through the code but couldn't figure it out...

[abravalheri]

Regarding setuptools, I am sure that the team is working a lot for this integration to happen... It might take some time though, because they have to move carefully to not break the entire ecosystem... And they just "finished" documenting setup.cfg recently (pypa/setuptools#1765, pypa/setuptools#2146). I don't imagine they have short-term plans for moving the way you declare dependencies and entrypoints from setup.cfg to pyproject.toml... I know that there is an effort to standardise editable installs and eliminate the need setup.py... but I don't see setup.cfg going anywhere so soon. At the same time it seems that setuptools will start encouraging people to have a pyproject.toml. So unfortunately we are stuck with at least 2 files for the time being 😝 ...

There is another detail... Once the editable installs issue is standardised to not depend on setup.py, we might decide to drop that file, however that implies that somewhere (most probably tox) we will need to add the following code:

$ pip install -q pep517
$ mkdir dist
$ python -m pep517.build .

So I believe we either keep setup.py or start enabling tox by default.

[abravalheri]

We can find a way to use click, maybe some high-order functions could do the trick (never understood exactly how the custom classes in click work, do HOF is my default workaround). We should have a look on the PyInquirer for the interactive feature as well, super cool.

For updates, we could only document the steps for migrating between the versions and remove automatic updates from old PyScaffolds that are not backward compatible, raising an error with the URL for the manual update once we detect a setup.cfg with an incompatible version number.

We can discuss those 2 topics in a different thread and keep this one for pyproject.toml.

[RonnyPfannschmidt]

pypa/pip#8212 is hopefully the way towards no longer needing a setup.py for editable installs