Read DOIT_CONFIG from (pyproject).toml

[bollwyvl]

Thanks for doit! Per the instructions on the template, this has a corresponding thread on the mailing list.

Elevator Pitch

Allow doit, if toml is installed, to look for DOIT_CONFIG in pyproject.toml by default, or any given TOML file.

Examples

TOML is quite similar, but a bit more flexible, than the ConfigParser syntax. A pyproject.toml most like the test doit.cfg:

[build-system]
requires = ["setuptools >= 40.6.0", "wheel"]
build-backend = "setuptools.build_meta"

[tools.doit.DOIT_CONFIG.GLOBAL]
optx = 6
opty = 7

[tools.doit.DOIT_CONFIG.COMMAND]
foo = "tests.sample_plugin:MyCmd"

Though users may prefer alternatives, with more commas:

[build-system]
requires = ["setuptools >= 40.6.0", "wheel"]
build-backend = "setuptools.build_meta"

[tools.doit.DOIT_CONFIG]
GLOBAL = {
  optx = 6,
  opty = 7,
}
COMMAND = {
  foo = "tests.sample_plugin:MyCmd"
}

Motivation

The python tool community is slowly making progress out of dotfile hell by standardizing on pyproject.toml. black, pytest, tox (to an extent, and not yet compatible with pytest graah) and others have already carved out their own respective namespace under tools.

Additional tools like dephell and flakehell make managing this content reasonable, and integrating tools that don't yet support pyproject.toml.

Technical Concerns

pyproject.toml was originally only intended for one heading, [build-system], for allowing isolated build environments, and avoiding the chicken-and-egg of setup_requires in setup.py and setup.cfg. However, once it became clear that this file wasn't going anywhere, the official stance softened to allow any number of extra headings under the tool key. The current explanation provides some insight. Of note, a project can't yet entirely replace setup.py for a "bog standard" setuptools build, but this is in-progress on PEP621.

The TOML language itself is not without quirks, and the underlying spec is not yet 1.0, but is already widely used for Officially Important Files in the julia and rust, and now python, packaging communities, and is unlikely to change in a backwards-incompatible way.

A TOML parser is not part of the python standard library, but the python toml package supports the 1.0.0rc* spec, and is already relied upon by a number of tools (e.g. black, flit).

Proposal

I'm happy to work up a PR that:

/blob/f61d4ed9d587575fd20ee5ece1674ddc3088ee2a/src/_pytest/config/findpaths.py#L65-L79))

• Adds an extras for toml to setup.py
  • With a reasonable version pin
• Adds pyproject.toml to the default list of config files (right after doit.cfg, which should still always "win")
• Pre-screens pyproject.toml (or whichever .toml the user might specify) for the magic DOIT_CONFIG string
  • If found...
    • try to import toml (a la [pytest](https://github.com/pytest-dev/pytest
      • if this fails, show a warning
    • otherwise parses and extracts parsed["tools"]["doit"]["DOIT_CONFIG"][*]
      • shims into the final ConfigParser output
• Tests!
• Docs!

Alternatives

• make toml a hard requirement (a la black)
  • 👍 don't have to specify an install
  • 👎 extra, unused dependency for many projects
• vendor toml
  • 👍 don't have to specify an install
  • 👎 have to manage version, etc.
  • 👎 vendoring is generally annoying from a downstream packaging perspective
    • disclaimer: I'm one of the conda-forge/doit-feedstock maintainers
• create a first- or third-party plugin, e.g. doit-config-loader-toml
  • 👎 potential version mismatches
• do-nothing: don't support pyproject.toml at all
  • 👍 no work!
  • 👍 no extra dependencies
  • 👎 still have a random doit.cfg rolling around in my repo

Follow-on

• with toml available, it may make sense to be able to load simple tasks from pyproject.toml in a predictable manner
  • especially those that are configured in pyproject.toml
  • these could be intermixed with complex tasks (e.g. advanced yields, etc) from dodo.py or wherever
  • perhaps rather than the approach on the yaml loader, provide something that, initially, more closely matches the raw strings, etc. that can instantiate a task
    • there are challenges with things like file globs

[build-system]
requires = ["setuptools >= 40.6.0", "wheel"]
build-backend = "setuptools.build_meta"

[tools.black]
line-length = 80

[tool.flakehell]
max_line_length = 90
show_source = true

[tool.flakehell.plugins]
pyflakes = ["+*", "-F401"]

[tools.doit.DOIT_CONFIG.GLOBAL]
backend = "sqlite3"
verbosity = 2
par_type = "thread"

# this uses the "nested tables" approach, allowing for groups
[[tools.doit.tasks.lint]]  
name = "black"
doc = "format python code"
actions = ["black > build/black.ok"]
targets = ["build/black.ok"]
file_dep = ["foo.py"]              
# this is a whole other story, or perhaps, with some more opinions...
# file_dep = [{ glob = "**/*.py" }]

[[tools.doit.tasks.lint]]
name = "flakehell"
doc = "check for python flake"
actions = ["flakehell > build/flakehell.ok"]
targets = ["build/flakehell.ok"]
file_dep = ["foo.py"]

• such a spec could be formally encoded with, e.g. a JSON schema, and reused for other well-behaved declarative formats.

If you made it this far, thanks for reading!

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[schettino72]

Thanks for detailed proposal.

Agree with proposal of TOML as an extra dependency over the alternatives.

I guess I would go with a mixed approach more like flakehell in your example.
Example bellow should cover everything support by current cfg file:

# DOIT_CONFIG / GLOBAL
[tools.doit]
optx = 6
opty = 7

# add PLUGINS
[tools.doit.plugins]
COMMAND = {
  foo = "tests.sample_plugin:MyCmd"
}

# COMMAND config
[tools.doit.cmd.list]
status = True
subtasks = True

# TASK config
[tools.doit.task.make_cookies]
cookie_type = chocolate
temp = 375F
duration = 12

What do you think?

Regarding "Follow-on" about creating tasks from donfig... I am happy to see things evolve in this direction.
I guess this will involve a bit more discussion and work.
I need to actually study TOML (I never heard about nested tables).
And better to have a separate issue for that after initial TOML support is added.