Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add TOML config loading, autodetect pyproject.toml #374

Closed
wants to merge 13 commits into from
Closed

Add TOML config loading, autodetect pyproject.toml #374

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
af76101
add (pyproject).toml config parsing
bollwyvl Oct 31, 2020
596c7fd
add packaging/dev metadata stuff for toml
bollwyvl Oct 31, 2020
42ad151
fix config file ordering, handle Paths
bollwyvl Oct 31, 2020
09aba9e
update config docs
bollwyvl Oct 31, 2020
dcf9f6e
add mapping from proposed TOML to ConfigParser, more docs
bollwyvl Nov 18, 2020
caac79f
hoist import detection, remove some early returns
bollwyvl Nov 18, 2020
e678632
more docs tweaks
bollwyvl Nov 18, 2020
2fb5f80
add tool.doit canary check, warning with supported libs
bollwyvl Nov 18, 2020
fca5cc3
Merge remote-tracking branch 'upstream/master' into add-toml
bollwyvl Mar 14, 2021
0f2b021
remove deprecated pytoml (no longer used by flit)
bollwyvl Mar 14, 2021
615edd1
Merge remote-tracking branch 'upstream/master' into add-toml
bollwyvl Mar 14, 2021
87cd2b3
merge upstream
bollwyvl May 17, 2021
cff267c
fix tools vs tool in docs
bollwyvl May 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions dev_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ pyflakes
pytest>=5.4.1
coverage>=4.0,<5
doit-py>=0.4.0
toml>=0.10.1
17 changes: 15 additions & 2 deletions doc/cmd_other.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -242,17 +242,30 @@ This can used to display desktop notifications, so you do not need to keep
an eye in the terminal to notice when tasks succeed or failed.

Example of sound and desktop notification on Ubuntu.

Contents of a `pyproject.toml` file:

.. code-block:: toml

[tool.doit.commands.auto]
success_callback = """
notify-send -u low -i /usr/share/icons/gnome/16x16/emotes/face-smile.png "doit: success"; aplay -q /usr/share/sounds/purple/send.wav
"""
failure_callback = """
notify-send -u normal -i /usr/share/icons/gnome/16x16/status/error.png "doit: fail"; aplay -q /usr/share/sounds/purple/alert.wav
"""


Contents of a `doit.cfg` file:

.. code-block:: INI
.. code-block:: ini

[auto]
success_callback = notify-send -u low -i /usr/share/icons/gnome/16x16/emotes/face-smile.png "doit: success"; aplay -q /usr/share/sounds/purple/send.wav
failure_callback = notify-send -u normal -i /usr/share/icons/gnome/16x16/status/error.png "doit: fail"; aplay -q /usr/share/sounds/purple/alert.wav




``watch`` parameter
^^^^^^^^^^^^^^^^^^^^^

Expand Down
101 changes: 95 additions & 6 deletions doc/configuration.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,94 @@ The name can be seem from ``doit help`` output::
-f ARG, --file=ARG load task from dodo FILE [default: dodo.py] (config: dodoFile)


pyproject.toml
--------------

`doit` configuration can be read from `pyproject.toml <https://www.python.org/dev/peps/pep-0518/>`_
under the `tool.doit` namespace. In the future, especially if a TOML parser becomes
part of the python standard library, this will become the preferred configuration source,
and may gain features not available in the legacy `doit.cfg`.

.. note::

As a TOML parser is _not_ yet part of the standard library, a third-party package is
required, one of:

- `toml <https://pypi.org/project/toml/>`_
- `tomlkit <https://pypi.org/project/tomlkit/>`_


TOML vs INI
^^^^^^^^^^^

While mostly similar, `TOML <https://toml.io>`_ differs from the INI format
in a few ways:

- all strings must be quoted with `'` or `"`
- triple-quoted strings may contain new line characters (`\n`) and quotes
- must be saved as UTF-8
- integers and floating point numbers can be written without quotes
- boolean values can be written unquoted and lower-cased, as `true` and `false`

Unlike "plain" TOML, `doit` will parse pythonic strings into their correct types,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? I think this is in anti-feature. And I could not see where this comes from, all python libs work like this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a hangover from being able to reuse the existing .ini parsing logic, which only speaks string.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh. i see... I think that is not acceptable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Welp, that config object (wherever it gets read from or built dynamically) gets handed around throughout the entire running application, it appears, and (only) the cmd_options know how to do the marshaling. It seems like changing (or even detecting the use of) that behavior would require deprecation of the ini format altogether, and a formal data schema throughout all these things... sounds very 1.0ish.

e.g. `"True"`, `"False"`, `"3"`, but using "native" TOML types may be preferable.


tool.doit
^^^^^^^^^

The `tool.doit` section may contain command line options that will be used
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess mentioned this before...

Why not just doit instead of tool.doit? For me it seems unnecessary verbose.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is specified by PEP 518.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see it is their recommendation... although I dont really understand/agree why limit to only 2 top tables.

So another question. Why put doit configuration on pyproject.toml instead of going with out own file doit.toml?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why put doit configuration on pyproject.toml

Indeed, that has a solid question, and mypy has fought back viciously, for some reason. But my feeling is that the more tools I can cram into a single, consensus-driven file, the better I'll be able to automate the creation (through schema, etc), updating, etc. vs code.

doit.toml?

A fine idea as well, and could be added to the list. I'd imagine this one wouldn't have the tool.doit prefix.

(if applicable) by any commands.

Example setting the DB backend type:

.. code-block:: toml

[tool.doit]
backend = "json"

All commands that have a `backend` option (*run*, *clean*, *forget*, etc),
will use this option without the need for this option in the command line.


tools.doit.commands
^^^^^^^^^^^^^^^^^^^

To configure options for a specific command, use a section with
the command name under `tools.doit.commands`:

.. code-block:: toml

[tools.doit.commands.list]
status = true
subtasks = true


tools.doit.plugins
^^^^^^^^^^^^^^^^^^

Check the :ref:`plugins <plugins>` section for an introduction
on available plugin categories.


tools.doit.tasks
^^^^^^^^^^^^^^^^

To configure options for a specific task, use a section with
the task name under `tool.doit.tasks`:

.. code-block:: toml

[tool.doit.tasks.make_cookies]
cookie_type = "chocolate"
temp = "375F"
duration = 12

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mentioned about being able to define tasks in TOML, right?

Have you thought about what the name would be? I would thing doit.tasks. So maybe get a different name for setting task parameters doit.params.<task_name>, what do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would thing doit.tasks. So maybe get a different name for setting task parameters doit.params.<task_name>, what do you think?

Again, the naming is just trying to keep the new format as parallel as possible while both config formats are supported.

I don't recall ever actually having used the task params, but I'm sure someone would want to! If anything, I'd imagine putting everything this PR has done thus far under tool.doit.config, leaving the top-level namespace clear for things like tasks....

You mentioned about being able to define tasks in TOML, right?

I had imagined this as a separate loader, e.g. TomlLoader which could be enabled via pyproject.toml#/tool/doit/(config)/loader which would then read pyproject.toml (again) and actually populate tasks. The toml loader could default to pyproject.toml, but, lacking inheritance, etc. it seems like a project might want paths = [path="pyproject.toml", ref="tool.doit.tasks"].

For the actual task definitions, I'm imagining the high road would, whether such a loader was part of this repo or standalone:

  • define and ship a JSON schema stored as JSON, (though built from TOML, perhaps, for readability)
    • ideally, this would support some measure of extensibility, for, e.g. config_changed without going full YAML tag nonsense (which TOML won't support anyway)
    • some other things are nasty, e.g.
      • enumerating files
        • maybe have a paths definition section which can use something like pathspec, and which tasks can reference by pointer
      • create_after
  • pick a default place in pyproject.toml, e.g. #/tool/doit/tasks or #/tool/doit-toml-tasks
  • optionally enable pre-validation (jsonschema is not exactly lightweight, as a dependency or a runtime)


doit.cfg
--------

`doit` uses an INI style configuration file
`doit` also supports an INI style configuration file
(see `configparser <https://docs.python.org/3/library/configparser.html>`_).
Note: key/value entries can be separated only by the equal sign `=`.

Expand All @@ -41,20 +123,24 @@ GLOBAL section
The `GLOBAL` section may contain command line options that will
be used (if applicable) by any commands.

Example setting the DB backend type::
Example setting the DB backend type:

.. code-block:: ini

[GLOBAL]
backend = json

All commands that has a `backend` option (*run*, *clean*, *forget*, etc),
will use this option without the need this option in the command line.
All commands that have a `backend` option (*run*, *clean*, *forget*, etc),
will use this option without the need for this option in the command line.


commands section
^^^^^^^^^^^^^^^^

To configure options for a specific command, use a section with
the command name::
the command name:

.. code-block:: ini

[list]
status = True
Expand All @@ -72,13 +158,16 @@ per-task sections
^^^^^^^^^^^^^^^^^

To configure options for a specific task, use a section with
the task name prefixed with "task:"::
the task name prefixed with "task:":

.. code-block:: ini

[task:make_cookies]
cookie_type = chocolate
temp = 375F
duration = 12


configuration at *dodo.py*
--------------------------

Expand Down
6 changes: 6 additions & 0 deletions doc/dictionary.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,13 +164,15 @@ Sylvain
TC4
TC5
TODO
TOML
TaskError
TaskFailed
TaskLoader
TaskLoader2
Tpng
Trento
USD
UTF
UptodateCalculator
Uz
VCS
Expand Down Expand Up @@ -301,6 +303,7 @@ hgrc
html
https
hunspell
ini
img
init
initializer
Expand Down Expand Up @@ -395,6 +398,7 @@ pytest
python2
python3
pythonic
pyproject
quickstart
refactor
regex
Expand Down Expand Up @@ -458,6 +462,8 @@ titlewithactions
to's
toc
toctree
toml
tomlkit
toolchain
toolchains
traceback
Expand Down
35 changes: 27 additions & 8 deletions doc/extending.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,13 +128,22 @@ config plugin

To enable a plugin create a section named after the plugin category.
The value is an entry point to the python class/function/object
that implements the plugin. The format is <module-name>:<attribute-name>.
that implements the plugin. The format is `<module-name>:<attribute-name>`.

Example of command plugin implemented in the *class* `FooCmd`,
located at the module `my_plugins.py`::
Example of command plugin configured in `pyproject.toml`, implemented in
the *class* `FooCmd`, located at the module `my_plugins.py`:

[COMMAND]
foo = my_plugins:FooCmd
.. code-block:: toml

[tools.doit.plugins]
COMMAND = { foo = "my_plugins:FooCmd" }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about?

[doit.plugins.foo]
kind = "command"
obj = "my_plugins:FooCmd"

Better name for "obj" ?

Copy link
Contributor Author

@bollwyvl bollwyvl May 17, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, naming things is hard. If it was purely data, the more verbose form would feel good, but i think in this case spareness wins.

I believe this could also be written

[tool.doit.plugins.COMMAND]
foo = "my_plugins:FooCmd"

(have updated docs to reflect both)


Similarly, in `doit.cfg`:

.. code-block:: ini

[COMMAND]
foo = my_plugins:FooCmd

.. note::

Expand Down Expand Up @@ -171,13 +180,23 @@ Creates a custom task loader. Check :ref:`loader <custom_loader>` section
for details on how to create a new command.

Apart from getting the plugin you also need to indicate which loader will be
used in the `GLOBAL` section of your config file.
used. In the `tool.doit` section in `pyproject.toml`:

.. code-block:: toml

.. code-block:: INI
[tool.doit]
loader = "my_loader"

[tool.doit.plugins]
LOADER = { my_loader = "my_plugins:Loader" }


In the `GLOBAL` section of `doit.cfg`:

.. code-block:: ini

[GLOBAL]
loader = my_loader

[LOADER]
my_loader = my_plugins:MyLoader

Loading