Use dependency groups for development and documentation dependencies

[km-64]

pyproject.toml used to list development and documentation dependencies (pytest, zuban, sphinx, etc.) as optional dependencies. This means that it's possible to install development-only extras from PyPI.

Provides-Extra: serial, simulator, documentation, development, all
-- https://pypi.org/project/pymodbus/

It doesn't really make sense to install dev/docs dependencies as an end user. Dependency groups (PEP 735) can be used instead to isolate those dependencies.

This specification defines dependency groups, a mechanism for storing package requirements in pyproject.toml files such that they are not included in project metadata when it is built.

Dependency groups are suitable for internal development use-cases like linting and testing, as well as for projects which are not built for distribution, like collections of related scripts.
-- https://packaging.python.org/en/latest/specifications/dependency-groups/

documentation and devlopment have been moved into dependency groups. Additionally development can be renamed to dev so that UV will automatically install them when working in the project (see: https://docs.astral.sh/uv/concepts/projects/dependencies/#development-dependencies)

[km-64]

I can also update the README with up to date information for how to setup the dev environment with uv

[janiversen]

Using dependency groups instead of just declaring the group as optional is a valid change.

Updating readme to include instructions to use uv should be done in a seperate pr.

[janiversen]

If possible ci changes are preferred in a isolated PR.

you can probably make the CI pr first and then update this PR.

[janiversen]

Note: the all= seems wrong it no longer installs simulator and pyserial.

[km-64]

@janiversen does it make sense to have all as an optional dependency too? I moved it into a dependency group in 4cc857e (this PR), but that means that if you wanted to install serial and simulator, you would have to do pymodbus[serial,simulator]. Would it be useful to have pymodbus[all]?

Having that as an option would also make it easier if just using pip for development. To install all dependencies in the current state of this PR with pip would be pip install -e ".[server,simulator]" --group all. If there is an all extra it could be simpler to use pip install -e ".[all]" --group all.

Edit, I just saw your comment:

Note: the all= seems wrong it no longer installs simulator and pyserial.

I can revert the commit removing add as an optional

[janiversen]

The current description in readme should still be correct without modification.

I am not sure I like the added "--group all", it complicates something that was simple.

[km-64]

Yeah I can simplify it. It would be confusing to have a group and an extra both named all. These are the changes that would have to be made to the README. Mainly to use groups with pip and to use dev instead of development.

diff --git a/README.rst b/README.rst
index 709a7e4c..d20f6cbd 100644
--- a/README.rst
+++ b/README.rst
@@ -216,11 +216,11 @@ or the bleeding edge::

 Install required development tools in editable mode::

-    pip install -e ".[development]"
+    pip install -e ".[all]" --group dev

 Install all (allows creation of documentation etc) in editable mode::

-    pip install -e ".[all]"
+    pip install -e ".[all]" --group documentation

 .. note::
    The use of the ``-e`` (editable) flag is recommended when making changes.

I'll also add some more instructions for how to work with uv since it detects the dev group secifically.

[janiversen]

Using "dev" instead of "development" is quite ok, but when when instructing pip to install "all" it should do so, the "--group" is confusing.

it is also confusing that "all" only refers to part of the installation...it is not clear what is part of "all" and what is part of group.

I am getting less convinced that using groups actually helps.

[km-64]

I can also organize the dependencies in the dev group into subgroups and include them in dev. So they can be grouped by test, lint, types, etc.

[dependency-groups]
dev = [
    {include-group = "test"},
    {include-group = "lint"},
    {include-group = "build"},
    {include-group = "type-checking"},
]
test = [
    "coverage>=7.14.3",
    "pytest>=9.1.1",
    "pytest-asyncio>=1.4.0",
    "pytest-cov>=7.1.0",
    "pytest-profiling>=1.7.0;python_version<'3.13'",
    "pytest-timeout>=2.3.1",
    "pytest-xdist>=3.6.1",
    "pytest-aiohttp>=1.1.1",
]
lint = [
    "codespell>=2.4.2",
    "pylint>=4.0.6",
    "ruff>=0.15.20",
]
build = [
    "build>=1.5.0",
    "twine>=6.2.0",
]
type-checking = [
    "types-Pygments",
    "types-pyserial",
    "zuban>=0.9.0"
]
documentation = [
    "recommonmark>=0.7.1",
    "Sphinx>=7.3.7;python_version<'3.12'",
    "Sphinx>=9.1.0;python_version>='3.12'",
    "sphinx-rtd-theme>=3.1.0"
]

Projects like FastAPI do that (https://github.com/fastapi/fastapi/blob/255b912928904e3ba5980425a54d6837c8bd1a1c/pyproject.toml#L122-L187)

[km-64]

Also I see that build is installed. If you have uv you can run uv build which is a bit faster (4s -> 1s).

[janiversen]

Let me put it simple...,when instructing pip to install "all" it should do so without extra parameters.

also ".[x] --group y" to install x and y are just confusing.

no need to split development into smaller groups, either you want to development or not.

[km-64]

I changed pip install -e ".[development]" to pip install -e ".[all]" --group dev because you'd probably want the all the extras in devlopment. Excluding all from the dev install would mean that pyserial and aiohttp wouldn't be installed.

To keep it exactly the same, pip install -e . --group dev could be done instead, just means that the extras have to be installed in another command.

I could also just add pymodbus[all] to the dev group so that this isn't a problem.

The problem with the current docs is that they say pip install -e ".[development]", but if you run tests that use pyserial or aiohttp they would fail due to import errors.

[km-64]

I am getting less convinced that using groups actually helps.

The main reason to use them is to stop dev only dependencies from being installed from PyPI, like doc dependencies being included in pymodbus[all]

[janiversen]

Again when you write ".[all]" it means ALL so "--group dev" do not make sense.

in the current version ".[development]" installs everything needed for development, but if you want to test with pyserial or make documentation that is not enough, which is why you have "all".

[janiversen]

But if you specify "all" it is because you want ALL....there are no "all-runtime" it would not make sense.

[janiversen]

We currently have:

   pip install pymodbus[x,y,z]"

and for convenience:
  pip install pymodbus[all]

That can be replaced without problems (IF there are a real advantage somewhere) with:

   pip install pymodbus --group x,y,z
or
  pip install pymodbus --group all

But any combination of "pymodbus[x,y,z] --group a,b,c" is just confusing and not something likely to be accepted.

[km-64]

What exactly should pymodbus[all] mean? As it is now it means install serial simulator and documentation development, but that only makes sense in development. If you wanted to install all the extras for pymodbus into your project pymodbus[all] would include unnecessary dependencies.

[km-64]

extras are additional package metadata declared in the [project.optional-dependencies] table. They provide names for lists of package specifiers which are published as part of a package’s metadata, and which a user can request under that name, as in pip install 'foo[bar]' to install foo with the bar extra.

Because extras are package metadata, they are not guaranteed to be statically defined and may require a build system to resolve. Furthermore, definition of a [project.optional-dependencies] indicates to many tools that a project is a package, and may drive tool behaviors such as validation of the [project] table.

For projects which are packages, extras are a common solution for defining development dependencies, but even under these circumstances they have downsides:

Because an extra defines optional additional dependencies, it is not possible to install an extra without installing the current package and its dependencies.
Because they are user-installable, extras are part of the public interface for packages. Because extras are published, package developers often are concerned about ensuring that their development extras are not confused with user-facing extras.
-- https://peps.python.org/pep-0735/#limitations-of-extras

[janiversen]

Pymodbus[all] means just as it works now, ALL ! It is just a convenience for developers, who typically install everything.

For runtime pymodbus only depend on pyserial, so "all" for runtime is just the same as "pyserial".

Please do not forget, this is a library not an app !! The comment about package managers worrying about "user-facing" vs "development" think in apps, not in a library.

Again I am not against "--group", but I am against the confusion that comes from mixing "pymodbus[x,y,z] with "--group".

[km-64]

It's possible to not have extras and group mixed when using pip.
pip -e . --group dev (install pymodbus and the dev) or uv eqlivalent uv sync (dev installed automatically)

I could make a dev-all group to install everything

[dependency-groups]
# ...
dev-all = [
	"pymodbus[all]",  # serial and simulator
	{include-group = "dev"},
	{include-group = "documentation"},
]

It could also be called all, which is allowed (groups and extras can have the same name), but that's maybe a bit confusing, so it can be called dev-all to be more expicit.

Note that this specification does not forbid having an extra whose name matches a Dependency Group.

[km-64]

README would have this to say (no extras and group mixing):

diff --git a/README.rst b/README.rst
index 709a7e4c..b9b75f9b 100644
--- a/README.rst
+++ b/README.rst
@@ -174,8 +174,6 @@ Pymodbus offers a number of extra options:

 - **serial**, needed for serial communication
 - **simulator**, needed by pymodbus.simulator
-- **documentation**, needed to generate documentation
-- **development**, needed for development
 - **all**, installs all of the above

 which can be installed as::
@@ -216,11 +214,11 @@ or the bleeding edge::

 Install required development tools in editable mode::

-    pip install -e ".[development]"
+    pip install -e . --group dev

 Install all (allows creation of documentation etc) in editable mode::

-    pip install -e ".[all]"
+    pip install -e . --group dev-all

 .. note::
    The use of the ``-e`` (editable) flag is recommended when making changes.

[janiversen]

This do not solve the base problem! you forget that pymodbus[pyserial,simulator] is still an option.

The installer still needs to know what is being installed with pymodbus[x,y,z] and what can only be installed with "--group x,y,z". I really do not see any advantages in confusing the installer.

You could argue, that with the current method we should add "all-only-runtime", but that is quite over the top.

[km-64]

If all is only meant for development then it should just be made into a group. End users can install pymodbus[serial,simulator] for all the extras in that case. Some projects use all as an extra like FastAPI.

[janiversen]

I agree that end users can install pyserial and simulator.

But on some platforms you cannot install "all" because aiohttp and/or sphinx are not available.

[km-64]

But on some platforms you cannot install "all" because aiohttp and/or sphinx are not available.

Do you mean that for dev only or also end users?

[janiversen]

endusers cannot install simulator and developers cannot install simulator and/or documentation

[janiversen]

The installation proces is not something invented overnight it has evolved over years and have been stable for the last couple of years.....so changing it is possible but you need to remember all the corner cases.

[km-64]

endusers cannot install simulator and developers cannot install simulator and/or documentation

Not sure what you mean by this. Do you mean that they shouldn't be able to do that or that they tried to with this PR and can't?

[km-64]

Using pip would work like this now:

 Install required development tools in editable mode::

-    pip install -e ".[development]"
+    pip install -e . --group dev

 Install all (allows creation of documentation etc) in editable mode::

-    pip install -e ".[all]"
+    pip install -e . --group all

Or with uv:
uv sync
uv sync --group all or uv sync --all-groups like in ci (the all group is mainly for pip which doesn't have something like --all-groups)

and end users can install the package pymodbus, pymodbus[serial], pymodbus[simulator], pymodbus[serial,simulator].

[janiversen]

So a developer, who cannot install simulator and documentation but wants pyserial, would need to do:

pip install -e ".[pyserial] --group dev

if correct the problem remains the same.

you either need to drop --group or .[x,y,z]

they cannot coexist without causing confusion.

I use that combination quite often to solve serial problems.

[janiversen]

Ps. if dropping the current version there need to be a very good reason, because that is a bigger change, which affect all downstream fork and most likely a couple of app developers as well.

alternatively you need to be able to continue using the current version and --group is an optional feature.

[km-64]

pip install -e ".[pyserial] --group dev that would be the case to install pyserial with dev in this PR in its current state. I'm not sure why this would be a huge issue unless it would cause breaking changes for downstream projects (they might have to update their pip commands for automations or similar).

You mentioned something about installing on an ESP board (I assume ESP32 or similar?). Are you installing pymodbus on the actual hardware? Like using micropython? Just curious about that.

I could take a look at those downstream projects and help out if it's a problem.

Alternatively we could support both (groups just reference the extras) so that either option will work. uv sync could install a dev group that just installs pymodbus[development]. Later support for the dev only extras could be removed.

[km-64]

I think I know a potential issue with using groups. If someone uses requirements.txt they can install an editable version of pymodbus (I guess from a submodule or something?), but can't specify groups since it's not possible to do that in requirements.txt.

[janiversen]

requirements.txt is not a problem for pymodbus, but might be for downstream.

You do not see the change as a problem, and I would agree if there was a real advantage in doing it....but right now there was only argument is "end-users could install dev tools without wanting it, however it is clearly stated in the documentation that all means all. If they do not read the documentation they do not know about all.

problem with downstream projects is that you do not know if they are active....and some have not forced, but use submodule to avoid listíng pymodbus as a requirement (my interpretation).

[km-64]

Yeah I guess we can just leave pyproject.toml as it is. Maybe it's worth adding a dev dependency group so that uv automatically installs development

[janiversen]

making uv install automatically sounds like a good idea.

you also wanted to document uv usage in the readme.