Inject local dependency version during publish/build

[q-aaronzolnailucas]

Issue Kind

Change in current behaviour

Description

Support injecting dependency versions during publish/build for multi-project releases.

My usecase

I have a multi-project, single repo situation. The separate projects share a single venv and a top level pyproject.toml that has dev dependencies and configuration for linters/cqa - so that these standards are shared.

the top-level pyproject.toml has
./pyproject.toml

[tool.poetry.dependencies]
product-one = { path = "./product-one", develop = true }
product-two = { path = "./product-two", develop = true }
...

That's all well and good, but product-two depends on product-one. We manage this by using the same release cycle for both.
./product-one/pyproject.toml

version = "1.2.0-dev"
...

./product-two/pyproject.toml

version = "1.2.0-dev"

[tool.poetry.dependencies]
product-one = "~1.2.0"
...

Now, using ~1.2.0 doesn't work for poetry install, with or without allow-prereleases. Specifying "1.2.0-dev" does work, but we'd like to allow different patch/build version differences just to be flexible, and not have to set the version in so many places. The best would be to leave this version as * everywhere, and then have a dependency version inject parameter at build time. This would play nice with versioning plugins too.

Example implementation:

pyproject.toml

[tool.poetry.dependencies]
product-one = { inject-version = true }  # alternatively, inject-version = "tag"

then publish:

poetry publish --inject-version "product-one='~1.2.0'"
# OR
poetry publish --inject-version "~1.2.0'"  # apply same version to every dependency that needs it

This could have the caveat that it only works for path and git dependencies.

Impact

Better multi-project release processes

Workarounds

using sed to fix versions before release pipelines run, modifying build artifacts

[dimbleby]

~1.2.0 doesn't work for poetry install, with or without allow-prereleases

correct, because 1.2.0.dev comes before 1.2.0: so 1.2.0.dev does not satisfy that requirement

I find it hard to follow what you are trying to achieve here but I will take a guess that it is a better fit for a plugin than something that would make sense in poetry proper.

Maybe prototype it out yourself and if you are convinced that it belongs, submit a pull request.

[q-aaronzolnailucas]

@dimbleby shouldn't allow-prereleases = true fix that?

Effectively, the feature I'm requesting is to inject dependency versions at release time. Since poetry has steered clear of using environment variables (#208 #481 ) this would be a neat way to hit one of the usecases that was asked for on those threads.

I'm not going to have capacity or expertise to develop this myself, as a plugin or otherwise.

[dimbleby]

shouldn't allow-prereleases = true fix that?

allow-prereleases = true would allow prereleases that do satisfy the requirement (eg 1.3.0.dev): but a version has to satisfy the requirement first.

injecting dependency versions at release time does not make any sense to me - how do you even know that there is a solution?

if you are not going to develop this then I think it is very likely that no-one else is either.

[q-aaronzolnailucas]

injecting dependency versions at release time does not make any sense to me

Well, I'm not the first to ask and it's standard practice for multiproject releases of other languages.

If you have 2 packages in the same repo, A and B, on the same release cycle, and B depends on A, then you can't release B v1.0 until you've released A v1.0. Currently, steps needed here are:

1. Change version of A/pyproject.toml to "1.0".
2. Build and release A
3. Update version of A dependency in B/pyproject.toml to "~1.0"
4. Change version of B in B/pyproject.toml to "~1.0".
5. Build and release B

It's not great that this pipeline causes such a delta in these files during excecution and is so sensitive to order. Manually writing CI pipelines like this is tiresome. Imagine this happened with a more complicated dependency graph, or with a circular dependency - which is possible. Instead, why not allow dependency injection for packages I'm releasing.

Here's a quick plugin that lets me use env vars from .env files or otherwise which is now my workaround:

import os.path

from typing import TYPE_CHECKING, Any, Dict

import poetry.core.pyproject.toml

from dotenv import load_dotenv
from poetry.plugins import Plugin


if TYPE_CHECKING:
    from cleo.io.io import IO
    from poetry.poetry import Poetry


def read_pyproject_toml(self: poetry.core.pyproject.toml.PyProjectTOML) -> Dict[str, Any]:
    """Patched version of poetry-core's PyProjectTOML.data."""
    if self._data is None:
        if not self.path.exists():
            self._data = {}
        else:
            with self.path.open("rb") as f:
                load_dotenv()
                content = os.path.expandvars(f.read().decode("utf-8"))
                self._data = poetry.core.pyproject.toml.tomllib.loads(content)
    return self._data


class EnvVarPoetryPlugin(Plugin):
    """Poetry plugin to patch the toml parser to interpolate environment variables."""

    def activate(self, _: "Poetry", io: "IO") -> None:
        """Activate the plugin."""
        if io.is_debug():
            io.write_line("<debug>Patching toml parser to interpolate environment variables</debug>")
        self.patch_toml_parse_expandvars()

    @staticmethod
    def patch_toml_parse_expandvars() -> None:
        """Patch the PyProjectTOML.data property to expand environment variables."""
        poetry.core.pyproject.toml.PyProjectTOML.data = property(read_pyproject_toml)  # type: ignore

[dimbleby]

You want to publish a B that depends on A 1.0, before A 1.0 even exists?

IMO this is clearly something that poetry should not support, poetry is all about ensuring that dependencies are consistent and can be resolved.

Exploring a plug-in based approach for this dangerous operation sounds like exactly the right way to go. I think it unlikely that this function will make it into poetry proper. Anyway I stand by the advice that if you don't do it yourself then probably no-one else will either.

[q-aaronzolnailucas]

You want to publish a B that depends on A 1.0, before A 1.0 even exists?
No, I want to publish them at the same time without CI knowing about the dependency graph.

Thanks for the helpful advice anyway - I'll keep the community posted if we develop on this.

[timothyjlaurent]

This is what the Poetry MonoRepo Dependency Plugin Does

Mix it with the Poetry Dynamic Versioning plugin and you can get a new dynamic version for your monorepo packages.

So you build all of your packages they'll all receive the dynamic version based on the git tag and then push them to the pypi repo- You'll then be able to install any of them and their dependencies will be available.