Using Poetry with Nox

[danieleades]

• I have searched the issues of this repo and believe that this is not a duplicate.

Issue

I've been trying to migrate an existing project to Poetry. It is using Tox to test a matrix of different package versions. The difficulties of trying to get Poetry and Tox to play nicely together for matrices of package versions is fairly well-documented (see https://stackoverflow.com/questions/59377071/how-can-i-get-tox-and-poetry-to-work-together-to-support-testing-multiple-versio)

I'm playing around with Nox as an alternative, and it's looking very promising for being easier to integrate with Poetry, but there are still a few caveats.

I have the following Nox script for testing against a couple of different python interpreters, and a couple of different versions of sphinx-

import nox

@nox.session(python = ["3.5", "3.6"]
@nox.parametrize("sphinx", ["^2.4", "^3.0"])
def tests(session, sphinx)
    session.run('poetry', 'add', f'sphinx@{sphinx}', external=True)
    session.run('poetry', 'install', external=True)
    session.run('pytest')

this works perfectly, except that it modifies the pyproject.toml file in my project's root. Is there a better way for me to arrange this?

[danieleades]

see also #1745

[danieleades]

cross-posted to wntrblm/nox#347

[max-hoffman]

I ran into the same problem this afternoon, decided to just use tmp directories so that I could modify and discard the repo repeatedly:

from distutils.dir_util import copy_tree
import os
import pathlib
import shutil
import tempfile

import nox

@nox.session(python=["3.8", "3.7", "3.6"])
@nox.parametrize("numpy", ["1.19.2", "^1.18.4", "^1.17.5"])
@nox.parametrize("pillow", ["7.0.0", "^6.0.0", "^5.4.0"])
def tests(session, numpy, pillow):
    with tempfile.TemporaryDirectory() as tmp_dir:
        base_dir = pathlib.Path(os.path.abspath("."))
        copy_tree(base_dir, tmp_dir)
        os.chdir(tmp_dir)

        args = session.posargs or ["--cov=flywheel"]
        session.run('poetry', 'add', f"numpy@{numpy}", f"pillow@{pillow}", external=True)
        session.run("poetry", "install", external=True)
        session.run("pytest", *args)

[danieleades]

@max-hoffman that's a neat solution. I was considering just caching and replacing the pyproject.toml and poetry.lock.
Tell me, are you using virtualenvs inside your project? Any interactions there?

[danieleades]

@sdispater, the Nox maintainers have reached out and I suspect would be keen to investigate a tighter integration between Poetry and Nox. Is this something you'd be interested in looking into?
Specifically, the use case is around using Poetry and Nox to run tests over a dependency matrix

[max-hoffman]

@danieleades a try/except/finally block where the original pyproject.toml and poetry.lock are always restored would probably be more efficient than what I wrote. I am running the code with poetry global mode in a docker image. I'd need to take a closer look at what caches are maintained between session runs, but I'm starting with fresh nox and poetry caches in the environment I'm running in.

[danieleades]

@max-hoffman the try/except/finally approach doesn't work for me.

i've got

import nox
from pathlib import Path

python_versions = [
    "3.8",
]

sphinx_versions = [
    "^2.3",
]

def cache(file: Path) -> (Path, str):
    with open(file) as f:
        cache = f.read()
    return (file, cache)

def dump(cache: (Path, str)):
    (file, data) = cache
    with open(file, 'w') as f:
        f.write(data)

project_file = cache('./pyproject.toml')
lockfile = cache('./poetry.lock')

try:

    @nox.session(python=python_versions)
    @nox.parametrize("sphinx", sphinx_versions)
    def tests(session, sphinx):
        session.run('poetry', 'add', f'sphinx@{sphinx}', external=True)
        session.run('poetry', 'install', external=True)
        session.run('nosetests')

finally:
    dump(project_file)
    dump(lockfile)

try as I might, I can't get that finally block to run. not sure how Nox handles exiting...

[danieleades]

I ran into the same problem this afternoon, decided to just use tmp directories so that I could modify and discard the repo repeatedly:

from distutils.dir_util import copy_tree
import os
import pathlib
import shutil
import tempfile

import nox

@nox.session(python=["3.8", "3.7", "3.6"])
@nox.parametrize("numpy", ["1.19.2", "^1.18.4", "^1.17.5"])
@nox.parametrize("pillow", ["7.0.0", "^6.0.0", "^5.4.0"])
def tests(session, numpy, pillow):
    with tempfile.TemporaryDirectory() as tmp_dir:
        base_dir = pathlib.Path(os.path.abspath("."))
        copy_tree(base_dir, tmp_dir)
        os.chdir(tmp_dir)

        args = session.posargs or ["--cov=flywheel"]
        session.run('poetry', 'add', f"numpy@{numpy}", f"pillow@{pillow}", external=True)
        session.run("poetry", "install", external=True)
        session.run("pytest", *args)

i was doing something like this and decided to throw in a print to double-check the interpreter version. I found that is was using the version installed by Poetry (3.8 for me) in every case, regardless of the version configured in the Nox session. Can you confirm whether you see the same behaviour?

[layday]

You could do something like this:

from shutils import copytree

import nox


@nox.session(python=["3.8", "3.7", "3.6"])
@nox.parametrize("numpy", ["1.19.2", "^1.18.4", "^1.17.5"])
@nox.parametrize("pillow", ["7.0.0", "^6.0.0", "^5.4.0"])
def tests(session, numpy, pillow):
    tmp_dir = session.create_tmp()
    copytree(base_dir, tmp_dir)

    session.install("poetry")
    session.run("poetry", "add", f"numpy@{numpy}", f"pillow@{pillow}")
    session.run("poetry", "install")

    args = session.posargs or ["--cov=flywheel"]
    session.run("pytest", *args)

The issue with this is that it introduces all of Poetry's dependencies in the test session's virtual env.

[layday]

You could also use nox-poetry which exports a requirements file with Poetry and defers to pip for installing packages.

[aidan-melen]

We should all thank @cjolowicz for this cookiecutter template taht implements nox-poetry: https://cookiecutter-hypermodern-python.readthedocs.io/en/2020.10.15.1/

He also wrote nox-poetry plugin

[Secrus]

Poetry team can't and won't keep track of all the tools that are used and without it, maintenance of such documentation would be impossible.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.