Using Nox with Poetry

[danieleades]

cross-posted to python-poetry/poetry#2920

I've been trying to migrate an existing project to Poetry. It is using Tox to test a matrix of different package versions. The difficulties of trying to get Poetry and Tox to play nicely together for matrices of package versions is fairly well-documented (see https://stackoverflow.com/questions/59377071/how-can-i-get-tox-and-poetry-to-work-together-to-support-testing-multiple-versio)

I'm playing around with Nox as an alternative, and it's looking very promising for being easier to integrate with Poetry, but there are still a few caveats.

I have the following Nox script for testing against a couple of different python interpreters, and a couple of different versions of sphinx-

import nox

@nox.session(python = ["3.5", "3.6"]
@nox.parametrize("sphinx", ["^2.4", "^3.0"])
def tests(session, sphinx)
    session.run('poetry', 'add', f'sphinx@{sphinx}', external=True)
    session.run('poetry', 'install', external=True)
    session.run('pytest')

this works perfectly, except that it modifies the pyproject.toml file in my project's root. Is there a better way for me to arrange this?

[theacodes]

How does it modify the file?

…

On Tue, Sep 15, 2020 at 12:25 PM danieleades ***@***.***> wrote: cross-posted to python-poetry/poetry#2920 <python-poetry/poetry#2920> I've been trying to migrate an existing project to Poetry. It is using Tox to test a matrix of different package versions. The difficulties of trying to get Poetry and Tox to play nicely together for matrices of package versions is fairly well-documented (see https://stackoverflow.com/questions/59377071/how-can-i-get-tox-and-poetry-to-work-together-to-support-testing-multiple-versio ) I'm playing around with Nox <https://nox.thea.codes/en/stable/index.html> as an alternative, and it's looking very promising for being easier to integrate with Poetry, but there are still a few caveats. I have the following Nox script for testing against a couple of different python interpreters, and a couple of different versions of sphinx- import nox @nox.session(python = ["3.5", "3.6"] @nox.parametrize("sphinx", ["^2.4", "^3.0"])def tests(session, sphinx) session.run('poetry', 'add', f'sphinx@{sphinx}', external=True) session.run('poetry', 'install', external=True) session.run('pytest') this works perfectly, except that it modifies the pyproject.toml file in my project's root. Is there a better way for me to arrange this? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#347>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB5I467I3V4VCLT65CNH7TSF6IPFANCNFSM4RNMOOZQ> .

[danieleades]

the poetry add step is changing the version of the 'sphinx' package in the pyproject.toml file. Once the script exits, I then need to revert that change and regenerate the lockfile

for example if sphinx version is set to 3.0 in the pyproject.toml file, say, and I use "2.6" in my nox session, the file will have sphinx set to "2.6" when the nox script exits.

[danieleades]

To be clear, I'm not suggesting that nox is misbehaving!
Im simply wondering if this workflow can be supported today, or if there is interest in implementing it (either on the nox side, or the Poetry side)

[theacodes]

We're definitely happy to support poetry use cases, I'm just not personally familiar enough to know. Maybe we can get one of the poetry maintainers on this thread to help out?

…

On Tue, Sep 15, 2020 at 4:31 PM danieleades ***@***.***> wrote: To be clear, I'm *not* suggesting that nox is misbehaving! Im simply wondering if this workflow can be supported today, or if there is interest in implementing it (either on the nox side, or the Poetry side) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#347 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB5I4ZIHJ7TB54ZA2TBZITSF7FKFANCNFSM4RNMOOZQ> .

[danieleades]

there are a couple of solutions proposed in python-poetry/poetry#2920

I wonder what a first class API would look like?

• one solution is to clone the entire working directory to a temporary folder for each run. This is a nice general solution (though it's kind of the nuclear option). I could see Nox having a config flag to perform this step automagically
• the second proposed solution is to cache and replace the pyproject.toml and poetry.lock files. This is lightweight and solves my problem, but it's very poetry-specific and won't help people that are, for whatever reason, modifying other project files during their sessions

[danieleades]

@theacodes you can see the approach i'm attempting to use to restore the pyproject.toml and poetry.lock files after execution here - python-poetry/poetry#2920 (comment)

it's not working because wrapping nox in a try/finally block doesn't seem to work.

Is there anyway to cleanup code after a session with Nox? Effectively a 'teardown' function

[layday]

it's not working because wrapping nox in a try/finally block doesn't seem to work.

Sure it works - dump gets called on import, before the tests run ;P You have to move the try block inside the session:

@nox.session(python=python_versions)
@nox.parametrize("sphinx", sphinx_versions)
def tests(session, sphinx):
    try:
        session.run('poetry', 'add', f'sphinx@{sphinx}', external=True)
        session.run('poetry', 'install', external=True)
        session.run('nosetests')
    finally:
        dump(project_file)
        dump(lockfile)

[danieleades]

i'm afraid this still isn't working

given this code-

import nox

@nox.session(python = ["3.5", "3.6"]
@nox.parametrize("sphinx", ["^2.4", "^3.0"])
def tests(session, sphinx)
    session.run('poetry', 'add', f'sphinx@{sphinx}', external=True)
    session.run('poetry', 'install', external=True)
    session.run('pytest')

the python interpreter used is always the one selected by Poetry, in my case 3.8.
I can't use this setup for testing a matrix of interpreter versions.
any ideas?

[layday]

Yes, you should install Poetry and pytest with Nox.

[cjolowicz]

Hi @danieleades would something like this work for you?

import nox

@nox.session(python=["3.5", "3.6"])
@nox.parametrize("sphinx", ["2.4.4", "3.2.1"])
def tests(session, sphinx):
    session.install(".", f"sphinx=={sphinx}", "pytest")
    session.run("pytest")

This will install and test your package with a specific sphinx version, using only the version constraints in the package metadata, not the lock file.

[danieleades]

I did get this working in the end! With the original solution. I suspect it was working all along ...

[theacodes]

Is this still a bug? Should I keep this open? It doesn't seem like there's anything left to resolve here, so I'm going to close for now but I'm happy to re-open if there's still more work to do.

[danieleades]

if anyone stumbles upon this, I've got a working implementation, integrated with a github action here - https://github.com/danieleades/sphinxcontrib-needs/actions