Allow passing callable venv_backend for creation of session virtualenv

[wpk-nist-gov]

How would this feature be useful?

There are are number of open issues regarding specific details of creating session virtual environments (e.g., #680, #662, #260). For example, the creation of conda virtual environments is often intimately linked to an environment.yaml file. While dependencies can often be installed after the virtual environment creation, this can be limiting. This is especially true for locked conda environments (e.g. from conda-lock) which combines creating the environment and installing all dependencies. If a callable could be passed as a venv_backend, you could do something like the following:

def create_conda_lock_env(
    location: str,
    interpreter: str | None = None,
    reuse_existing: bool = False,
    venv_params: Any = None,
) -> CondaEnv:
    venv = CondaEnv(
        location=location,
        interpreter=interpreter,
        reuse_existing=reuse_existing,
        venv_params=venv_params,
    )

    # assume venv_params contains conda-lock.yml file
    if isinstance(venv_params, str) and os.path.exists(venv_params):
        lock_file = venv_params
    else:
        raise ValueError("pass `venv_params={conda-lock-file}`")

    # Custom creating (based on CondaEnv.create)
    if not venv._clean_location():
        logger.debug(f"Re-using existing conda env at {venv.location_name}.")
        venv._reused = True

    else:
        cmd = ["conda-lock", "install", "--prefix", venv.location, lock_file]

        logger.info(
            f"Creating conda env in {venv.location_name} with conda-lock {lock_file}"
        )
        nox.command.run(cmd, silent=True, log=nox.options.verbose or False)

    return venv


# Note that it's on the end user/custom backend to make sure passing python=.... makes sense.
@nox.session(
    python="3.11",
    venv_backend=create_conda_lock_env,
    venv_params="py311-test-conda-lock.yml",
)
def test(session: nox.Session) -> None:
    ...

Similarly, you could use this approach to:

• Create environments using micromamba
• Customize python search path for python interpreters
• Use conda env create -f environment.yaml
• Make creation of development environment trivial

For the last one, you could do the following:

from nox.virtualenv import VirtualEnv
def create_venv_override_location(
    location: str,
    interpreter: str | None = None,
    reuse_existing: bool = False,
    venv_params: Any = None,
) -> None:
    assert isinstance(venv_params, str), "supply location with venv_params"

    location = venv_params
    venv = VirtualEnv(
        location=location,
        interpreter=interpreter,
        reuse_existing=True,
    )

    venv.create()
    return venv

@nox.session(
    name="dev-example",
    python="3.11",
    venv_backend=create_venv_override_location,
    venv_params="./.venv",
)
def dev_example(session: nox.Session) -> None:
     ...

In general, this would provide a means for third party packages to hook in to nox.

Describe the solution you'd like

I believe this should be straight forward. I'd simply replace the logic in nox.session.SesssionRunner._create_env

with something like:

    def _create_venv(self) -> None:

        if callable(self.func.venv_backend):
            # if passed a callable backend, always use just that
            # i.e., don't override
            backend = self.func.venv_backend
        else:
            backend = (
                self.global_config.force_venv_backend
                or self.func.venv_backend
                or self.global_config.default_venv_backend
            )

        if backend == "none" or self.func.python is False:
            self.venv = PassthroughEnv()
            return

        reuse_existing = (
            self.func.reuse_venv or self.global_config.reuse_existing_virtualenvs
        )

        if callable(backend):
            self.venv = backend(
                location=self.envdir,
                interpreter=self.func.python,
                reuse_existing=reuse_existing,
                venv_params=self.func.venv_params,
            )
            return

        elif backend is None or backend == "virtualenv":
...

Down the road, it would be nice to be able to create custom environment classes by subclassing nox.virtualenv.ProcessEnv.

Describe alternatives you've considered

You can currently use things like conda-lock with nox, but they require creating the environment twice. First nox creates the environment, then conda-lock recreates the environment. This is not ideal.

Anything else?

I have a minimally working example. I'm not totally sure how to integrate this into the test suite. If theres interest, I'm happy to make a PR.

[wpk-nist-gov]

In the meantime, I'm using the following code to patch venv_backed creation:

from nox.sessions import SessionRunner
import logging

logger = logging.getLogger("nox")
_create_venv_super = SessionRunner._create_venv

def override_sessionrunner_create_venv(self) -> None:
    """
    Override SessionRunner._create_venv
    """


    if callable(self.func.venv_backend):
        # if passed a callable backend, always use just that
        # i.e., don't override
        logger.info(f"Using custom callable venv_backend")

        self.venv = self.func.venv_backend(
            location=self.envdir,
            interpreter=self.func.python,
            reuse_existing=self.func.reuse_venv or self.global_config.reuse_existing_virtualenvs,
            venv_params=self.func.venv_params,
            runner=self,
        )
        return
    else:
        logger.info(f"Using nox venv_backend")
        return _create_venv_super(self)

SessionRunner._create_venv = override_sessionrunner_create_venv

[theacodes]

This sounds great to me, would be happy to see a PR to implement it.

[wpk-nist-gov]

Yay! I'll put that together asap!