layout poetry doesn't work

[nickcorona]

Describe the bug
It doesn't activate the existing poetry virtual environment.

To Reproduce
Steps to reproduce the behavior:
Create .envrc with layout poetry in it after adding the function to ~/.config/direnv/direnvrc. Restart terminal. Observe that it doesn't load the environment.

Expected behavior
Loads the environment.

Environment

                   -`                    nick@desktop 
                  .o+`                   ------------ 
                 `ooo/                   OS: Arch Linux x86_64 
                `+oooo:                  Host: MS-7751 2.0 
               `+oooooo:                 Kernel: 5.5.3-arch1-1 
               -+oooooo+:                Uptime: 19 hours, 12 mins 
             `/:-:++oooo+:               Packages: 1275 (pacman) 
            `/++++/+++++++:              Shell: zsh 5.7.1 
           `/++++++++++++++:             Resolution: 1920x1080 
          `/+++ooooooooooooo/`           DE: Budgie 
         ./ooosssso++osssssso+`          WM: Mutter(Budgie) 
        .oossssso-````/ossssss+`         Theme: Nordic [GTK2/3] 
       -osssssso.      :ssssssso.        Icons: Paper [GTK2/3] 
      :osssssss/        osssso+++.       Terminal: gnome-terminal 
     /ossssssss/        +ssssooo/-       CPU: Intel i5-3570K (4) @ 4.200GHz 
   `/ossssso+/:-        -:/+osssso+-     GPU: NVIDIA GeForce GTX 670 
  `+sso+:-`                 `.-/+oso:    Memory: 3360MiB / 24006MiB 

direnv version: 2.21.2

[rolep]

Had similar error, because previously used poetry run which python may have used system / global poetry (not the one installed in virtualenv after poetry install) and this caused bad output -> bad VENV and PATH settings -> invalid environment and so on.

Have changed layout_poetry function it to (wiki also updated):

local VENV=$(poetry env list --full-path | cut -d' ' -f1)/bin
if [[ ! -d $VENV ]]; then
  log_error 'No created poetry virtual environment found.  Use `poetry install` to create one first.'
  exit 2
fi

[Shados]

@rolep If there are no poetry environments at all, that will result in "VENV=/bin", and /bin is quite likely to exist, causing a false positive.

[rolep]

You are right. Added empty var check - if [[ -z $VENV || ! -d $VENV/bin ]];

[tebeka]

Are there any plans to add layout_poetry to the standard library?

[Diaoul]

I had no success with the version on the wiki so I ended up with this one, assembling ideas from here and there.

layout_poetry() {
  if [[ ! -f pyproject.toml ]]; then
    log_error 'No pyproject.toml found. Use `poetry new` or `poetry init` to create one first.'
    exit 2
  fi

  local VENV=$(poetry env info --path)
  if [[ -z $VENV || ! -d $VENV/bin ]]; then
    log_error 'No poetry virtual environment found. Use `poetry install` to create one first.'
    exit 2
  fi

  export VIRTUAL_ENV=$VENV
  export POETRY_ACTIVE=1
  PATH_add "$VENV/bin"
}

Edit 2022-10-13:

- local VENV=$(poetry env list --full-path | cut -d' ' -f 1)
+ local VENV=$(poetry env info --path)

[nickgarber]

Would a PR to add Poetry support into direnv be considered?

[doolio]

@Diaoul am I right in that your solution requires you to cd into an out of the directory after first generating the pyproject.toml and then again after using poetry install to create the virtual environment? Only then will it be activated. Thanks.

[Diaoul]

If this is a new project, you would've to poetry init to generate the pyproject.toml, this will also create the virtual environment so you don't need to poetry install.

If this is an existing project, pyproject.toml will be there so you don't need poetry init but there will be no virtual environment so you will need to run poetry install.

In all cases, that means cd-ing only once after you ran the appropriate command.

[doolio]

I have the latest (stable) poetry v1.2.2 installed and poetry init creates the pyproject.toml but does not create the virtual environment at least not for me. I'm only just learning to programme (started with python) so it may be a misunderstanding on my part.

My workflow:

1. Create new project directory, demo-project say with mkdir demo-project
2. cd demo-project... I actually have a bash function I name md which combines steps 1 (mkdir) and 2 (cd)
3. Create .envrc in this directory with echo 'layout poetry' > .envrc
4. direnv allow
5. With your layout_poetry solution prompted to run either poetry new or poetry init. But executing poetry new would create a new project directory within the already created project directory so perhaps your log_error message should only suggest to use poetry init, no? In any case, execute poetry init which interactively creates the pyproject.toml but no virtual environment is created. This can be seen by running poetry env list which returns nothing.
6. cd .. -> direnev: unloading
7. cd demo-project
8. Prompted to run poetry install as no virtual environment exists. This creates the virtual environment (as shown by poetry env list) but it is not activated. In addition, none of the ENV variables $VIRTUAL_ENV, $POETRY_ACTIVE or $PATH are set as we exited the layout_poetry function.
9. cd .. -> direnev: unloading
10. cd demo-project. Virtual environment exists and is activated. In addition, the ENV variables are set as we expect.
11. Start to work.

So as you can see quite a lot of cding.

I had been working on the following solution (based on layout_pipenv from the stdlib.sh) but it wasn't initially working for me but now appears to be. I also use asdf and believe it wasn't working as I had installed poetry (before asdf) with my system python (v3.9.2) rather than the version I installed with asdf namely v3.10.7. Uninstalling poetry and re-installing it with the asdf managed python seems to make it now work. I don't quite know why or if it was not working for another reason. For a long time it seemed the command substitutions $( ) weren't working but I couldn't understand why they would not be.

layout_poetry() {
    PYPROJECT_TOML="${PYPROJECT_TOML:-pyproject.toml}"
    if [[ ! -f "$PYPROJECT_TOML" ]]; then
        log_status "No pyproject.toml found. Executing \`poetry init\` to create a \`$PYPROJECT_TOML\` first."
        poetry init
    fi

    VIRTUAL_ENV=$(poetry env info --path 2>/dev/null ; true)

    if [[ -z $VIRTUAL_ENV || ! -d $VIRTUAL_ENV ]]; then
        log_status "No virtual environment exists. Executing \`poetry install\` to create one."    
        poetry install
        VIRTUAL_ENV=$(poetry env info --path)
    fi

    PATH_add "$VIRTUAL_ENV/bin"
    export POETRY_ACTIVE=1
    export VIRTUAL_ENV
}

With this my workflow is as follows:

1. Create new project directory, demo-project say with mkdir demo-project
2. cd demo-project ... I actually have a bash function I name md which combines steps 1 (mkdir) and 2 (cd)
3. Create .envrc file with echo 'layout poetry' > .envrc
4. direnv allow -> poetry init executes interactively creating the pyproject.toml...poetry install executes creating the virtual environment, adding the virtual environment python version to my $PATH and finally activating the virtual environment setting the ENV variables $VIRTUAL_ENV and $POETRY_ACTIVE as we expect.
5. Start to work.

So with this only 3 or 4 steps before starting to work. The only issue after the poetry lock file is created I get the following error(?) message:

/home/doolio/projects/learning/python/demo-project/demo_project does not contain any element

Though this might be expected.

For the poetry init Iine it could be improved further with something like the following:

# Specify the active python rather than the one used to install poetry in case they are not the same.
poetry init --python ^$(python3 --version 2>/dev/null | cut -d' ' -f2 | cut -d. -f1-2)

We could also include the --no-interaction option to simply make the pyproject.toml with the default suggestions.

To avail of the project structure provided by poetry new my workflow is as follows:

1. poetry new --src demo-project
2. cd demo-project
3. Create .envrc file with echo 'layout poetry' > .envrc
4. direnv allow -> poetry install executes creating the virtual environment, adding the virtual environment python version to my $PATH and finally activating the virtual environment setting the ENV variables $VIRTUAL_ENV and $POETRY_ACTIVE as we expect.
5. Start to work.

So 4 steps before starting to work. With this workflow I don't get the error message after the poetry lock file is written.

I think I will submit this solution as a PR for consideration so poetry support is built-in.

Edit: Update to layout_poetry above to use log_status instead of log_error. Also, added a log_status before the poetry install command executes to not alarm the user.

[Diaoul]

poetry init creates the pyproject.toml but does not create the virtual environment

This is correct. I happened to have a virtualenv already existing for that path from a prior test so it picked it up again.

[kbd]

Here's what I've done to fix this. Based on the official Poetry recipe in the wiki, change:

VIRTUAL_ENV=$(poetry env info --path 2>/dev/null ; true)

to:

VIRTUAL_ENV="$(poetry env list --full-path | head -1)"

it still doesn't activate the virtualenv on first run though so exit the directory and re-enter. An update to the recipe on the wiki that works properly would be welcome.

[doolio]

it still doesn't activate the virtualenv on first run though so exit the directory and re-enter. An update to the recipe on the wiki that works properly would be welcome.

Why would it? That line in the wiki is determining whether the VIRTUAL_ENV already exists and if it does to set the ENV variable appropriately. Your change is providing the same information.

I used poetry env info --path rather than poetry env list --full-path so I didn't have to then cut the "(Activated)" part.

The function provided in the wiki creates and activates the virtual environment on first use. direnv automatically activates it on subsequent entry to the project directory. Either command will tell you the virtual environment is activated or you could update your $PS1 to inform you it has. HTH.

[kortsi]

it still doesn't activate the virtualenv on first run though so exit the directory and re-enter. An update to the recipe on the wiki that works properly would be welcome.

One thing that might work would be to add a file watcher to your .envrc to monitor poetry virtualenv changes:

layout poetry
watch_file ~/.cache/pypoetry/virtualenvs/envs.toml

That will reload direnv when that file changes. This seems to work for me with Poetry 1.4. It also detects when you change Python version with poetry env use. But if you work on two projects at the same time with this option set, both of them will reload if you change virtualenv in one of them, since it's one file for all projects. This seems like a minor inconvenience, though.

[haizaar]

Here is what I have ended up with. poetry.lock is watch'ed and I keep local hash to rerun install whenever it updates.
This ensures other developers get their env updated after pull.

layout_poetry() {
  if [[ ! -f pyproject.toml ]]; then
    log_error 'No pyproject.toml found. Use `poetry new` or `poetry init` to create one first.'
    exit 2
  fi

  LOCK="$PWD/poetry.lock"
  watch_file "$LOCK"

  local VENV=$(poetry env info --path)
  if [[ -z $VENV || ! -d $VENV/bin ]]; then
    log_status 'No poetry virtual environment found. Running `poetry install` to create one.'
    poetry install
    VENV=$(poetry env info --path)
  else
    HASH="$PWD/.poetry.lock.sha512"
    if ! sha512sum -c $HASH --quiet >&/dev/null ; then
        log_status 'poetry.lock has been updated. Running `poetry install`'
        poetry install
        sha512sum "$LOCK" > "$HASH"
    fi
  fi
  
  export VIRTUAL_ENV=$VENV
  export POETRY_ACTIVE=1
  PATH_add "$VENV/bin"
}

One downside is that poetry install is run again after, e.g. poetry add, but it takes no time and I don't add/remove packages every day, so it's an acceptable compromise for me.