Should I use --no-root ? Some description of building problems, and lots of strange behaviour

[aahnik]

I am observing some strange things. I would love to hear some explanation of the concepts.

( Extra context: I recently read: https://hynek.me/articles/testing-packaging/ by @hynek, and #3923 is also related)

Structure

I have a project directory like this:

.
├── poetry.lock
├── pyproject.toml
├── README.rst
├── src
│   ├── another_p
│   │   ├── __init__.py
│   └── custom_learn_p
│       ├── __init__.py
├── test.py

Files

1. pyproject.toml is the default one as generated by poetry new --src --name custom_learn_p new_folder
2. another_p/__init__.py looks like this

def another():
    return 'another'

3. custom_learn_p/__init__.py looks like this

def whatever():
    return 'hello'

4. Now, the test.py in the root of the project directory looks like this:

from custom_learn_p import whatever
from another_p import another

assert whatever() == 'hello'
assert another() == 'another'

print('end of test')

Running

From inside the root of the project, I open my terminal and run:

$ poetry shell
# success
(.venv) $ poetry install
# success
(.venv)  $ python test.py
end of test
# no errors raised

In the project virtual env, everything worked.

Problem

Now, I build the package:

$ poetry build

The dist folder is now has custom_learn_p-0.1.0-py3-none-any.whl

Now, I create another fresh virtual environment and install the whl:

$ python -m venv .new_venv
$ source .new_venv/bin/activate
(.new_venv) $ pip install dist/custom_learn_p-0.1.0-py3-none-any.whl

Now, from inside this fresh virtual environment, I run,

$ python test.py
Traceback (most recent call last):
  File "/home/aahnik/Projects/new_folder/test.py", line 2, in <module>
    from another_p import another
ModuleNotFoundError: No module named 'another_p'

What I realize: package custom_learn_p from the src directory was included, but not another_p. ( I know, that I can include it, in the pyproject.toml, by specifying packages under [tool.poetry] )

I realized this fault, only after using a separate virtual environment, and pretending to be a user.

In the project's virtual env, another_p is available (it got available, as soon as I wrote the code, before even running poetry install (I had run a poetry install earlier, after the writing the package custom_learn_p) ?).. but not in the build?

why?

If I would not have pretended to be a user, and installed the whl via pip, I would not have caught this packaging problem, as everything worked fine in my project virtual environment.

While developing, should I always run poetry install --no-root, so my package does not get installed? and thus I can build and install, manually, so that these errors, get caught?

I think poetry installs the current package in editable mode. Any change in my code immediately reflects.

Any new package added under the src directory, gets available as soon as they are written, and without re-running install, inside the project's virtual environment. But they are not included in the build unless specified in the pyproject.toml.

Is this expected behavior?

---

Thanks a lot for your time.

[finswimmer]

Hello @aahnik,

this is (more or less) an expected behavior, due to how editable installs works. A path configuration file is used to add the root folder of the package to the search path for modules. This means that the src folder is now included in this search path and every package folder that is located there, can be found as well.

This is a good reason why you want to use tox for testing, which will build your package and install it into a new venv for testing. You can see how I do this here

fin swimmer

[aahnik]

This means that the src folder is now included in this search path and every package folder that is located there, can be found as well.

Thank you for clearing my confusion.

and yup, planning to learn and start using tox.