project / package installation error with deactivated virtualenvironments #9190
Labels
kind/bug
Something isn't working as expected
status/waiting-on-response
Waiting on response from author
Comments
pascal456
added
kind/bug
|
This looks more like an environment issue than Poetry. The oddities I see, and I cannot comment without verbose logs and also possible a reproducer container, are as follows.
Please provide a concise reproducer. |
abn
added
status/waiting-on-response
|
Thanks for having a look!
This is what the docs suggest to do. While Poetry automatically creates virtual environments to always work isolated from the global Python installation, there are valid reasons why it’s not necessary and is an overhead, like when working with containers. Further, I have set up other projects with poetry and containers are working fine actually. But for whatever reason, the above package installation steps try to activate an virtualenv. I will try to reproduce another example, that is working. Until then, maybe you can make another guess 🙂 ?
Yes I was wondering. I am not into the details of poetry, which is why I opened the issue. My first guess or feeling was, that this is due to poetries inner workings.
Sure 👍, here a full example repo: https://github.com/pascal456/exampleproject Interestingly, there is no issue when I install with the devcontainer. |
|
And here is an example (branch) with a working installation configuration: https://github.com/pascal456/exampleproject/tree/working-example The difference lies only in the dependencies (pyproject.toml + corresponding poetry.lock changes): pascal456/exampleproject@58e2f74 |
That should say that is an option that can be used if you "really" want it and you "really" understand what the implications are. The "like when working with containers" really need to be removed. As there is no realistic overhead so to speak off, and most of the time the use cases that need base container packages can simply use the virtualenvs.options.system-site-packages option (I am also aware there are a couple bugs here that needs squashing). And often times than not, you will really end up foot-guning yourself if you disable venvs.
Thanks for that, much appreciated. We can take a look see what we end up with. |
|
The failure is a bit more clearer with verbose logging. console.log
[virtualenv] find interpreter for spec PythonSpec(path=/usr/local/bin/python3.11)
[virtualenv] proposed PythonInfo(spec=CPython3.11.8.final.0-64, exe=/usr/local/bin/python3.11, platform=linux, version='3.11.8 (main, Mar 12 2024, 11:52:02) [GCC 12.2.0]', encoding_fs_io=utf-8-utf-8)
[virtualenv] accepted PythonInfo(spec=CPython3.11.8.final.0-64, exe=/usr/local/bin/python3.11, platform=linux, version='3.11.8 (main, Mar 12 2024, 11:52:02) [GCC 12.2.0]', encoding_fs_io=utf-8-utf-8)
[virtualenv] create virtual environment via CPython3Posix(dest=/tmp/tmp487coft9/.venv, clear=False, no_vcs_ignore=False, global=False)
[virtualenv] create folder /tmp/tmp487coft9/.venv/bin
[virtualenv] create folder /tmp/tmp487coft9/.venv/lib/python3.11/site-packages
[virtualenv] write /tmp/tmp487coft9/.venv/pyvenv.cfg
[virtualenv] home = /usr/local/bin
[virtualenv] implementation = CPython
[virtualenv] version_info = 3.11.8.final.0
[virtualenv] virtualenv = 20.25.1
[virtualenv] include-system-site-packages = false
[virtualenv] base-prefix = /usr/local
[virtualenv] base-exec-prefix = /usr/local
[virtualenv] base-executable = /usr/local/bin/python3.11
[virtualenv] symlink /usr/local/bin/python3.11 to /tmp/tmp487coft9/.venv/bin/python
[virtualenv] create virtualenv import hook file /tmp/tmp487coft9/.venv/lib/python3.11/site-packages/_virtualenv.pth
[virtualenv] create /tmp/tmp487coft9/.venv/lib/python3.11/site-packages/_virtualenv.py
[virtualenv] ============================== target debug ==============================
[virtualenv] debug via /tmp/tmp487coft9/.venv/bin/python /usr/local/lib/python3.11/site-packages/virtualenv/create/debug.py
[virtualenv] {
[virtualenv] "sys": {
[virtualenv] "executable": "/tmp/tmp487coft9/.venv/bin/python",
[virtualenv] "_base_executable": "/usr/local/bin/python3.11",
[virtualenv] "prefix": "/tmp/tmp487coft9/.venv",
[virtualenv] "base_prefix": "/usr/local",
[virtualenv] "real_prefix": null,
[virtualenv] "exec_prefix": "/tmp/tmp487coft9/.venv",
[virtualenv] "base_exec_prefix": "/usr/local",
[virtualenv] "path": [
[virtualenv] "/usr/local/lib/python311.zip",
[virtualenv] "/usr/local/lib/python3.11",
[virtualenv] "/usr/local/lib/python3.11/lib-dynload",
[virtualenv] "/tmp/tmp487coft9/.venv/lib/python3.11/site-packages"
[virtualenv] ],
[virtualenv] "meta_path": [
[virtualenv] "<class '_virtualenv._Finder'>",
[virtualenv] "<class '_frozen_importlib.BuiltinImporter'>",
[virtualenv] "<class '_frozen_importlib.FrozenImporter'>",
[virtualenv] "<class '_frozen_importlib_external.PathFinder'>"
[virtualenv] ],
[virtualenv] "fs_encoding": "utf-8",
[virtualenv] "io_encoding": "utf-8"
[virtualenv] },
[virtualenv] "version": "3.11.8 (main, Mar 12 2024, 11:52:02) [GCC 12.2.0]",
[virtualenv] "makefile_filename": "/usr/local/lib/python3.11/config-3.11-x86_64-linux-gnu/Makefile",
[virtualenv] "os": "<module 'os' (frozen)>",
[virtualenv] "site": "<module 'site' (frozen)>",
[virtualenv] "datetime": "<module 'datetime' from '/usr/local/lib/python3.11/datetime.py'>",
[virtualenv] "math": "<module 'math' from '/usr/local/lib/python3.11/lib-dynload/math.cpython-311-x86_64-linux-gnu.so'>",
[virtualenv] "json": "<module 'json' from '/usr/local/lib/python3.11/json/__init__.py'>"
[virtualenv] }
[urllib3:urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/f2/91/e5ae454da8200c6eb6cf94ca05d799b51e2cb2cc458a7737aebc0c5a21bb/croniter-1.4.1-py2.py3-none-any.whl HTTP/1.1" 200 19579
[virtualenv] add activators for Bash, CShell, Fish, Nushell, PowerShell, Python
Stack trace:
20 /usr/local/lib/python3.11/site-packages/poetry/installation/executor.py:281 in _execute_operation
279│
280│ try:
→ 281│ result = self._do_execute_operation(operation)
282│ except EnvCommandError as e:
283│ if e.e.returncode == -2:
19 /usr/local/lib/python3.11/site-packages/poetry/installation/executor.py:391 in _do_execute_operation
389│ return 0
390│
→ 391│ result: int = getattr(self, f"_execute_{method}")(operation)
392│
393│ if result != 0:
18 /usr/local/lib/python3.11/site-packages/poetry/installation/executor.py:516 in _execute_install
514│
515│ def _execute_install(self, operation: Install | Update) -> int:
→ 516│ status_code = self._install(operation)
517│
518│ self._save_url_reference(operation)
17 /usr/local/lib/python3.11/site-packages/poetry/installation/executor.py:554 in _install
552│ archive = self._download_link(operation, Link(package.source_url))
553│ else:
→ 554│ archive = self._download(operation)
555│
556│ operation_message = self.get_operation_message(operation)
16 /usr/local/lib/python3.11/site-packages/poetry/installation/executor.py:746 in _download
744│ self._yanked_warnings.append(message)
745│
→ 746│ return self._download_link(operation, link)
747│
748│ def _download_link(self, operation: Install | Update, link: Link) -> Path:
15 /usr/local/lib/python3.11/site-packages/poetry/installation/executor.py:781 in _download_link
779│ self._write(operation, message)
780│
→ 781│ archive = self._chef.prepare(archive, output_dir=original_archive.parent)
782│
783│ # Use the original archive to provide the correct hash.
14 /usr/local/lib/python3.11/site-packages/poetry/installation/chef.py:123 in prepare
121│ return self._prepare(archive, destination=destination, editable=editable)
122│
→ 123│ return self._prepare_sdist(archive, destination=output_dir)
124│
125│ def _prepare(
13 /usr/local/lib/python3.11/site-packages/poetry/installation/chef.py:194 in _prepare_sdist
192│ destination.mkdir(parents=True, exist_ok=True)
193│
→ 194│ return self._prepare(
195│ sdist_dir,
196│ destination,
12 /usr/local/lib/python3.11/site-packages/poetry/installation/chef.py:130 in _prepare
128│ from subprocess import CalledProcessError
129│
→ 130│ with ephemeral_environment(self._env.python) as venv:
131│ env = IsolatedEnv(venv, self._pool)
132│ builder = ProjectBuilder.from_isolated_env(
11 /usr/local/lib/python3.11/contextlib.py:137 in __enter__
135│ del self.args, self.kwds, self.func
136│ try:
→ 137│ return next(self.gen)
138│ except StopIteration:
139│ raise RuntimeError("generator didn't yield") from None
10 /usr/local/lib/python3.11/site-packages/poetry/utils/env/__init__.py:49 in ephemeral_environment
47│ # TODO: cache PEP 517 build environment corresponding to each project venv
48│ venv_dir = Path(tmp_dir) / ".venv"
→ 49│ EnvManager.build_venv(
50│ path=venv_dir,
51│ executable=executable,
9 /usr/local/lib/python3.11/site-packages/poetry/utils/env/env_manager.py:678 in build_venv
676│ args.append(str(path))
677│
→ 678│ cli_result = virtualenv.cli_run(args, setup_logging=False)
679│
680│ # Exclude the venv folder from from macOS Time Machine backups
8 /usr/local/lib/python3.11/site-packages/virtualenv/run/__init__.py:33 in cli_run
31│ with of_session:
32│ of_session.run()
→ 33│ return of_session
34│
35│
7 /usr/local/lib/python3.11/site-packages/virtualenv/run/session.py:46 in run
44│
45│ def run(self):
→ 46│ self._create()
47│ self._seed()
48│ self._activate()
6 /usr/local/lib/python3.11/site-packages/virtualenv/run/session.py:65 in _activate
63│ if self.activators:
64│ logging.info(
→ 65│ "add activators for %s",
66│ ", ".join(type(i).__name__.replace("Activator", "") for i in self.activators),
67│ )
5 /usr/local/lib/python3.11/site-packages/virtualenv/activation/via_template.py:27 in generate
25│ def generate(self, creator):
26│ dest_folder = creator.bin_dir
→ 27│ replacements = self.replacements(creator, dest_folder)
28│ generated = self._generate(replacements, self.templates(), dest_folder, creator)
29│ if self.flag_prompt is not None:
4 /usr/local/lib/python3.11/site-packages/virtualenv/activation/via_template.py:44 in _generate
42│ def _generate(self, replacements, templates, to_folder, creator):
43│ generated = []
→ 44│ for template in templates:
45│ text = self.instantiate_template(replacements, template, creator)
46│ dest = to_folder / self.as_name(template)
3 /usr/local/lib/python3.11/site-packages/virtualenv/activation/via_template.py:60 in instantiate_template
58│ text = binary.decode("utf-8", errors="strict")
59│ for key, value in replacements.items():
→ 60│ value = self._repr_unicode(creator, value)
61│ text = text.replace(key, value)
62│ return text
2 /usr/local/lib/python3.11/site-packages/virtualenv/activation/via_template.py:13 in read_binary
11│ from .activator import Activator
12│
→ 13│ if sys.version_info >= (3, 7):
14│ from importlib.resources import read_binary
15│ else:
1 /usr/local/lib/python3.11/pathlib.py:1050 in read_bytes
1048│ Open the file in bytes mode, read it, and close the file.
1049│ """
→ 1050│ with self.open(mode='rb') as f:
1051│ return f.read()
1052│
FileNotFoundError
[Errno 2] No such file or directory: '/usr/local/lib/python3.11/site-packages/virtualenv/activation/nushell/activate.nu'
at /usr/local/lib/python3.11/pathlib.py:1044 in open
[urllib3:urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/5a/8f/a9d39ec15f40e8169cb134317824ee4618b864b2e4b91a9b310d3ef94729/deepdiff-6.7.1-py3-none-any.whl HTTP/1.1" 200 76554
1040│ the built-in open() function does.
[urllib3:urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/cb/56/f4845ed78723a4eb8eb22bcfcb46e1157a462c78c0a5ed318c68c98f9a79/gdown-5.1.0-py3-none-any.whl HTTP/1.1" 200 17769
1041│ """
1042│ if "b" not in mode:
1043│ encoding = io.text_encoding(encoding)
→ 1044│ return io.open(self, mode, buffering, encoding, errors, newline)
1045│
1046│ def read_bytes(self):
1047│ """
1048│ Open the file in bytes mode, read it, and close the file.
Cannot install ffmpy.The issue happens when the isolated build environment for the sdist build for the dependencies that fail to install run. |
|
I will also note that not disabling virtual environments, works well. The issue here really is likely that the package installation modifying the environment in which poetry is running while it is running. podman run --rm -i --entrypoint bash -v poetry-cache:/root/.cache/pypoetry docker.io/python:3.11-slim-bookworm <<EOF
set -xe
python -m pip install --disable-pip-version-check -q poetry
poetry config virtualenvs.in-project true
poetry config virtualenvs.options.no-pip true
poetry config virtualenvs.options.no-setuptools true
export DEBIAN_FRONTEND=noninteractive
apt update -qq && apt install -qq -y git-all libgomp1 libffi-dev
git clone https://github.com/pascal456/exampleproject.git /opt/app
pushd /opt/app
#poetry config virtualenvs.create false
poetry install --only main --compile
EOFWarning: This (used without an existing cache will download around 7GB of data. console.log
+ git clone https://github.com/pascal456/exampleproject.git /opt/app
Cloning into '/opt/app'...
+ pushd /opt/app
/opt/app /
+ poetry install --only main --compile
Creating virtualenv exampleproject in /opt/app/.venv
Installing dependencies from lock file
Package operations: 310 installs, 0 updates, 0 removals
- Installing pip (24.0)
- Installing six (1.16.0)
- Installing jmespath (1.0.1)
- Installing pyasn1 (0.5.1)
- Installing python-dateutil (2.9.0.post0)
- Installing urllib3 (2.2.1)
- Installing botocore (1.34.67)
- Installing cachetools (5.3.3)
- Installing certifi (2024.2.2)
- Installing charset-normalizer (3.3.2)
- Installing cmake (3.28.3)
- Installing filelock (3.13.1)
- Installing frozenlist (1.4.1)
- Installing idna (3.6)
- Installing lit (18.1.1)
- Installing markupsafe (2.1.5)
- Installing mdurl (0.1.2)
- Installing mpmath (1.3.0)
- Installing multidict (6.0.5)
- Installing protobuf (4.25.3)
- Installing pyasn1-modules (0.3.0)
- Installing rsa (4.9)
- Installing setuptools (69.2.0)
- Installing sniffio (1.3.1)
- Installing wheel (0.43.0)
- Installing aiosignal (1.3.1)
- Installing anyio (3.7.1)
- Installing attrs (23.2.0)
- Installing cymem (2.0.8)
- Installing google-auth (2.29.0)
- Installing googleapis-common-protos (1.63.0)
- Installing h11 (0.14.0)
- Installing jinja2 (3.1.3)
- Installing markdown-it-py (3.0.0)
- Installing murmurhash (1.0.10)
- Installing networkx (3.2.1)
- Installing numpy (1.26.4)
- Installing nvidia-cublas-cu11 (11.10.3.66)
- Installing nvidia-cuda-cupti-cu11 (11.7.101)
- Installing nvidia-cuda-nvrtc-cu11 (11.7.99)
- Installing nvidia-cuda-runtime-cu11 (11.7.99)
- Installing nvidia-cudnn-cu11 (8.5.0.96)
- Installing nvidia-cufft-cu11 (10.9.0.58)
- Installing nvidia-curand-cu11 (10.2.10.91)
- Installing nvidia-cusolver-cu11 (11.4.0.1)
- Installing nvidia-cusparse-cu11 (11.7.4.91)
- Installing nvidia-nccl-cu11 (2.14.3)
- Installing nvidia-nvtx-cu11 (11.7.91)
- Installing packaging (23.2)
- Installing pygments (2.17.2)
- Installing pytz (2023.4)
- Installing requests (2.31.0)
- Installing s3transfer (0.10.1)
- Installing sympy (1.12)
- Installing triton (2.0.0)
- Installing typing-extensions (4.10.0)
- Installing tzdata (2024.1)
- Installing wcwidth (0.2.13)
- Installing xmod (1.8.1)
- Installing yarl (1.9.4)
Installing /opt/app/.venv/lib/python3.11/site-packages/nvidia/__init__.py over existing file
Installing /opt/app/.venv/lib/python3.11/site-packages/nvidia/__init__.py over existing file
Installing /opt/app/.venv/lib/python3.11/site-packages/nvidia/__init__.py over existing file
- Installing aiohttp (3.9.3)
- Installing appdirs (1.4.4)
- Installing asttokens (2.4.1)
- Installing blessed (1.20.0)
- Installing blinker (1.7.0)
- Installing blis (0.7.11)
- Installing boto3 (1.34.67)
- Installing catalogue (1.0.2)
- Installing click (8.1.7)
- Installing contourpy (1.2.0)
- Installing cycler (0.12.1)
- Installing dill (0.3.7)
- Installing distlib (0.3.8)
- Installing faker (24.3.0)
- Installing fonttools (4.50.0)
- Installing executing (2.0.1)
- Installing fsspec (2024.3.1)
- Installing google-api-core (2.17.1)
- Installing greenlet (3.0.3)
- Installing httpcore (1.0.4)
- Installing itsdangerous (2.1.2)
- Installing joblib (1.3.2)
- Installing kiwisolver (1.4.5)
- Installing lightning-utilities (0.11.0)
- Installing llvmlite (0.42.0)
- Installing nvidia-ml-py (12.535.133)
- Installing opencensus-context (0.1.3)
- Installing pandas (2.1.4)
- Installing parso (0.8.3)
- Installing patsy (0.5.6)
- Installing pillow (10.2.0)
- Installing plac (1.1.3)
- Installing preshed (3.0.9)
- Installing psutil (5.9.8)
- Installing ptyprocess (0.7.0)
- Installing pure-eval (0.2.2)
- Installing pydantic (1.10.14)
- Installing pyparsing (3.1.2)
- Installing pyrsistent (0.20.0)
- Installing pysocks (1.7.1)
- Installing pyyaml (6.0.1)
- Installing rich (13.7.1)
- Installing runs (1.2.2)
- Installing scipy (1.11.4)
- Installing smmap (5.0.1)
- Installing soupsieve (2.5)
- Installing srsly (1.0.7)
- Installing starlette (0.27.0)
- Installing tbb (2021.11.0)
- Installing tenacity (8.2.3)
- Installing threadpoolctl (3.4.0)
- Installing torch (2.0.0)
- Installing tqdm (4.66.2)
- Installing traitlets (5.14.2)
- Installing wasabi (0.10.1)
- Installing werkzeug (3.0.1)
- Installing wrapt (1.16.0)
- Installing zipp (3.18.1)
- Installing aiohttp-cors (0.7.0)
- Installing autogluon-common (1.0.0)
- Installing beautifulsoup4 (4.12.3)
- Installing cloudpickle (3.0.0)
- Installing colorama (0.4.6)
- Installing colorful (0.5.6)
- Installing cython (3.0.9)
- Installing daal (2024.1.0)
- Installing dash-core-components (2.0.0)
- Installing dash-html-components (2.0.0)
- Installing dash-table (5.0.0)
- Installing decorator (5.1.1)
- Installing distro (1.9.0)
- Installing editor (1.6.6)
- Installing fastapi (0.104.1)
- Installing fastcore (1.5.29)
- Installing fastprogress (1.0.3)
- Installing flask (3.0.2)
- Installing future (1.0.0)
- Installing gitdb (4.0.11)
- Installing gpustat (1.1.1)
- Installing graphviz (0.20.3)
- Installing grpcio (1.62.1)
- Installing httpx (0.27.0)
- Installing huggingface-hub (0.21.4)
- Installing importlib-metadata (7.1.0)
- Installing jedi (0.19.1)
- Installing jsonschema (4.17.3)
- Installing mako (1.3.2)
- Installing markdown (3.6)
- Installing matplotlib (3.7.5)
- Installing matplotlib-inline (0.1.6)
- Installing msgpack (1.0.8)
- Installing msgspec (0.18.6)
- Installing multiprocess (0.70.15)
- Installing mypy-extensions (1.0.0)
- Installing nest-asyncio (1.6.0)
- Installing numba (0.59.1)
- Installing opencensus (0.11.4)
- Installing openxlab (0.0.11)
- Installing ordered-set (4.1.0)
- Installing pexpect (4.9.0)
- Installing platformdirs (4.2.0)
Installing /opt/app/.venv/lib/python3.11/site-packages/opencensus/__init__.py over existing file
Installing /opt/app/.venv/lib/python3.11/site-packages/opencensus/common/__init__.py over existing file
- Installing plotly (5.20.0)
- Installing polyfactory (2.15.0)
- Installing prometheus-client (0.20.0)
- Installing prompt-toolkit (3.0.43)
- Installing py-spy (0.3.14)
- Installing py4j (0.10.9.7)
- Installing pyarrow (14.0.2)
- Installing pycryptodome (3.20.0)
- Installing pyjwt (2.8.0)
- Installing python-multipart (0.0.9)
- Installing readchar (4.0.6)
- Installing regex (2023.12.25)
- Installing retrying (1.3.4)
- Installing rich-click (1.7.4)
- Installing scikit-learn (1.4.1.post1)
- Installing smart-open (7.0.3)
- Installing sqlalchemy (2.0.28)
- Installing stack-data (0.6.3)
- Installing statsmodels (0.14.1)
- Installing tensorboardx (2.6.2.2)
- Installing thinc (7.4.6)
- Installing toolz (0.12.1)
- Installing torchmetrics (1.1.2)
- Installing types-python-dateutil (2.9.0.20240316)
- Installing uvicorn (0.29.0)
- Installing virtualenv (20.4.7)
- Installing websocket-client (1.7.0)
- Installing websockets (11.0.3)
- Installing xxhash (3.4.1)
- Installing absl-py (2.1.0)
- Installing aiofiles (23.2.1)
- Installing alembic (1.13.1)
- Installing altair (5.2.0)
- Installing arrow (1.3.0)
- Installing async-timeout (4.0.3)
- Installing autogluon-core (1.0.0)
- Installing autogluon-features (1.0.0)
- Installing antlr4-python3-runtime (4.9.3)
- Installing backoff (2.2.1)
- Installing comm (0.2.2)
- Installing catboost (1.2.3)
- Installing datasets (2.14.4)
- Installing daal4py (2024.1.0)
- Installing dash (2.16.1)
- Installing dateutils (0.6.12)
- Installing deepdiff (6.7.1)
- Installing croniter (1.4.1)
- Installing docker (7.0.0)
- Installing dynaconf (3.2.5)
- Installing entrypoints (0.4)
- Installing fastdownload (0.0.7)
- Installing fastjsonschema (2.19.1)
- Installing ffmpy (0.3.2)
- Installing gdown (5.1.0)
- Installing gitpython (3.1.42)
- Installing gradio-client (0.6.1)
- Installing gunicorn (21.2.0)
- Installing hyperopt (0.2.7)
- Installing imageio (2.34.0)
- Installing importlib-resources (6.3.2)
- Installing inquirer (3.2.4)
- Installing ipython (8.22.2)
- Installing iterative-telemetry (0.0.8)
- Installing jupyter-core (5.7.2)
- Installing jupyterlab-widgets (3.0.10)
- Installing lazy-loader (0.3)
- Installing lightgbm (4.1.0)
- Installing lightning-cloud (0.5.65)
- Installing litestar (2.7.0)
- Installing model-index (0.1.11)
- Installing nltk (3.8.1)
- Installing opendatalab (0.0.10)
- Installing orjson (3.9.15)
- Installing pmdarima (2.0.4)
- Installing pydub (0.25.1)
- Installing pytorch-lightning (2.0.9.post0)
- Installing pywavelets (1.5.0)
- Installing querystring-parser (1.2.4)
- Installing ray (2.6.3)
- Installing responses (0.18.0)
- Installing safetensors (0.4.2)
- Installing scikit-base (0.7.5)
- Installing semantic-version (2.10.0)
- Installing sentencepiece (0.2.0)
- Installing spacy (2.3.9)
- Installing sqlparse (0.4.4)
- Installing starsessions (1.3.0)
- Installing tabulate (0.9.0)
- Installing tensorboard-data-server (0.7.2)
- Installing tifffile (2024.2.12)
- Installing tokenizers (0.13.3)
- Installing torchvision (0.15.1)
- Installing tsdownsample (0.1.2)
- Installing typer (0.9.0)
- Installing typing-inspect (0.9.0)
- Installing utilsforecast (0.0.10)
- Installing watchdog (4.0.0)
- Installing widgetsnbextension (4.0.10)
- Installing window-ops (0.0.15)
- Installing xgboost (2.0.3)
/opt/app/.venv/lib/python3.11/site-packages/ray/dashboard/modules/log/log_agent.py:217: SyntaxWarning: "is not" with a literal. Did you mean "!="?
keep_alive_interval_sec >= 0 and end_offset is not -1
- Installing accelerate (0.21.0)
- Installing autogluon-tabular (1.0.0)
- Installing category-encoders (2.6.3)
- Installing defusedxml (0.7.1)
- Installing deprecation (2.1.0)
- Installing evaluate (0.4.1)
- Installing evidently (0.4.17)
- Installing fastai (2.7.14)
- Installing gluonts (0.14.4)
- Installing gradio (3.50.2)
- Installing imbalanced-learn (0.12.0)
- Installing ipywidgets (8.1.2)
- Installing kaleido (0.2.1)
- Installing kmodes (0.12.2)
- Installing lightning (2.0.9.post0)
- Installing m2cgen (0.10.0)
- Installing mlflow (2.10.2)
- Installing mlforecast (0.10.0)
- Installing nbformat (5.10.3)
- Installing mlxtend (0.23.1)
- Installing nlpaug (1.1.11)
- Installing nptyping (2.4.1)
- Installing nvidia-ml-py3 (7.352.0)
Installing /opt/app/.venv/lib/python3.11/site-packages/pynvml.py over existing file
- Installing omegaconf (2.2.3)
- Installing openmim (0.3.9)
- Installing plotly-resampler (0.9.2)
- Installing pyod (1.1.3)
- Installing pytesseract (0.3.10)
- Installing pytorch-metric-learning (1.7.3)
- Installing schemdraw (0.15)
- Installing scikit-image (0.20.0)
- Installing scikit-learn-intelex (2024.1.0)
- Installing scikit-plot (0.3.7)
- Installing seqeval (1.2.2)
- Installing sktime (0.27.0)
- Installing statsforecast (1.4.0)
- Installing tbats (1.1.3)
- Installing tensorboard (2.16.2)
- Installing text-unidecode (1.3)
- Installing timm (0.9.16)
- Installing transformers (4.31.0)
- Installing wurlitzer (3.0.3)
- Installing yellowbrick (1.5)
Installing /opt/app/.venv/lib/python3.11/site-packages/docs/source/conf.py over existing file
- Installing autogluon-multimodal (1.0.0)
- Installing autogluon-timeseries (1.0.0)
- Installing pycaret (3.3.0)
- Installing autogluon (1.0.0)
- Installing fhdw-modelling (7.2.0)
- Installing redis (5.0.3)
Installing the current project: exampleproject (0.0.1)
|
|
I have found a workaround to achieve what I am looking for. Originally I wanted to have a folder / project structure like: so that I could run the container somelike in the following docker run --rm examplepackage:latest processes/examplescript.py -hNow that the global python does not hold the packages anymore, when having to create a virtual env, I cannot run the scripts just with the previous ENTRYPOINT [ "python" ]I had to change this to the following now: To activate the environment I could change the adapted the ENTRYPOINT [ "poetry", "run", "python", "-m", "package.processes.examplescript" ]
CMD [ "-h" ]and can now run it with docker run --rm exampleproject:latestwhich directly runs the script. This is working for now. Full Dockerfile: FROM python:3.11-slim-bookworm
RUN apt update
RUN apt install -y git-all libgomp1
RUN apt clean
# Essentials
WORKDIR /app
COPY ./package ./package
COPY setup.cfg .
COPY pyproject.toml .
COPY poetry.lock .
COPY README.md .
# install package and dependencies
RUN python -m pip install --no-cache-dir poetry==1.8.2
RUN poetry config virtualenvs.in-project true
RUN poetry install --only main --compile
ENTRYPOINT [ "poetry", "run", "python", "-m", "package.processes.examplescript" ]
CMD [ "-h" ]Discussion: I am not 100% pleased with this solution. Maybe this is a matter of taste but when working with containers I always think of the container as a whole as the isolated portion of my code. Having to add an additional virtual env inside the container feels like overhead to me or let's say like something double. Is this really the intended use case here? Since I first totally identified with the statement from the docs saying there are valid reasons why it’s not necessary and is an overhead, like when working with containers. This is actually my very use case here. For example the next step could now be to add another script, but I would like to reuse this environment / Dockerfile; then I would have to build the container again (maybe different Tag) or alternatively change the Entrypoint again, so that I give the Maybe there is a more neat or artful way of solving this, like I did. How would you solve this use case of running a script ? |
You can just do ENTRYPOINT ["/app/.venv/bin/python"]Or you can add the env activation into your shell profile. Alternatively, you can also prepend the virtualenv path as well. ENV PATH="/app/.venv/bin:$PATH"This will let you keep the previous entrypoint, or even better use a shell/bash entrypoint and load your "examplescript" as an entrypoint console script in your poetry project :) Getting you something like this. docker run --rm examplepackage:latest example-script -h
This is a common misconception imho. Also the "default" site inside most containers are externally managed and should not installed to via pip or poetry etc. And the only real overhead aer some directories being created to contain the specific set of dependencies you need for your project, and for you get some really nice benefits. Once again, I would strongly urge that virtual environments be always used, even within containers. Also, in your case you might want to consider using cache mounts for your builds. For both the apt caches as well as |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Installing (local) package / project with poetry fails when attempting to containerize my app (tbh just a single script).
Installation on dev machine runs fine. From here I copy the necessary files into the container:
Workarounds
The only way to get the installation process working is to omit
RUN poetry config virtualenvs.create falseBut this is not a solution since I would then have to activate the venv inside the running container, but that would not make much sense. Also the instructions in poetry docs basically describe this scenario for containers.
Poetry Installation Method
pip
Operating System
official python container image (python:3.11-slim-bookworm)
Poetry Version
1.8.2
Poetry Configuration
sorry cannot provide since the build is not successful, have no chance to attach to the full contianer.
From an intermediate container I get the following result:
Python Sysconfig
Also from the intermediate container:
Example pyproject.toml
No response
Poetry Runtime Logs
The text was updated successfully, but these errors were encountered: