Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Support dynamic optional-dependencies in hatch env #1349

Open
potiuk opened this issue Mar 25, 2024 · 2 comments
Open

Feature: Support dynamic optional-dependencies in hatch env #1349

potiuk opened this issue Mar 25, 2024 · 2 comments

Comments

@potiuk
Copy link
Contributor

potiuk commented Mar 25, 2024

Currently when project defines "optional-dependencies" as dynamic property, hatch cannot use such dynamic property in env definition.

Example is Apache Airflow. After we turned optional-dependencies to dynamic field, we had to remove devel from features:

https://github.com/apache/airflow/blob/main/pyproject.toml#L159

[tool.hatch.envs.default]
python = "3.8"
platforms = ["linux", "macos"]
description = "Default environment with Python 3.8 for maximum compatibility"
features = []

When you try to use devel feature:

[tool.hatch.envs.default]
python = "3.8"
platforms = ["linux", "macos"]
description = "Default environment with Python 3.8 for maximum compatibility"
features = ["devel"]  # <-- add `devel` feature here

The hatch env command almost immediately fails with:

ValueError: Feature `devel` of field `tool.hatch.envs.default.features` is not defined in field `project.optional-dependencies`
@potiuk potiuk changed the title Support dynamic optional-dependencies in hatch env Feature: Support dynamic optional-dependencies in hatch env Mar 25, 2024
@potiuk potiuk changed the title Feature: Support dynamic optional-dependencies in hatch env Feature: Support dynamic optional-dependencies in hatch env Mar 25, 2024
@ofek
Copy link
Collaborator

ofek commented Apr 14, 2024

Can you please try master to see if this fixes it? #1387

@potiuk
Copy link
Contributor Author

potiuk commented Apr 15, 2024

Tried, seems it still does not work for airflow:

  1. Checked out latest airflow (https://github.com/apache/airflow)

  2. Installed latest hatch pip install "hatch @ git+https://github.com/pypa/hatch"

  3. Double check:

[jarek:~/code/airflow] [apache-airflow] main+ ± hatch --version                                                                                        
Hatch, version 1.9.2.dev90
  1. Modify pyproject.toml:
[tool.hatch.envs.default]
python = "3.8"
platforms = ["linux", "macos"]
description = "Default environment with Python 3.8 for maximum compatibility"
features = []

to:

[tool.hatch.envs.default]
python = "3.8"
platforms = ["linux", "macos"]
description = "Default environment with Python 3.8 for maximum compatibility"
features = ["devel"]
  1. hatch env create
/home/jarek/.pyenv/versions/3.10.13/lib/python3.10/site-packages/hatch/cli/__init__.py:221 in    │
│ main                                                                                             │
│                                                                                                  │
│   218                                                                                            │
│   219 def main():  # no cov                                                                      │
│   220 │   try:                                                                                   │
│ ❱ 221 │   │   hatch(prog_name='hatch', windows_expand_args=False)                                │
│   222 │   except Exception:  # noqa: BLE001                                                      │
│   223 │   │   import sys                                                                         │
│   224                                                                                            │
│                                                                                                  │
│ /home/jarek/.pyenv/versions/3.10.13/lib/python3.10/site-packages/click/core.py:1157 in __call__  │
│                                                                                                  │
│ /home/jarek/.pyenv/versions/3.10.13/lib/python3.10/site-packages/click/core.py:1078 in main      │
│                                                                                                  │
│ /home/jarek/.pyenv/versions/3.10.13/lib/python3.10/site-packages/click/core.py:1688 in invoke    │
│                                                                                                  │
│ /home/jarek/.pyenv/versions/3.10.13/lib/python3.10/site-packages/click/core.py:1688 in invoke    │
│                                                                                                  │
│ /home/jarek/.pyenv/versions/3.10.13/lib/python3.10/site-packages/click/core.py:1434 in invoke    │
│                                                                                                  │
│ /home/jarek/.pyenv/versions/3.10.13/lib/python3.10/site-packages/click/core.py:783 in invoke     │
│                                                                                                  │
│ /home/jarek/.pyenv/versions/3.10.13/lib/python3.10/site-packages/click/decorators.py:45 in       │
│ new_func                                                                                         │
│                                                                                                  │
│ /home/jarek/.pyenv/versions/3.10.13/lib/python3.10/site-packages/hatch/cli/env/create.py:38 in   │
│ create                                                                                           │
│                                                                                                  │
│   35 │   │   │                                                                                   │
│   36 │   │   │   app.abort(f'Environment `{env}` is incompatible: {e}')                          │
│   37 │   │                                                                                       │
│ ❱ 38 │   │   app.prepare_environment(environment)                                                │
│   39 │                                                                                           │
│   40 │   if incompatible:                                                                        │
│   41 │   │   num_incompatible = len(incompatible)                                                │
│                                                                                                  │
│ /home/jarek/.pyenv/versions/3.10.13/lib/python3.10/site-packages/hatch/cli/application.py:125 in │
│ prepare_environment                                                                              │
│                                                                                                  │
│   122 │   │   │   │                                                                              │
│   123 │   │   │   │   with environment.app_status_project_installation():                        │
│   124 │   │   │   │   │   if environment.dev_mode:                                               │
│ ❱ 125 │   │   │   │   │   │   environment.install_project_dev_mode()                             │
│   126 │   │   │   │   │   else:                                                                  │
│   127 │   │   │   │   │   │   environment.install_project()                                      │
│   128                                                                                            │
│                                                                                                  │
│ /home/jarek/.pyenv/versions/3.10.13/lib/python3.10/site-packages/hatch/env/virtual.py:164 in     │
│ install_project_dev_mode                                                                         │
│                                                                                                  │
│   161 │   def install_project_dev_mode(self):                                                    │
│   162 │   │   with self.safe_activation():                                                       │
│   163 │   │   │   self.platform.check_command(                                                   │
│ ❱ 164 │   │   │   │   self.construct_pip_install_command(['--editable', self.apply_features(st   │
│   165 │   │   │   )                                                                              │
│   166 │                                                                                          │
│   167 │   def dependencies_in_sync(self):                                                        │
│                                                                                                  │
│ /home/jarek/.pyenv/versions/3.10.13/lib/python3.10/site-packages/hatch/env/plugin/interface.py:9 │
│ 19 in apply_features                                                                             │
│                                                                                                  │
│    916 │   │   A convenience method that applies any user defined [features](../../config/envir  │
│    917 │   │   to the given requirement.                                                         │
│    918 │   │   """                                                                               │
│ ❱  919 │   │   if self.features:                                                                 │
│    920 │   │   │   features = ','.join(self.features)                                            │
│    921 │   │   │   return f'{requirement}[{features}]'                                           │
│    922                                                                                           │
│                                                                                                  │
│ /home/jarek/.pyenv/versions/3.10.13/lib/python3.10/site-packages/hatch/env/plugin/interface.py:4 │
│ 43 in features                                                                                   │
│                                                                                                  │
│    440 │   │   │   │   │   │   f'Feature `{normalized_feature}` of field `tool.hatch.envs.{self  │
│    441 │   │   │   │   │   │   f'defined in field `project.optional-dependencies`'               │
│    442 │   │   │   │   │   )                                                                     │
│ ❱  443 │   │   │   │   │   raise ValueError(message)                                             │
│    444 │   │   │   │                                                                             │
│    445 │   │   │   │   all_features.add(normalized_feature)                                      │
│    446                                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Feature `devel` of field `tool.hatch.envs.default.features` is not defined in field `project.optional-dependencies`
  1. double check that installation with devel extra works with pip in editable mode:

Change back pyproject.toml tpo features=[ ]

hatch env create -> works and install airflow in editable mode without devel deps (for example without pytest)

> hatch shell

> python -c 'import pytest'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pytest'

Re-install airflow in --editable mode and with devel extras:

  • pip install -e ".[devel]"
Successfully installed SecretStorage-3.3.3 aiofiles-23.2.1 aioresponses-0.7.6 amqp-5.2.0 apache-airflow-2.10.0.dev0 asttokens-2.4.1 backcall-0.2.0 backports.tarfile-1.0.0 beautifulsoup4-4.12.3 billiard-4.2.0 black-24.4.0 cachetools-5.3.3 celery-5.3.6 cfgv-3.4.0 click-didyoumean-0.3.1 click-plugins-1.1.1 click-repl-0.3.0 coverage-7.4.4 decorator-5.1.1 distlib-0.3.8 duckdb-0.10.1 editables-0.5 execnet-2.1.1 executing-2.0.1 filelock-3.13.4 flower-2.0.1 gitdb-4.0.11 gitpython-3.1.43 google-auth-2.29.0 hatch-1.9.4 hatchling-1.21.1 humanize-4.9.0 hyperlink-21.0.0 icdiff-2.0.7 identify-2.5.35 incremental-22.10.0 iniconfig-2.0.0 ipdb-0.13.13 ipython-8.12.3 jaraco.classes-3.4.0 jaraco.context-5.3.0 jaraco.functools-4.0.0 jedi-0.19.1 jeepney-0.8.0 keyring-25.1.0 kombu-5.3.7 kubernetes-29.0.0 kubernetes_asyncio-29.0.0 matplotlib-inline-0.1.7 mypy-1.9.0 mypy-extensions-1.0.0 nh3-0.2.17 nodeenv-1.8.0 oauthlib-3.2.2 parso-0.8.4 pexpect-4.9.0 pickleshare-0.7.5 pip-24.0 pipdeptree-2.18.1 pkginfo-1.10.0 platformdirs-4.2.0 pprintpp-0.4.0 pre-commit-3.5.0 prometheus-client-0.20.0 prompt-toolkit-3.0.43 ptyprocess-0.7.0 pure-eval-0.2.2 pyasn1-0.6.0 pyasn1-modules-0.4.0 pygithub-2.3.0 pynacl-1.5.0 pytest-7.4.4 pytest-asyncio-0.23.6 pytest-cov-5.0.0 pytest-custom-exit-code-0.3.0 pytest-icdiff-0.9 pytest-instafail-0.5.0 pytest-mock-3.14.0 pytest-rerunfailures-14.0 pytest-timeouts-1.2.1 pytest-xdist-3.5.0 readme-renderer-43.0 redis-5.0.3 requests-oauthlib-2.0.0 requests_mock-1.12.1 restructuredtext-lint-1.4.0 rfc3986-2.0.0 rich-click-1.7.4 rsa-4.9 ruff-0.3.3 semver-3.0.2 shellingham-1.5.4 smmap-5.0.1 soupsieve-2.5 stack-data-0.6.3 tomli-2.0.1 tomli-w-1.0.0 tomlkit-0.12.4 tornado-6.4 towncrier-23.11.0 traitlets-5.14.2 trove-classifiers-2024.4.10 twine-5.0.0 types-Deprecated-1.2.9.20240311 types-Markdown-3.6.0.20240316 types-PyMySQL-1.1.0.1 types-PyYAML-6.0.12.20240311 types-aiofiles-23.2.0.20240403 types-certifi-2021.10.8.3 types-croniter-2.0.0.20240321 types-docutils-0.20.0.20240406 types-paramiko-3.4.0.20240311 types-protobuf-4.25.0.20240410 types-pyOpenSSL-24.0.0.20240311 types-python-dateutil-2.9.0.20240316 types-python-slugify-8.0.2.20240310 types-pytz-2024.1.0.20240203 types-redis-4.6.0.20240409 types-requests-2.31.0.20240406 types-setuptools-69.5.0.20240415 types-tabulate-0.9.0.20240106 types-termcolor-1.1.6.2 types-toml-0.10.8.20240310 userpath-1.9.2 vine-5.1.0 virtualenv-20.25.1 wcwidth-0.2.13 websocket-client-1.7.0 wheel-0.43.0 yamllint-1.35.1 zstandard-0.22.0

potiuk added a commit to potiuk/hatch that referenced this issue May 20, 2024
This is a very draft proposal on how to fix pypa#1348 and pypa#1349 - more
to see if this is a good direction. It should likely be split to
two PRs (and of course tests and docs are needed):

* extending custom metadata plugin to allow different versions for
  standard and editable builds (also including extending the hatchling
  CLIs.

* extending hatchling interface used by hatch to include standard/
  editable version when querying hatchling for available dependencies

Not all changes have been yet applied, this is more to check if
this is the right direction.
potiuk added a commit to potiuk/hatch that referenced this issue May 20, 2024
This is a very draft proposal on how to fix pypa#1348 and pypa#1349 - more
to see if this is a good direction. It should likely be split to
two PRs (and of course tests and docs are needed):

* extending custom metadata plugin to allow different versions for
  standard and editable builds (also including extending the hatchling
  CLIs.

* adding version to CLI where hatch queries hatchling to include standard/
  editable version when querying for available dependencies.

Not all changes have been yet applied, this is more to check if
this is the right direction.
potiuk added a commit to potiuk/hatch that referenced this issue May 27, 2024
This is a very draft proposal on how to fix pypa#1348 and pypa#1349 - more
to see if this is a good direction. It should likely be split to
two PRs (and of course tests and docs are needed):

* extending custom metadata plugin to allow passing version through
  hook's version property to distinguish standard and editable builds
  (also including extending the hatchling CLIs.

* adding version to CLI where hatch queries hatchling to include standard/
  editable version when querying for available dependencies.

Not all changes have been yet applied, this is more to check if
this is the right direction.
potiuk added a commit to potiuk/hatch that referenced this issue May 27, 2024
This is a very draft proposal on how to fix pypa#1348 and pypa#1349 - more
to see if this is a good direction. It should likely be split to
two PRs (and of course tests and docs are needed):

* extending custom metadata plugin to allow passing version through
  hook's version property to distinguish standard and editable builds
  (also including extending the hatchling CLIs.

* adding version to CLI where hatch queries hatchling to include standard/
  editable version when querying for available dependencies.

Not all changes have been yet applied, this is more to check if
this is the right direction.
potiuk added a commit to potiuk/hatch that referenced this issue May 27, 2024
This is a very draft proposal on how to fix pypa#1348 and pypa#1349 - more
to see if this is a good direction. It should likely be split to
two PRs (and of course tests and docs are needed):

* extending custom metadata plugin to allow passing version through
  hook's version property to distinguish standard and editable builds
  (also including extending the hatchling CLIs.

* adding version to CLI where hatch queries hatchling to include standard/
  editable version when querying for available dependencies.

Not all changes have been yet applied, this is more to check if
this is the right direction.
potiuk added a commit to potiuk/hatch that referenced this issue May 29, 2024
This is a very draft proposal on how to fix pypa#1348 and pypa#1349 - more
to see if this is a good direction. It should likely be split to
two PRs (and of course tests and docs are needed):

* extending custom metadata plugin to allow passing version through
  hook's version property to distinguish standard and editable builds
  (also including extending the hatchling CLIs.

* adding version to CLI where hatch queries hatchling to include standard/
  editable version when querying for available dependencies.

Not all changes have been yet applied, this is more to check if
this is the right direction.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants