[DO NOT MERGE, DO NOT REVIEW] Switch to poetry #2192

aucampia · 2023-01-02T19:23:59Z

Just testing things so I don't create spam on #2187

- also consolidating other settings to pyproject.toml when possible

- both envs fail when run - not clear that the failures are new since they seem unrelated to my changes

…"extra" html

… now

… str

…est poetry-core. - poetry-core is at 1.4.0; require that to avoid risk of encountering old bugs - "flake8" was causing problems with pip during validate jobs, in which pip tried to do this: py37: install_package_deps> python -I -m pip install black==22.12.0 'flake8>=4.0.1; or extra == "flake8"' 'flakeheaven<4.0.0,>=3.2.1; or extra == "flake8"' 'importlib_metadata<5.0.0,>=4.2.0; python_version >= "3.7" and python_version < "3.8"' 'isodate<0.7.0,>=0.6.1' 'isort<6.0.0,>=5.10.0' mypy==0.991 'networkx<3.0.0,>=2.6.2; (python_full_version > "3.7.0" and python_version < "3.8") and extra == "networkx"' 'networkx<3.0.0,>=2.8.8; (python_version >= "3.8" and python_version < "4.0") and extra == "networkx"' 'pep8-naming<0.14.0,>=0.13.2; or extra == "flake8"' 'pyparsing<4.0.0,>=3.0.9' 'pytest-cov<5.0.0,>=4.0.0' 'pytest<8.0.0,>=7.1.3' and complained about: pip._vendor.packaging.markers.InvalidMarker: Invalid marker: 'or extra == "flake8"', parse error at 'or extra'

…stead of requirements.txt files

…ependencies - exception: docker:prepare task is not yet updated

coveralls · 2023-01-03T19:04:45Z

Coverage: 90.656% (+0.002%) from 90.654% when pulling 807ad03 on aucampia:switch-to-poetry into 9b778b3 on RDFLib:main.

- `.github/workflows/validate.yaml`: - Cache the default `XDG_CACHE_HOME` instead of messing with `PIP_CACHE_DIR` and the variable. This simplifies caching quite a bit and also ensures caching works for poetry. - Install poetry for `extra-tasks`, so that poetry install works correctly. - Get an auth token in `extra-tasks` so we don't get hit by rate limiting. - `docs/conf.py`: - Disable nit picking on pre-python 3.7 as this causes problems with older sphinx. - `docs/developers.rst` and `docs/docs.rst`: - Incorporate poetry into instructions. - `docker-compose.yml`: - Set `POETRY_VIRTUALENVS_IN_PROJECT=1` so that the `.venv` goes into a docker volume instead of going to emphemeral storage. - `Dockerfile.devcontainer` - Don't install dependencies globally. With poetry, everyone should use poetry, and when using poetry the global python environment should not be used, so if people use the devcontainer correctly the global dependencies should have no effect. - `poetry.lock` - Ran `poetry lock` after changes to `pyproject.toml`, most notable change is the stuff added to the dev Poetry dependency group is no longer optional. - `pyproject.toml`: - Simplified the dependency constrains for `networkx` as this was causing problems with tox, as tox was running pip install with `'networkx<3.0.0,>=2.6.2; (python_version >= "3.7" and python_version < "3.8") and extra == "networkx"' 'networkx<3.0.0,>=2.8.8; (python_version >= "3.8" and python_version < "4.0") and extra == "networkx"'` and the `extra == "networkx"` here would never be true. - Added a `dev` poetry dependendcy group with everything that is needed for a sane development environment so that `poetry install` gets you a sane dev environment. - `Taskfile.yml`: - Changed the commands so that everything works as before but now instead runs things with poetry. - Removed the legacy venv handling and replaced it with poetry equivalents. - `tox.ini`: - Don't set nitpicky (`-n`) on the command line as it is being set conditionally on python version in `docs/conf.py` now.

poetry related changes

jclerman · 2023-01-10T04:35:10Z

tox.ini

@@ -16,7 +16,7 @@ setenv =
    extensive: POETRY_ARGS_extensive = --extras=html --extras=berkeleydb --extras=networkx
    lxml: POETRY_ARGS_lxml = --extras=lxml
 commands_pre =
-    py3{7,8,9,10,11}: poetry --version
+    py3{7,8,9,10,11}: poetry lock --check
    py3{7,8,9,10,11}: poetry install --no-root --only=main --only=dev --only=tests --extras=html  {env:POETRY_ARGS_docs:} {env:POETRY_ARGS_extensive:} {env:POETRY_ARGS_lxml:} --sync


There may be an issue with using "poetry install" here instead of "poetry export" followed by "pip install" using the exported requirements. The problem is that "poetry install" will install into a poetry-controlled venv - but tox creates/uses its own virtual environments. Even if the former method works to some extent, it'll start to be problematic with tox -p, since the various poetry install commands will all be installing into the same venv. That causes much pain, as I've learned.

By the way, instead of the multiple specifications of "--only", which could be confusing, you could also do --with=dev --with=tests - main will be included by default.

poetry seems to be installing into the virtual environment indicated by VIRTUAL_ENV - so for tox -e py37 it is using .tox/py37: https://gist.github.com/aucampia/acc2fb08b0f577a7592042e1021f8910#file-output-txt-L69-L83

I will commit shortly with more debugging output and link to the build output there, but as far as I can tell it is fine.

It was surprising to me that it worked when I noticed it but I think it makes some sense. See also:

Inconsistent treatment of the VIRTUAL_ENV variable python-poetry/poetry#2056

How can I ignore the 'VIRTUAL_ENV' environment variable? python-poetry/poetry#281

This seems to be a consequence of:

https://github.com/python-poetry/poetry/blob/3fe8838659415f7495d415503ff64eea1aa0bf57/src/poetry/utils/env.py#L691-L752

This is also how poetry recommends using it: https://python-poetry.org/docs/faq/#usecase-2

I asked about documentation about this here.

Build log can be seen here: https://github.com/RDFLib/rdflib/actions/runs/3887504878/jobs/6633837906#step:8:224

For this commit: https://github.com/aucampia/rdflib/tree/2292b877281f0f29d20701c445064677c6551a31

This is mainly so that poetry export is sane, but also because the extras should not be polluted with things that peope who use the wheel don't want or use. Other changes: - GitHub Actions: - Use `abatilo/actions-poetry` instead of `snok/install-poetry` as snok does not work well on Windows. - Install poetry before running tox so tox can call poetry. - Fix caching on Windows and MacOS - Add requirements files for poetry so its versions is managed by dependabot and so that the same version can be used in multiple contexts. - Add requirements file for Sphinx version used in readthedocs. - The problem with nitpicky was related to sphinx 4, not to python 3.7. So nitpicky is now disabled on Sphinx <= 5. - `test/test_literal/test_xmlliterals.py`: Only run html5lib test when html5lib is present. - Check poetry and poetry.lock from `.pre-commit-config.yaml`. - Changed `.readthedocs.yaml` to use poetry. - Added an `extra` for lxml.

This reverts commit fa1cc18.

Move dev dependencies from extras into poetry dep groups

This adds a tox environment and GHA test matrix entry for testing the lowest allowed versions of dependencies.

test minimum versions and decrease the lower bounds for dependencies to earliest versions that pass

This reverts commit a4da002.

…o install poetry with all extras flag.

…mmands inside the devcontainer with docker volumes from the host

This should avoid interference between different containers and the host environment.

Fix devcontainer stuff

change `Black` -> `isort` in isort config.

aucampia · 2023-01-18T08:19:38Z

Closing this as it has served its purpose.

jclerman added 10 commits December 28, 2022 21:30

first draft of moving to poetry for dependency management

c8e1f2e

- also consolidating other settings to pyproject.toml when possible

update to get docs and flake8 tox environments running.

debd140

- both envs fail when run - not clear that the failures are new since they seem unrelated to my changes

make html5lib an optional dependency, configured for installation as …

bd1d4c7

…"extra" html

preserving all previous extras, doing away with poetry dep-groups for…

5609c41

… now

type-annotation to reassure mypy that __version__ is expected to be a…

5fd8da2

… str

install devcontainer development dependencies from pyproject.toml, in…

7f04648

…stead of requirements.txt files

update Taskfile to use pyproject.toml & poetry to determine/install d…

a4d9d57

…ependencies - exception: docker:prepare task is not yet updated

move coverage config to pyproject.toml; update tox.ini to accommodate

39184f5

remove newline from package description to mollify poetry

d6a97ca

aucampia force-pushed the switch-to-poetry branch from 2f6bd3b to b170655 Compare January 2, 2023 21:29

Merge branch 'main' into switch-to-poetry

16e61a4

aucampia force-pushed the switch-to-poetry branch from f86bb60 to 2ad4c9e Compare January 3, 2023 19:51

aucampia mentioned this pull request Jan 3, 2023

move to poetry for dependency management; consolidate more settings into pyproject.toml #2187

Merged

6 tasks

aucampia force-pushed the switch-to-poetry branch from 24a1969 to 46e30c2 Compare January 3, 2023 23:10

aucampia mentioned this pull request Jan 3, 2023

poetry related changes jclerman/rdflib#1

Merged

aucampia requested review from a team and removed request for a team January 3, 2023 23:20

jclerman and others added 3 commits January 3, 2023 16:00

Merge pull request #1 from aucampia/switch-to-poetry

6aa9d6d

poetry related changes

removing now-obsolete packaging files

c873a53

updating library-development-related docs to reflect use of poetry

7341453

jclerman reviewed Jan 10, 2023

View reviewed changes

aucampia force-pushed the switch-to-poetry branch 2 times, most recently from 752f61a to a5ecf45 Compare January 10, 2023 22:54

aucampia force-pushed the switch-to-poetry branch from a5ecf45 to fe43722 Compare January 10, 2023 23:09

edmondchuc added 2 commits January 11, 2023 13:27

chown mounted container volumes

fa1cc18

Revert "chown mounted container volumes"

8c113cd

This reverts commit fa1cc18.

Merge pull request #2 from aucampia/switch-to-poetry

9539db6

Move dev dependencies from extras into poetry dep groups

aucampia force-pushed the switch-to-poetry branch 2 times, most recently from bc183bd to bf6ca06 Compare January 12, 2023 21:46

test: minimum versions

95021f6

This adds a tox environment and GHA test matrix entry for testing the lowest allowed versions of dependencies.

aucampia force-pushed the switch-to-poetry branch from bf6ca06 to 95021f6 Compare January 12, 2023 21:49

jclerman and others added 5 commits January 12, 2023 23:59

Merge pull request #4 from aucampia/switch-to-poetry

c154844

test minimum versions and decrease the lower bounds for dependencies to earliest versions that pass

Add chown for container volumes

ead5864

Revert "Add chown for container volumes"

76d82c7

This reverts commit a4da002.

Add separate docker-compose for devcontainer. Add postCreateCommand t…

be4bba6

…o install poetry with all extras flag.

Use one docker-compose.yml with a new 'run' service to run one-off co…

1832327

…mmands inside the devcontainer with docker volumes from the host

aucampia force-pushed the switch-to-poetry branch from 2e0419c to 2be43bf Compare January 16, 2023 00:06

change local state directories

2cee383

This should avoid interference between different containers and the host environment.

aucampia force-pushed the switch-to-poetry branch from 2be43bf to 2cee383 Compare January 16, 2023 00:09

jclerman and others added 3 commits January 15, 2023 19:40

Merge pull request #5 from aucampia/switch-to-poetry

20e4b17

Fix devcontainer stuff

bump versions

0a5db59

build: fix comment

807ad03

change `Black` -> `isort` in isort config.

aucampia closed this Jan 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DO NOT MERGE, DO NOT REVIEW] Switch to poetry #2192

[DO NOT MERGE, DO NOT REVIEW] Switch to poetry #2192

aucampia commented Jan 2, 2023 •

edited

coveralls commented Jan 3, 2023 •

edited

jclerman Jan 10, 2023

jclerman Jan 10, 2023

aucampia Jan 10, 2023 •

edited

aucampia Jan 10, 2023

aucampia commented Jan 18, 2023

[DO NOT MERGE, DO NOT REVIEW] Switch to poetry #2192

[DO NOT MERGE, DO NOT REVIEW] Switch to poetry #2192

Conversation

aucampia commented Jan 2, 2023 • edited

coveralls commented Jan 3, 2023 • edited

jclerman Jan 10, 2023

Choose a reason for hiding this comment

jclerman Jan 10, 2023

Choose a reason for hiding this comment

aucampia Jan 10, 2023 • edited

Choose a reason for hiding this comment

aucampia Jan 10, 2023

Choose a reason for hiding this comment

aucampia commented Jan 18, 2023

aucampia commented Jan 2, 2023 •

edited

coveralls commented Jan 3, 2023 •

edited

aucampia Jan 10, 2023 •

edited