Fork lexicon without poetry

[ArtKoKo]

Poetry add a complexity process for build in FreeBSD
Can you keep a branch of lexicon without using poetry ?

[adferrand]

In fact you can continue to use your workflow with pip/setuptools as before, and not care about poetry if you want, thanks to two mechanisms available here:

• recent versions of pip (>=10.0), implements PEP 517 and PEP 518, which is a convention to prepare in isolation the build system required in the poetry.toml of a project. Basically you can checkout the project, run pip install lexicon, and pip will fetch poetry on its own to install the project. There is a limitation for editable installations (pip install -e) which falls in the next point
• for older versions of pip, or for editable installations, an installation through setuptools is required and still possible. Indeed a shim for setup.py has been designed by the packaging community for retro-compatibility purpose, and I added it to Lexicon. In this case setuptools will kick in and you will not benefit from Poetry features, like advanced dependency graph resolution, but for your case I think it will be sufficient.

[ArtKoKo]

Thank you but when using setuptools, nothing gets installed, other that a few stub files:

  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/UNKNOWN-0.0.0-py3.8.egg-info/PKG-INFO
  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/UNKNOWN-0.0.0-py3.8.egg-info/SOURCES.txt
  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/UNKNOWN-0.0.0-py3.8.egg-info/dependency_links.txt
  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/UNKNOWN-0.0.0-py3.8.egg-info/top_level.txt

I must have missed something...

[adferrand]

No it is me. The shim does not cover the case of old versions of pip, just the case of pip install -e (editable mode). You still need at least pip 10.0+ to make it work through the build isolation.

Having a more recent version of pip on FreeBSD is not doable ?

[ArtKoKo]

No because I use an another package manager and I don't want to use pip to avoid package management issues...

[adferrand]

Which package manager are you using, and which version of FreeBSD ? Maybe I can figure out something.

[adferrand]

Also, depending of the use case:

• if it is for development, you should anyway use a Python virtual environment, and in it you will be able to use an up-to-date version of pip
• if it is to use it, the wheel is better than installing from sources (https://files.pythonhosted.org/packages/4b/3b/dcaa4f6a0930703c73d44e3c6c7520132a353a640180b9da1b5b3436ccb6/dns_lexicon-3.4.3-py3-none-any.whl)

[ArtKoKo]

I spoke with one of the developers of the package manager MacPorts:
The problem is that lexicon is a dependency of the certbot-dns extensions of certbot and that adding another package manager to handle the dependencies of the dependency obviously becomes counterproductive...

[adferrand]

Of course, but MacPorts maintainers need to find a solution for PEP 517 compliant packages, because more and more packages will move to pyproject.toml since it is a standard now.

I looked at MacPorts behavior, but failed to understand how a Python package is actually installed. I suppose it is something like python setup.py install, which is exclusive to setuptools. For a maximum compatibility I would recommend to use pip: in fact, pip would not be used as package manager here, but as the build frontend to install the package, with a command like pip install --no-deps. In this case pip will leverage the actual installation to setuptools for legacy, and the defined build system for PEP 517 compliant projects.

[Schamschula]

I'm the port maintainer that ArtKoKo has been referring to. The mechanism MacPorts uses to install Python packages is the python PortGroup:

https://github.com/macports/macports-ports/blob/master/_resources/port1.0/group/python-1.0.tcl

It currently works on nearly 6k modules and packages (with a lot of double counting due to multiple Python branches).

I had a look and could only find a small handful of ports that use py-pip (none use poetry, even though the package itself exists). If we were to use pip as a build system someone would first have to write a corresponding PortGroup to avoid adding the same code into a large number of Portfiles. As we are a volunteer organization, there first would need to be greater perceived demand before this happens.

[ArtKoKo]

@adferrand not inspired ?

[Schamschula]

Unfortunately, it seems that trying to use pip to install python modules within MacPorts is a fool's errand. I tried building another package that lacks setup.py but provides a .toml file (meshio), and found pip couldn't locate setuptools when specifying --no-dependencies.

  ERROR: Could not find a version that satisfies the requirement setuptools>=42 (from versions: none)
  ERROR: No matching distribution found for setuptools>=42

given

port installed py38-setuptools
The following ports are currently installed:
  py38-setuptools @49.6.0_0
  py38-setuptools @50.3.0_0 (active)

My next step is to try to figure out if I can get PEP 517 to work. Not too optimistic.

[adferrand]

Hello @Schamschula, what is the version of pip used in this case?

[adferrand]

@Schamschula I worked a little around this problem. I could get a similar situation where setuptools is missing and pip will fail entirely;

python3 -m venv venv
. venv/bin/activate
python3 -m pip uninstall --yes pkg-resources setuptools

At that point, the only remaining package in the virtual environment is pip. If you try to install a PEP-517 compliant package from sources, like python3 -m pip install --no-binary :all: meshio, you will get an error, because setuptools is missing and required to prepare isolation build environments:

Collecting meshio
  Using cached meshio-4.3.0.tar.gz (458 kB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /home/aferrand/Dev/test/venv/bin/python3 /home/aferrand/Dev/test/venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-7ui9wzt0/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=42' wheel
       cwd: None
  Complete output (13 lines):
  Collecting setuptools>=42
    Using cached setuptools-50.3.0.zip (2.2 MB)
  Collecting wheel
    Using cached wheel-0.35.1.tar.gz (59 kB)
      ERROR: Command errored out with exit status 1:
       command: /home/aferrand/Dev/test/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-nltdv2oz/wheel/setup.py'"'"'; __file__='"'"'/tmp/pip-install-nltdv2oz/wheel/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-qc4oe52i
           cwd: /tmp/pip-install-nltdv2oz/wheel/
      Complete output (3 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
      ModuleNotFoundError: No module named 'setuptools'
      ----------------------------------------
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /home/aferrand/Dev/test/venv/bin/python3 /home/aferrand/Dev/test/venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-7ui9wzt0/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=42' wheel Check the logs for full command output.

Say now that you have setuptools installed somewhere else. For instance, in another virtual environment located at /full/path/to/venv. The site-packages, containing the packages of the second virtual environment will be located at /full/path/to/venv/lib/python3.8/site-packages for Python 3.8.

Thanks to the PYTHONPATH environment variable, you can add additional locations to a given Python interpreter to find additional packages located at arbitrary places. Finally with the following command, the installation will work:

PYTHONPATH=/full/path/to/venv/lib/python3.8/site-packages python3 -m pip install --no-binary :all: meshio

Please note that pip must be invoked as a module (python3 -m pip) and not as an executable, otherwise the PYTHONPATH environment variable is not consumed.

I believe this approach would allow to inject the port of setuptools during the pip call in MacPorts.

[Schamschula]

I currently have

The following ports are currently installed:
  py38-pip @20.2.3_0 (active)

I also tried building lexicon (using the recipe from py-tensorflow_estimator https://github.com/macports/macports-ports/blob/9b6fda7da756e1ea03675e0906b37e2fb7412f20/python/py-tensorflow_estimator/Portfile)

In this case I get the same issue with poetry

  ERROR: Could not find a version that satisfies the requirement poetry>=0.12 (from versions: none)
  ERROR: No matching distribution found for poetry>=0.12

and, yes poetry is installed:

port installed poetry
The following ports are currently installed:
  poetry @1.0.3_0 (active)

[Schamschula]

MacPorts has recently added a pep517 mechanism. Thus, I've updated py-dns-lexicon to 3.5.2 for those versions of python that support it.
See: macports/macports-ports@38b820f and subsequent cleanup.

[Schamschula]

This issue has come up again: https://trac.macports.org/ticket/62682

The py-dns-lexicon Portfile requires both poetry and py-poetry-core. Apparently, for those who have multiple branches of python installed, forcing poetry and py-poetry-core to be of the same Python branch is causing build failures, particularly on the MacPorts build bots.

In the above ticket it was suggested to only use py-poetry-core, see Textualize/rich#289

[harens]

This issue should hopefully now be fixed in the next release, following which, lexicon can be packaged in the same way as other python projects.

See #787 for further details.