Cannot install pyodbc with the installer.no-binary option in macOS Ventura

[capitantoto]

• Poetry version: 1.7.1
• Python version: 3.11.4
• OS version and name: macOS Ventura 13.4.1
• pyproject.toml: https://gist.github.com/capitantoto/84dfe7a5ad98d9534d0e741abe30167c

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below. (but it doesn't truly add anything new).

Issue

Description

Hi there! I'm very sorry if there's any missing information here, it's the very first time in several years as a developer in which a previous issue has not been enough to get me through a bug/wrinkle in the matrix.

In brief: I need to installdbt-sqlserver for an ETL project (involving DBT and whatnot), which in turn requires pyodbc, version <=4.0.39 (when the current one is >=5.0.0. If I do just that, and try to poetry install a project with dbt-sqlserver="^1.4.3" as a sole dependency, poetry gets it just right, no issues involved. Now, In macOS, apparently pyodbc has to be installed with the no-binary option in order to avoid namespace conflicts down the road (link) . That made me add the install.no-binary = ["pyodbc"] option to poetry.toml, but when I do so, I get a ChefBuildError from poetry when it gets to pyodbc. This was not a problem with PyODBC >=5, but the dbt-sqlserver package is not compatible with that version 😢

Expected behavior

I should be able to install the pyodbc==4.0.39 drivers without issues, even when the no-binary option is passed to the installer.

pip version

23.3.1 (almost sure)

Python version

3.11.4

OS

macOS Ventura 13.4.1

How to Reproduce

Create a folder named testit, cd inside it and

1. copy the pyproject.toml and poetry.toml files provided below, and
2. create second folder named testit.
3. run poetry install.

pyproject.toml

[tool.poetry]
name = "testit"
version = "0.1.0"
description = ""
authors = ["Gonzalo Barrera Borla <gonzalobb@gmail.com>"]

[tool.poetry.dependencies]
python = "^3.11"
dbt-sqlserver = "^1.4.3"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

poetry.toml

[installer]
no-binary = ["pyodbc"]

Output

gonzalo@ohana testit % poetry install -vvv         
Loading configuration file /Users/gonzalo/Library/Application Support/pypoetry/config.toml
Loading configuration file /Users/gonzalo/Git/testit/poetry.toml
Using virtualenv: /Users/gonzalo/Library/Caches/pypoetry/virtualenvs/testit-g_9hSRQf-py3.11
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 2 installs, 0 updates, 0 removals, 58 skipped

  • Installing pyodbc (4.0.39): Pending...
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for pypi.org
[urllib3.connectionpool] Starting new HTTPS connection (1): pypi.org:443
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/pyodbc/4.0.39/json HTTP/1.1" 304 0
[filelock] Attempting to acquire lock 4377592208 on /Users/gonzalo/Library/Caches/pypoetry/cache/repositories/PyPI/_http/b/f/1/1/c/bf11c6d6434fe23aa0235deb8d0196871e31d5564261f3393b644ab7.lock
[filelock] Lock 4377592208 acquired on /Users/gonzalo/Library/Caches/pypoetry/cache/repositories/PyPI/_http/b/f/1/1/c/bf11c6d6434fe23aa0235deb8d0196871e31d5564261f3393b644ab7.lock
[filelock] Attempting to release lock 4377592208 on /Users/gonzalo/Library/Caches/pypoetry/cache/repositories/PyPI/_http/b/f/1/1/c/bf11c6d6434fe23aa0235deb8d0196871e31d5564261f3393b644ab7.lock
[filelock] Lock 4377592208 released on /Users/gonzalo/Library/Caches/pypoetry/cache/repositories/PyPI/_http/b/f/1/1/c/bf11c6d6434fe23aa0235deb8d0196871e31d5564261f3393b644ab7.lock
Skipping wheel for pyodbc-4.0.39-cp310-cp310-macosx_10_9_x86_64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp310-cp310-macosx_11_0_arm64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp310-cp310-win32.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp310-cp310-win_amd64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp311-cp311-macosx_10_9_x86_64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp311-cp311-macosx_11_0_arm64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp311-cp311-win32.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp311-cp311-win_amd64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp36-cp36m-macosx_10_9_x86_64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp36-cp36m-win32.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp36-cp36m-win_amd64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp37-cp37m-macosx_10_9_x86_64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp37-cp37m-win32.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp37-cp37m-win_amd64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp38-cp38-macosx_10_9_x86_64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp38-cp38-macosx_11_0_arm64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp38-cp38-win32.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp38-cp38-win_amd64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp39-cp39-macosx_10_9_x86_64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp39-cp39-macosx_11_0_arm64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp39-cp39-win32.whl as requested in no binary policy for package (pyodbc)
Skipping wheel for pyodbc-4.0.39-cp39-cp39-win_amd64.whl as requested in no binary policy for package (pyodbc)
  • Installing pyodbc (4.0.39): Preparing...
  • Installing pyodbc (4.0.39): Failed

  Stack trace:

  8  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/installation/executor.py:269 in _execute_operation
      267│ 
      268│             try:
    → 269│                 result = self._do_execute_operation(operation)
      270│             except EnvCommandError as e:
      271│                 if e.e.returncode == -2:

  7  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/installation/executor.py:379 in _do_execute_operation
      377│             return 0
      378│ 
    → 379│         result: int = getattr(self, f"_execute_{method}")(operation)
      380│ 
      381│         if result != 0:

  6  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/installation/executor.py:504 in _execute_install
      502│ 
      503│     def _execute_install(self, operation: Install | Update) -> int:
    → 504│         status_code = self._install(operation)
      505│ 
      506│         self._save_url_reference(operation)

  5  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/installation/executor.py:542 in _install
      540│             archive = self._download_link(operation, Link(package.source_url))
      541│         else:
    → 542│             archive = self._download(operation)
      543│ 
      544│         operation_message = self.get_operation_message(operation)

  4  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/installation/executor.py:746 in _download
      744│             self._yanked_warnings.append(message)
      745│ 
    → 746│         return self._download_link(operation, link)
      747│ 
      748│     def _download_link(self, operation: Install | Update, link: Link) -> Path:

  3  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/installation/executor.py:781 in _download_link
      779│             self._write(operation, message)
      780│ 
    → 781│             archive = self._chef.prepare(archive, output_dir=original_archive.parent)
      782│ 
      783│         # Use the original archive to provide the correct hash.

  2  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/installation/chef.py:123 in prepare
      121│             return self._prepare(archive, destination=destination, editable=editable)
      122│ 
    → 123│         return self._prepare_sdist(archive, destination=output_dir)
      124│ 
      125│     def _prepare(

  1  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/installation/chef.py:194 in _prepare_sdist
      192│             destination.mkdir(parents=True, exist_ok=True)
      193│ 
    → 194│             return self._prepare(
      195│                 sdist_dir,
      196│                 destination,

  ChefBuildError

  Backend subprocess exited when trying to invoke get_requires_for_build_wheel
  
  Traceback (most recent call last):
    File "/Users/gonzalo/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/Users/gonzalo/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/gonzalo/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
             ^^^^^^^^^^^^^^^^^^^^^
    File "/var/folders/4h/7ndr70pd3j178zcftn42zhsw0000gn/T/tmprcggl1pi/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/var/folders/4h/7ndr70pd3j178zcftn42zhsw0000gn/T/tmprcggl1pi/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
      self.run_setup()
    File "/var/folders/4h/7ndr70pd3j178zcftn42zhsw0000gn/T/tmprcggl1pi/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 371, in <module>
    File "<string>", line 77, in main
    File "<string>", line 205, in get_compiler_settings
  ValueError: not enough values to unpack (expected 2, got 0)
  

  at ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/installation/chef.py:164 in _prepare
      160│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│ 
      163│             if error is not None:
    → 164│                 raise error from None
      165│ 
      166│             return path
      167│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with pyodbc (4.0.39) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "pyodbc (==4.0.39)"'.

Running the exact command poetry suggests I try out, pip wheel --no-cache-dir --use-pep517 "pyodbc (==4.0.39)" runs perfectly fine, so I'm at a loss on how to continue. For reference, my current Poetry version is 1.7.1 and this is the minimal project configuration I've come up with to reproduce the error, using macOS Ventura 13.4.1:

[dimbleby]

this is a problem with pyodbc not poetry, you should report it at that repository

[Secrus]

As @dimbleby said, it's a problem with pyodbc (probably some quirks around building a wheel from their source code). Your pip command lacks --no-binary so my guess is it works because it uses a pre-built wheel.

[capitantoto]

Hi @Secrus @dimbleby , thanks for your answers. Could you help me with a bit more context? I can actually build the pyodbc wheel with the --no-binary option just right, and then install the package with the wheel without issue:

gonzalo@ohana % python -m venv tryagain
gonzalo@ohana % source tryagain/bin/activate
(tryagain) gonzalo@ohana % pip install --upgrade pip
Requirement already satisfied: pip in ./tryagain/lib/python3.11/site-packages (23.1.2)
Collecting pip
  Using cached pip-23.3.2-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.1.2
    Uninstalling pip-23.1.2:
      Successfully uninstalled pip-23.1.2
Successfully installed pip-23.3.2
(tryagain) gonzalo@ohana % pip wheel --no-binary=pyodbc --no-cache-dir --use-pep517 "pyodbc (==4.0.39)"
Collecting pyodbc==4.0.39
  Downloading pyodbc-4.0.39.tar.gz (282 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 282.4/282.4 kB 4.6 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: pyodbc
  Building wheel for pyodbc (pyproject.toml) ... done
  Created wheel for pyodbc: filename=pyodbc-4.0.39-cp311-cp311-macosx_11_0_arm64.whl size=68822 sha256=d4ff0f713b93113fd45a85fc976620afd5d647e149b874199ec673cb196f2f64
  Stored in directory: /private/var/folders/4h/7ndr70pd3j178zcftn42zhsw0000gn/T/pip-ephem-wheel-cache-w087481y/wheels/02/e3/a6/4ca8226ab78efb3fe31e6d540383017f29ea95e5adf5968aee
Successfully built pyodbc
(tryagain) gonzalo@ohana % pip install pyodbc-4.0.39-cp311-cp311-macosx_11_0_arm64.whl 
Processing ./pyodbc-4.0.39-cp311-cp311-macosx_11_0_arm64.whl
Installing collected packages: pyodbc
Successfully installed pyodbc-4.0.39

I only seem to be getting this error when installing the package through poetry, and using the installer.no-binary = ["pyodbc"] option at the same time, and that's why I'm posting it here.

[dimbleby]

not only is this not a poetry issue, it is already tracked at pyodbc - you could have looked there in the first place!

mkleehammer/pyodbc#1213

[capitantoto]

Hi @dimbleby , and thanks again for the patience. I see where I got confused now: there were two separate issues at play:

1. wheels for macOS Apple Silicon systems are not working for pyodbc: Can macOS Apple Silicon binary wheels be provided? mkleehammer/pyodbc#1263 (reply in thread)
2. if odbc_config is not present, pyodbc's setup.y fails, which you mentioned: setup.py fails if odbc_config can't be found mkleehammer/pyodbc#1213

I had unixodbc correctly installed when I started out, but in the creation and removal of several virtual environments when testing this out, I messed up my shell's PATH or something of the kind, and odbc_config couldn't be resolved, causing issue (2). Due to a combination of my limited coding abilities and a diagonal read of the error messages, I conflated the two 🙈 .

In any case, thank you for pointing me in the correct direction!

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.