Google-re2 installation failing on OSX

[xylar]

See conda-forge/airflow-feedstock#114

Building wheels for collected packages: google-re2
  Building wheel for google-re2 (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [15 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.9-x86_64-cpython-38
      copying re2.py -> build/lib.macosx-10.9-x86_64-cpython-38
      running build_ext
      building '_re2' extension
      creating build/temp.macosx-10.9-x86_64-cpython-38
      gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/user/miniconda3/envs/airflow.kr.c3s-pan/include -arch x86_64 -I/Users/user/miniconda3/envs/airflow.kr.c3s-pan/include -arch x86_64 -I/opt/homebrew/opt/krb5/include -I/Users/user/miniconda3/envs/airflow.kr.c3s-pan/include/python3.8 -c _re2.cc -o build/temp.macosx-10.9-x86_64-cpython-38/_re2.o -fvisibility=hidden
      _re2.cc:11:10: fatal error: 'pybind11/pybind11.h' file not found
      #include <pybind11/pybind11.h>
               ^~~~~~~~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for google-re2
  Running setup.py clean for google-re2
Failed to build google-re2
ERROR: Could not build wheels for google-re2, which is required to install pyproject.toml-based projects

See also
apache/airflow#32852
conda-forge/conda-forge.github.io#1844 (comment)

[xylar]

So far, I don't have enough information about the workflow that causes this issue to help debug it. Anyone facing this issue, please describe in detail the conda and/or pip commands you used and provide conda list and conda info output.

[isuruf]

I can reproduce this with the following conditions

• Intel OSX or Apple silicon with Rosetta2 (Apple silicon with native python does not have this issue)
• python 3.8, 3.9 or 3.10 with defaults (unrelated to conda-forge)
• python 3.8 with conda-forge

(python 3.7 is unsupported. Please upgrade. It's not maintained anymore.)

The issue is that google-re2 binary wheels are available only on 11.0+.
When the user is on macOS <11, this issue crops up. Also, when using python 3.8 with macOS >=11 (either Intel or Apple silicon with rosetta2), python binary report 10.16, therefore the binary wheel is not used anymore.

Installing google-re2 from source breaks because they don't have correct dependencies there. i.e. pybind11 requirement is missing in the source package.

Solutions (Either of the following five should work)

1. Tell the system to not use compat. i.e. export SYSTEM_VERSION_COMPAT=0
2. Use python 3.9+ with conda-forge or python 3.11+ with defaults.
3. Do pip install pybind11 and when installing airflow do pip install --no-build-isolation .
4. If you are on M1 mac, use a native python installation and don't use x64 environment with Rosetta2.
5. Ask google-re2 devs to add pybind11 to their setup-requires.

[xylar]

@4sushi, I may be misunderstanding something but this command:

python -m pip install -U -q google-re2

is installing the pypi version of google-re2, not the conda-forge version, which would be installed with:

conda install -y -c conda-forge google-re2

[xylar]

In general, mixing pieces from conda-forge and pieces from pypi like you're doing in your workflow is likely to lead to incompatibilities, like the missing pybind11 that you're seeing. PyPI and conda-forge don't necessarily have compatible dependencies.

[xylar]

@4sushi, thanks for the clarification.

We can keep discussing your issue here but it doesn't relate the the conda-forge package for google-re2 (nor, I think, does the rest of this issue) so it kind of feels like the discussion is happening in the wrong place. But I understand that there's not a natural home for this discussion.

The issue as far as I can tell is that the dependency pybind11 needs to be installed explicitly for your workflow to work with more recent releases of google-re2. I don't believe these changes are related to conda-forge. I think they are related to PyPi and whether a wheel can be found or whether the package is being built from source. Whatever changed for you, it relates somehow to not finding a compatible wheel anymore.

If you can find a relevant place for this discussion that isn't conda-forge, I would appreciate it since, again, I don't think it has anything to do with us. But if there isn't any such place, we can keep the conversation going here...

[pingsutw]

@xylar did you figure out how to fix it?

[xylar]

@pingsutw, this isn't really a conda-forge problem so there isn't anything I can fix. I believe the response above #6 (comment) is likely the most useful.