Google-re2 installation fails

[byunks]

Apache Airflow version

2.6.3

What happened

Airflow 2.6.3 Installation Environment
mac M1 Ventura 13.5
Python : 3.7.2
conda(miniconda3) : 4.12.0

Error log when installed in the above environment

Collecting google-re2==1.0
  Using cached google-re2-1.0.tar.gz (9.8 kB)
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: google-re2
  Building wheel for google-re2 (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [16 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.7-x86_64-cpython-37
      copying re2.py -> build/lib.macosx-10.7-x86_64-cpython-37
      running build_ext
      building '_re2' extension
      creating build/temp.macosx-10.7-x86_64-cpython-37
      gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/user/miniconda3/envs/airflow.kr.c3s-pan/include -arch x86_64 -I/Users/user/miniconda3/envs/airflow.kr.c3s-pan/include -arch x86_64 -I/opt/homebrew/opt/krb5/include -I/Users/user/miniconda3/envs/airflow.kr.c3s-pan/include/python3.7m -c _re2.cc -o build/temp.macosx-10.7-x86_64-cpython-37/_re2.o -fvisibility=hidden
      clang: warning: include path for libstdc++ headers not found; pass '-stdlib=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
      _re2.cc:5:10: fatal error: 'memory' file not found
      #include <memory>
               ^~~~~~~~
      1 error generated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for google-re2
  Running setup.py clean for google-re2
Failed to build google-re2
Installing collected packages: google-re2
  Running setup.py install for google-re2 ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for google-re2 did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      running install
      /Users/user/miniconda3/envs/airflow.kr.c3s-pan/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        setuptools.SetuptoolsDeprecationWarning,
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.7-x86_64-cpython-37
      copying re2.py -> build/lib.macosx-10.7-x86_64-cpython-37
      running build_ext
      building '_re2' extension
      creating build/temp.macosx-10.7-x86_64-cpython-37
      gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/user/miniconda3/envs/airflow.kr.c3s-pan/include -arch x86_64 -I/Users/user/miniconda3/envs/airflow.kr.c3s-pan/include -arch x86_64 -I/opt/homebrew/opt/krb5/include -I/Users/user/miniconda3/envs/airflow.kr.c3s-pan/include/python3.7m -c _re2.cc -o build/temp.macosx-10.7-x86_64-cpython-37/_re2.o -fvisibility=hidden
      clang: warning: include path for libstdc++ headers not found; pass '-stdlib=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
      _re2.cc:5:10: fatal error: 'memory' file not found
      #include <memory>
               ^~~~~~~~
      1 error generated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> google-re2

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

What you think should happen instead

No response

How to reproduce

Installation will fail.

Operating System

mac M1 Ventura 13.5

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[byunks]

Python 3.8.17

Building wheels for collected packages: google-re2
  Building wheel for google-re2 (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [15 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.9-x86_64-cpython-38
      copying re2.py -> build/lib.macosx-10.9-x86_64-cpython-38
      running build_ext
      building '_re2' extension
      creating build/temp.macosx-10.9-x86_64-cpython-38
      gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/user/miniconda3/envs/airflow.kr.c3s-pan/include -arch x86_64 -I/Users/user/miniconda3/envs/airflow.kr.c3s-pan/include -arch x86_64 -I/opt/homebrew/opt/krb5/include -I/Users/user/miniconda3/envs/airflow.kr.c3s-pan/include/python3.8 -c _re2.cc -o build/temp.macosx-10.9-x86_64-cpython-38/_re2.o -fvisibility=hidden
      _re2.cc:11:10: fatal error: 'pybind11/pybind11.h' file not found
      #include <pybind11/pybind11.h>
               ^~~~~~~~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for google-re2
  Running setup.py clean for google-re2
Failed to build google-re2
ERROR: Could not build wheels for google-re2, which is required to install pyproject.toml-based projects

Python 3.9.17

Building wheels for collected packages: google-re2, pendulum
  Building wheel for google-re2 (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [15 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.9-x86_64-cpython-39
      copying re2.py -> build/lib.macosx-10.9-x86_64-cpython-39
      running build_ext
      building '_re2' extension
      creating build/temp.macosx-10.9-x86_64-cpython-39
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/user/miniconda3/envs/airflow.kr.c3s-pan/include -arch x86_64 -I/Users/user/miniconda3/envs/airflow.kr.c3s-pan/include -fPIC -O2 -isystem /Users/user/miniconda3/envs/airflow.kr.c3s-pan/include -arch x86_64 -I/opt/homebrew/opt/krb5/include -I/Users/user/miniconda3/envs/airflow.kr.c3s-pan/include/python3.9 -c _re2.cc -o build/temp.macosx-10.9-x86_64-cpython-39/_re2.o -fvisibility=hidden
      _re2.cc:11:10: fatal error: 'pybind11/pybind11.h' file not found
      #include <pybind11/pybind11.h>
               ^~~~~~~~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for google-re2
  Running setup.py clean for google-re2
  Building wheel for pendulum (pyproject.toml) ... done
  Created wheel for pendulum: filename=pendulum-2.1.2-cp39-cp39-macosx_13_0_x86_64.whl size=125122 sha256=fa5d6eb630f7f09458535781707f0d3909b1ec18287568a574fedd566be9b1ea
  Stored in directory: /Users/user/Library/Caches/pip/wheels/c9/c7/29/3cf0033406b18958d1a7b107c38be2405ef9c081b64a264fd5
Successfully built pendulum
Failed to build google-re2
ERROR: Could not build wheels for google-re2, which is required to install pyproject.toml-based projects

[xylar]

@byunks, could you follow up on conda-forge/airflow-feedstock#114?

Specifically, can you provide more info on the conda commands you're using to create a new environment and install airflow, and about your system (conda info)?

[shailshouryya]

The original error you ran into seems different, but your most recent error seems similar to google/re2#437, specifically google/re2#437 (comment) and google/re2#437 (comment).

It's not clear why this started failing now, since the google-re2 has the following in the setup.py file:

Known issues with regard to building the C++ extension:

  * Building requires RE2 to be installed on your system.
    On Debian, for example, install the libre2-dev package.
  * Building requires pybind11 to be installed on your system OR venv.
    On Debian, for example, install the pybind11-dev package.
    For a venv, install the pybind11 package from PyPI.
  * Building on macOS is known to work, but has been known to fail.
    For example, the system Python may not know which compiler flags
    to set when building bindings for software installed by Homebrew;
    see https://docs.brew.sh/Homebrew-and-Python#brewed-python-modules.
  * Building on Windows has not been tested yet and will probably fail.

(see https://github.com/google/re2/blob/e66463312e1d30d427bbde6c40e7fd627dcfb82e/python/setup.py#L30-L42)

There also don't seem to be many changes in release 2023-07-01 of google-re2 (release page), and release 2.11.1 of pybind11 (release page) also does not change much. Something else in the dependency chain may have changed, or perhaps there is some change in one of these recent releases that caused the break.

[pankajkoti]

This is likely due to broken Python installation. How are you installing Python /virtualenv? Are you using conda/anaconda or some other 3rd party mechanism?

Deleting & reinstalling conda/anaconda would help.

Other users have also faced this and doing the above helped them. This is likely not an Airflow issue. So converting it to a discussion.

[pankajkoti]

The Airflow OSS Slack community has a thread on this here: https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1689595095291319

[byunks]

@pankajkoti
Installing python 3.7, python 3.8, and python 3.9 in a python 3.7 base miniconda environment causes slightly different installation errors for each environment when installing airflow 2.6.3.

And the phenomenon is the same in the latest version of miniconda.

There was no problem with the installation of Airflow 2.6.2 but is there a problem with the package added to Airflow 2.6.3 in the conda environment?

[potiuk]

Yes. Your conda is broken and when it tries to install package that has no binary for 1.10.9 osx version, it will try to compile it and it will fail. If you don't reinstall your conda environment now, more and more packages you will have will get broken.

creating build/temp.macosx-10.9-x86_64-cpython-38

Your conda environment things that it is on Mac OS 10.9 and you are on 10.13.

So you have the choice of either fixing it now or having a flurry of packages broken soon. This is conda problem on MacOS not Airflow. Please reinstall it, this is the only way of not shooting yourself in your foot.

See discussion on conda forge here: conda-forge/conda-forge.github.io#1844 - if you have old compilers and old conda on your system it will try to compile for the oldest compiler you have in your system. Since November Conda maintainers discuss whether they should force this move (to 10.13) or not.

The best way you can help others, you can chime in inthe issue and describe the issue you have, because they claim it is not a widesprad one. It will be more and more widespread, the more new packages we are going to upgrade to. "Doing" something with google-re2 is not a solution, because in the next version there might be 5 more newer versions of the packages that will start compiling and failing, so the "google-re2" is the canary in the mine signalling that the big problem is coming unless conda fixes it.

Your solution is simple:

• remove and reinstall conda
• remove and reinstall Xcode

Try again

BTW. The only officially supported way of installing is pip anyway. https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html#using-pypi

[potiuk]

I also commented in the conda-forge/conda-forge.github.io#1844 from conda and I suggest you @byunks to continue the discussion there not here if you still do not want to follow our workaround with reinstalling conda.

I also tagged @Xyler whe has been once a maintainer of Airlfow packages for conda (I don't know if this is still the case), I hope he can shed some light on the issue - and whether Conda is going to do anything about it, because essentially this is a Conda problem not ours and we have a workaround in place.

[potiuk]

Thanks @xylar . Yes it's unfortunate that so many people depend on conda and that the packages there have to be maintained separately from pypi (and by some volunteers who have to manage it). I sympathise with you, I hope conda will get it more automated in the future - pypi is the canonical place where people upload packages and they are automatically published on conda rather than having to rely on volunteers doing so.

[potiuk]

Please ask users with conda-forge related issues to post here:
https://github.com/conda-forge/airflow-feedstock/issues
rather than (or at least in addition to) on your discussion pages.

Yes @byunks -> please create an issue there.

[xylar]

Yes it's unfortunate that so many people depend on conda and that the packages there have to be maintained separately from pypi (and by some volunteers who have to manage it). I sympathise with you, I hope conda will get it more automated in the future - pypi is the canonical place where people upload packages and they are automatically published on conda rather than having to rely on volunteers doing so.

I don't want to derail this thread but want to be clear that conda-forge also has some pretty major advantages over pypi when in comes to mutually compatible shared libraries. It seems like you're not a fan, but I wouldn't be a volunteer there if I didn't think it had a lot to offer.

[potiuk]

I don't want to derail this thread but want to be clear that conda-forge also has some pretty major advantages over pypi when in comes to mutually compatible shared libraries. It seems like you're not a fan, but I wouldn't be a volunteer there if I didn't think it had a lot to offer.

It's not that I am not a fan, it's just not part of offiial Python Packaging group - which is the official Python group. We want to support one and only one way of producing and deploying our packages. PyPI is the choice and canonical solution. This is our official mechanism of installing airflow and with our 650+ dependencies, we have a sophsisticated mechanism to use pip resolver to bulld reporoducible installs with constraints, automated upgrades to latest (and more secure versions of all the packages that are consistent with each otehr) and pip is the only supported environemnt that works there.

There is simply no way we are going to manually replicate it with different packaging, resolving https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html and mbanually solve problems of people who use other mechanisms. this is far too much effort for no benefits, and directing our users who have problems to switch to pip is the best and fastest way to not having to spend hours and days on thinking what kind of environmental issue might be in play (not mentioning the fact that conda packaging is incompatible with wheels of PyPI and that we would have to vote on another set of upt to a 100 packages we relase bi-monthly. That is infeasible. so we won't even look at SECOND distribution mechanism.

I have nothing agains conda, it's just not the way vast majoirity of our users consume airflow.

[xylar]

Sounds good. I think we're on the same page and I appreciate your thoughts.

Whenever your users complain about the conda version of Airflow not working, it makes sense for your response to be that the pypi installation is the only one you yourselves support and that if users really want to use the conda-forge version, they need to get help at https://github.com/conda-forge/airflow-feedstock/issues, not here.

[byunks]

@potiuk
ok. There is no problem with airflow, and there is a problem in the conda environment of the mac pc during the package installed in the airflow installation.

So reinstall the conda, xcode, etc. and if you still have a problem, you should ask the conda, right?

[xylar]

Yes, from my perspective, that's right.

It seems like one possibility is a mix of conda packages and pip (pypi) packages or a mix of different conda channels. We should move that discussion to:
conda-forge/airflow-feedstock#114
or
conda-forge/google-re2-feedstock#6
depending on if we think Airflow is even involved in the problem.

[xylar]

By the way, I think if things are working properly with conda, XCode shouldn't figure into things at all because packages are never meant to be built from source.

[byunks]

@xylar
ok. Then we'll proceed with what you said and if there's still a problem, we'll talk about it at conda-forge/airflow-feedstock#114, conda-forge/google-re2-feedstock#6.

I tried to use airflow 2.6.3 but I couldn't install it right away, so I installed airflow 2.6.2 and I'm using it, so I'm not in a hurry

[potiuk]

I do not know conda that well but my educated guess of what is happening is as follows:

I think the problem happens when your conda is old and has been installed years ago when you were still on macos 10.9. then you likely still have 10.9 xcode libraries and tooling installed. Then it was likely carried across multiple upgrades (or at some point in time you installed 10.9 compatible xcode libraries and xcode tooling - that is another scenario when you could have conda using those as 'baseline'.

Then Conda tries to get the package for the minimum version of the osx libraries installed - and tries to get binary packages for 10.9 and since it does not find it, it ties to build it.

That's my guess and looking at the issue I linked above seems that conda has 10.9 as minimum version it will try even though 10.9 is end-of-life for years now. So fixing conda to not try to get 10.9 builds is the ultimate solution.

I believe reinstalling conda (and possibly xcode) is solving it differently - conda will not see the 10.9 xcode libraries and will try to get the build compatible with current (10.13) version of osx. But this is a workaround.

Suppport for 10.9 ENDED 6 (!!!!!) Years ago. Years not months(!)

OS X 10.9 (Mavericks)9 years ago (22 Oct 2013)Ended 6 years ago (01 Dec 2016)

I think the right solution is to raise the minimum version of platform conda attempts to get the packages for even if older xcode libs are installed. It makes completely no sense to attempt to build a software for os version that is not supported for 6 years already. But maybe it's just me.

In any way - this is Conda, not Airflow issue, whatever the underlying reason is for using 10.9 support under the hood..

[xylar]

@potiuk, I understand conda (and conda-forge specifically) a little differently than you. As I understand it, we just build one package for OSX 64 (we build a separate package for OSX ARM 64). For maximum compatibility (unnecessarily conservative, I fully agree), it is build using the OSX 10.9 SDK and with compatibility back to OSX 10.9. We can certainly discuss if that's necessary or not, but I don't think that relates to the problem that @byunks is having.

Conda-forge builds binaries so the fact that google-re2 is making any attempt whatsoever to build code suggests something has gone wrong with the installation process. Without knowing exactly where @byunks started from and what the installation procedure was, I don't feel in a good position to guess what that was. What I can say is that I was able to successfully install the binary google-re2 and the version of Airflow that uses it in CI on OSX in tests I ran before building the packages for the public so I suspect it's not purely a conda problem. The most likely things would seem to be mixing pypi and conda packages, which is not something that is generally recommended but that's really a hunch.

The version of Xcode that users have shouldn't affect conda. At least I've never heard of that in my many years of working with conda packages. We build the binaries ahead of time on CI so there will never be an attempt to build code when the packages are installed. At least I have never seen that happen before and there are many safeguards in place as I understand it to prevent that.

I'd be more than happy to continue the discussion over at the google-re2 or airflow feedstocks.

[potiuk]

Hello @byunks - can you please report the issue with all details here: conda-forge/google-re2-feedstock#6 - i got the advice from conda contributor. @isuruf in conda-forge/conda-forge.github.io#1844 (comment) , that this is the right place to report the issue.

[4sushi]

Hi,

We faced similar error during Airflow installation (with conda). Install google-re2 with export SYSTEM_VERSION_COMPAT=0 setting seems to fix the error.

Example:

$(VENV_ACTIVATE); SYSTEM_VERSION_COMPAT=0 python -m pip install -U -q google-re2
$(VENV_ACTIVATE); AIRFLOW_GPL_UNIDECODE=yes pip install -U -q  "apache-airflow[google]==${AIRFLOW_VERSION}" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

Related conda issue : conda-forge/google-re2-feedstock#6

[xylar]

It looks like you are installing google-re2 and apache-airflow from pypi (pip) and not conda. I think if you are using a conda environment, you need to be installing those packages from conda-forge instead. Mixing conda-forge packages with pypi packages can have weird consequences like those above.

[ivanleoncz]

The issue, still remains, at least in Ubuntu 20.04 - output of pip install apache-airflow, is quite big, so I just remarked the google-re2:

  ----------------------------------------
  ERROR: Failed building wheel for google-re2
  Running setup.py clean for google-re2
Failed to build unicodecsv methodtools google-re2
ERROR: connexion 2.14.2 has requirement werkzeug<2.3,>=1.0, but you'll have werkzeug 2.3.8 which is incompatible.
ERROR: opentelemetry-api 1.26.0 has requirement importlib-metadata<=8.0.0,>=6.0, but you'll have importlib-metadata 8.2.0 which is incompatible.
ERROR: flask-caching 2.3.0 has requirement cachelib<0.10.0,>=0.9.0, but you'll have cachelib 0.13.0 which is incompatible.
Installing collected packages: lazy-object-proxy, psutil, markupsafe, werkzeug, jinja2, click, zipp, importlib-metadata, itsdangerous, flask, greenlet, sqlalchemy, six, python-dateutil, pyjwt, packaging, marshmallow, marshmallow-oneofschema, termcolor, typing-extensions, pygments, mdurl, markdown-it-py, rich, rich-argparse, importlib-resources, setproctitle, uc-micro-py, linkify-it-py, pathspec, tenacity, rpds-py, attrs, referencing, jsonschema-specifications, pkgutil-resolve-name, jsonschema, PyYAML, clickclick, inflection, certifi, idna, urllib3, charset-normalizer, requests, connexion, apache-airflow-providers-common-io, text-unidecode, python-slugify, pycparser, cffi, cryptography, wtforms, flask-wtf, unicodecsv, python-nvd3, wirerope, methodtools, pytz, croniter, wrapt, deprecated, opentelemetry-api, flask-login, dnspython, email-validator, Babel, Flask-Babel, apispec, ordered-set, limits, Flask-Limiter, colorama, Flask-JWT-Extended, Flask-SQLAlchemy, sqlalchemy-utils, prison, marshmallow-sqlalchemy, flask-appbuilder, google-re2, jmespath, apache-airflow-providers-fab, cron-descriptor, blinker, mdit-py-plugins, Mako, alembic, protobuf, opentelemetry-proto, opentelemetry-semantic-conventions, opentelemetry-sdk, opentelemetry-exporter-otlp-proto-common, grpcio, googleapis-common-protos, opentelemetry-exporter-otlp-proto-grpc, opentelemetry-exporter-otlp-proto-http, opentelemetry-exporter-otlp, fsspec, universal-pathlib, apache-airflow-providers-smtp, more-itertools, sqlparse, apache-airflow-providers-common-sql, apache-airflow-providers-sqlite, rfc3339-validator, h11, httpcore, exceptiongroup, sniffio, anyio, httpx, configupdater, asgiref, requests-toolbelt, multidict, yarl, async-timeout, frozenlist, aiosignal, aiohappyeyeballs, aiohttp, apache-airflow-providers-http, lockfile, setuptools, docutils, python-daemon, time-machine, tzdata, backports.zoneinfo, pendulum, dill, gunicorn, cachelib, flask-session, argcomplete, tabulate, colorlog, flask-caching, apache-airflow-providers-imap, apache-airflow-providers-ftp, pluggy, sqlalchemy-jsonfield, apache-airflow
    Running setup.py install for unicodecsv ... done
    Running setup.py install for methodtools ... done
    Running setup.py install for google-re2 ... error
    ERROR: Command errored out with exit status 1:
     command: /tmp/test/venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-311m26qc/google-re2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-311m26qc/google-re2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-txvk8k0a/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/test/venv/include/site/python3.8/google-re2
         cwd: /tmp/pip-install-311m26qc/google-re2/
    Complete output (16 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/re2
    copying re2/__init__.py -> build/lib.linux-x86_64-3.8/re2
    running build_ext
    building '_re2' extension
    creating build/temp.linux-x86_64-3.8
    x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/tmp/test/venv/include -I/usr/include/python3.8 -c _re2.cc -o build/temp.linux-x86_64-3.8/_re2.o -fvisibility=hidden
    _re2.cc:15:10: fatal error: absl/strings/string_view.h: No such file or directory
       15 | #include "absl/strings/string_view.h"
          |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------

[potiuk]

Your system needs some system level dependencies. Unfortunatley - depending on the type of distro and it's variant, you might miss some depedencies or the other and your task is to figure out and install all missing one. It's absolutely impossible to tell all kind of prerequisites for all kinds of installations and distro types.

When you have general problem with installing dependencies of python that require compilation (whe no binaries exist for your system), you nee to make sure you have all prerequisites - look at the output and see what you need to fix and install to make compilation works. This is "standard" python binary dependencies issues - and for every dependency you have problem with, you need to look it up - serch for error - google possible missing solutions and install missing dependencies.

For example in the part of the output you deleted (looking at your originally pasted stack trace):

error: invalid command 'bdist_wheel'

Means that on you have no wheel installed - which you have to install.

The other things - you need to google. - for example tensorflow/tensorflow#22320 when I googled it suggest installing newer protobuf version.

My recommendation however is to use much newer version of Ubuntu (and not a stripped variant) and use Python installed from official python installation documentation as well. This should generally work

Generally speaking - we recommend Debian bookworm as base system and this is where we test everything - see https://airflow.apache.org/docs/docker-stack/index.html and our reference image.