Problem with package: rapidfuzz - undefined symbol: __atomic_compare_exchange_8

[Jackenmen]

Package name

rapidfuzz

Package version

2.1.4 - 2.4.2

PyPI URL

https://pypi.org/project/rapidfuzz/

piwheels URL

https://www.piwheels.org/project/rapidfuzz/

Python version

• Python 3.5
• Python 3.7
• Python 3.9

I am the maintainer

• Yes

More information

On freshly flashed Raspberry Pi OS 10 (Legacy) and Raspberry Pi OS 11 armhf, this can be reproduced by running these commands:

sudo apt install python3-venv
python3 -m venv repro
. repro/bin/activate
python -m pip install -U rapidfuzz
python -c 'import rapidfuzz'

Which results in output:

pi@raspberrypi:~ $ python3 -m venv repro
pi@raspberrypi:~ $ . repro/bin/activate
(repro) pi@raspberrypi:~ $ python -m pip install -U rapidfuzz
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting rapidfuzz
  Downloading https://www.piwheels.org/simple/rapidfuzz/rapidfuzz-2.4.2-cp39-cp39-linux_armv7l.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 1.3 MB/s 
Collecting jarowinkler<2.0.0,>=1.2.0
  Downloading https://www.piwheels.org/simple/jarowinkler/jarowinkler-1.2.0-cp39-cp39-linux_armv7l.whl (92 kB)
     |████████████████████████████████| 92 kB 140 kB/s 
Installing collected packages: jarowinkler, rapidfuzz
Successfully installed jarowinkler-1.2.0 rapidfuzz-2.4.2
(repro) pi@raspberrypi:~ $ python -c 'import rapidfuzz'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/pi/repro/lib/python3.9/site-packages/rapidfuzz/__init__.py", line 8, in <module>
    from rapidfuzz import process, distance, fuzz, string_metric, utils
  File "/home/pi/repro/lib/python3.9/site-packages/rapidfuzz/process.py", line 10, in <module>
    cdist = _fallback_import("rapidfuzz.process_cdist", "cdist")
  File "/home/pi/repro/lib/python3.9/site-packages/rapidfuzz/utils.py", line 17, in _fallback_import
    mod = importlib.import_module(module + "_cpp")
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/pi/repro/lib/python3.9/site-packages/rapidfuzz/process_cdist_cpp.py", line 3, in <module>
    from rapidfuzz.process_cdist_cpp_impl import (
ImportError: /home/pi/repro/lib/python3.9/site-packages/rapidfuzz/process_cdist_cpp_impl.cpython-39-arm-linux-gnueabihf.so: undefined symbol: __atomic_compare_exchange_8
(repro) pi@raspberrypi:~ $

When not using the built wheels, this error does not occur:

pi@raspberrypi:~ $ python3 -m venv manual_build
@raspberrypi:~ $ . manual_build/bin/activate
(manual_build) pi@raspberrypi:~ $ python -m pip install -U rapidfuzz --no-binary :all:
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting rapidfuzz
  Downloading rapidfuzz-2.4.2.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 1.4 MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting jarowinkler<2.0.0,>=1.2.0
  Downloading jarowinkler-1.2.0.tar.gz (84 kB)
     |████████████████████████████████| 84 kB 385 kB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: rapidfuzz, jarowinkler
  Building wheel for rapidfuzz (PEP 517) ... done
  Created wheel for rapidfuzz: filename=rapidfuzz-2.4.2-py3-none-any.whl size=42562 sha256=984737dad7f822a72a3b602bfcbad2a51b4eddd611e595da60100c949aa54ef8
  Stored in directory: /home/pi/.cache/pip/wheels/af/99/46/b4f077eb37c0b9b373ef6a18e861eb7fad8def017b50f95f8d
  Building wheel for jarowinkler (PEP 517) ... done
  Created wheel for jarowinkler: filename=jarowinkler-1.2.0-py3-none-any.whl size=6729 sha256=70ed085038c7bde06812694e2d40c6545c7f5710ac14de6457dfcc2358a2a4c2
  Stored in directory: /home/pi/.cache/pip/wheels/e8/9a/b5/479eb2f7259ba5d0c17fe12140829c71ccef2b16c3ba77a537
Successfully built rapidfuzz jarowinkler
Installing collected packages: jarowinkler, rapidfuzz
Successfully installed jarowinkler-1.2.0 rapidfuzz-2.4.2
(manual_build) pi@raspberrypi:~ $ python -c 'import rapidfuzz'
(manual_build) pi@raspberrypi:~ $

I have also tried using --no-binary rapidfuzz instead of --no-binary :all: and in that case, there are also no errors.

At first, I thought that this is probably an issue with the package but since I tried installing it without using pre-built wheels and it worked fine, I figured that maybe it has something to do with how piwheels built the package.

[maxbachmann]

@jack1142 just to make sure does your manually build python package include the C++ version?

import os
os.environ["RAPIDFUZZ_IMPLEMENTATION"] = "cpp"
import rapidfuzz

does this work?

I think this might need a fix inside the rapidfuzz library similar to: https://github.com/arvidn/libtorrent/blob/RC_2_0/CMakeLists.txt#L649 which appears to help with a bug in the gcc compiler: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81358

[Jackenmen]

just to make sure does your manually build python package include the C++ version?

Huh, apparently not...

pi@raspberrypi:~ $ python3 -m venv manual_build
pi@raspberrypi:~ $ . manual_build/bin/activate
(manual_build) pi@raspberrypi:~ $ python -m pip install -U rapidfuzz --no-binary :all:
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting rapidfuzz
  Using cached rapidfuzz-2.4.2.tar.gz (1.1 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting jarowinkler<2.0.0,>=1.2.0
  Using cached jarowinkler-1.2.0.tar.gz (84 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: rapidfuzz, jarowinkler
  Building wheel for rapidfuzz (PEP 517) ... done
  Created wheel for rapidfuzz: filename=rapidfuzz-2.4.2-py3-none-any.whl size=42562 sha256=3b485c4bdd078cfd51a2495371daa7260217043026d717c7a750086a8ce6e57d
  Stored in directory: /home/pi/.cache/pip/wheels/af/99/46/b4f077eb37c0b9b373ef6a18e861eb7fad8def017b50f95f8d
  Building wheel for jarowinkler (PEP 517) ... done
  Created wheel for jarowinkler: filename=jarowinkler-1.2.0-py3-none-any.whl size=6729 sha256=32acf0183fd8c654d3b43a6b3e40798980dadfd07c3b1c7744e3283c52a012cc
  Stored in directory: /home/pi/.cache/pip/wheels/e8/9a/b5/479eb2f7259ba5d0c17fe12140829c71ccef2b16c3ba77a537
Successfully built rapidfuzz jarowinkler
Installing collected packages: jarowinkler, rapidfuzz
Successfully installed jarowinkler-1.2.0 rapidfuzz-2.4.2
(manual_build) pi@raspberrypi:~ $ cat repro.py
import os
os.environ["RAPIDFUZZ_IMPLEMENTATION"] = "cpp"
import rapidfuzz
(manual_build) pi@raspberrypi:~ $ python repro.py
Traceback (most recent call last):
  File "/home/pi/repro.py", line 3, in <module>
    import rapidfuzz
  File "/home/pi/manual_build/lib/python3.9/site-packages/rapidfuzz/__init__.py", line 8, in <module>
    from rapidfuzz import process, distance, fuzz, string_metric, utils
  File "/home/pi/manual_build/lib/python3.9/site-packages/rapidfuzz/process.py", line 4, in <module>
    from rapidfuzz.utils import _fallback_import
  File "/home/pi/manual_build/lib/python3.9/site-packages/rapidfuzz/utils.py", line 29, in <module>
    default_process = _fallback_import("rapidfuzz.utils", "default_process")
  File "/home/pi/manual_build/lib/python3.9/site-packages/rapidfuzz/utils.py", line 12, in _fallback_import
    mod = importlib.import_module(module + "_cpp")
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'rapidfuzz.utils_cpp'
(manual_build) pi@raspberrypi:~ $ 

[Jackenmen]

Huh, apparently not...

I honestly didn't expect that rapidfuzz would just continue on with pure Python build when the build fails since most packages don't do that 😄 I guess I should have expected something is wrong since I didn't even have to install python3-dev package :P
Anyway, now that I've installed python3-dev and cmake, I ran another install and that in fact caused me to run into the exact same error:

(manual_build) pi@raspberrypi:~ $ python repro.py
Traceback (most recent call last):
  File "/home/pi/repro.py", line 3, in <module>
    import rapidfuzz
  File "/home/pi/manual_build/lib/python3.9/site-packages/rapidfuzz/__init__.py", line 8, in <module>
    from rapidfuzz import process, distance, fuzz, string_metric, utils
  File "/home/pi/manual_build/lib/python3.9/site-packages/rapidfuzz/process.py", line 10, in <module>
    cdist = _fallback_import("rapidfuzz.process_cdist", "cdist")
  File "/home/pi/manual_build/lib/python3.9/site-packages/rapidfuzz/utils.py", line 12, in _fallback_import
    mod = importlib.import_module(module + "_cpp")
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/pi/manual_build/lib/python3.9/site-packages/rapidfuzz/process_cdist_cpp.py", line 3, in <module>
    from rapidfuzz.process_cdist_cpp_impl import (
ImportError: /home/pi/manual_build/lib/python3.9/site-packages/rapidfuzz/process_cdist_cpp_impl.cpython-39-arm-linux-gnueabihf.so: undefined symbol: __atomic_compare_exchange_8

Looks like this is indeed something that needs to be addressed by rapidfuzz.

[maxbachmann]

I honestly didn't expect that rapidfuzz would just continue on with pure Python build when the build fails since most packages don't do that

Yes this is mostly done to have the library working on all systems. E.g. https://github.com/jamesturk/jellyfish is another text matching library that does the same.

Looks like this is indeed something that needs to be addressed by rapidfuzz.

Could you try to build from the latest main version?

git clone --recursive https://github.com/maxbachmann/RapidFuzz.git
cd RapidFuzz
pip install .

I think this should fix the error.

[Jackenmen]

Could you try to build from the latest main version?

Works fine 👍

Here's the full build log:
build.log

[Jackenmen]

Just tried installing the released rapidfuzz 2.4.3 that's also already available on piwheels and it works fine. Looks like this can be closed, thanks for the fix!