import cld3 -> not found after 'correct' pip install

[MariusMez]

Hi @bsolomon1124

First of all, thanks for your work on this python module :-)

I try to use it on my mac (10.15.2, python 3.7.1 (conda env) and libprotoc 3.11.2) according to the readme (brew install protobuf too), the command python -m pip install pycld3 works well (no errors)

But when in my code I import cld3 I have an error telling me : Library not loaded: /usr/local/opt/protobuf/lib/libprotobuf.21.dylib

After checking on my system, seems I have libprotobuf.22.dylib installed

Don't know what to do ?

[bsolomon1124]

@MariusMez I am actually having this same exact problem on a fresh test install ☹️ . Same case for me (Python 3.7.5, pycld3-0.16, Mac OS X 10.15.2, pip/venv, libprotoc 3.11.2). Investigating now, stay tuned...

[bsolomon1124]

Okay, it seems like this is related to the fact that the only wheel is for OS X 10.15 / CPython 3.7. I.e., installing on OS X with Python 3.7 will point pip/conda to the wheel (pycld3-0.16-cp37-cp37m-macosx_10_15_x86_64.whl), whereas installing with anything else (even just Python 3.8 on OS X 10.15) will download pycld3-0.16.tar.gz and build it locally:

$ python -m pip install --no-cache-dir pycld3
Collecting pycld3
  Downloading https://files.pythonhosted.org/packages/df/f9/8a3c0b13e94102aec5fdb7a0b9873dd093987739f83081248ccea02b4038/pycld3-0.16.tar.gz (721kB)
     |████████████████████████████████| 727kB 3.8MB/s 
Installing collected packages: pycld3
  Running setup.py install for pycld3 ... done
Successfully installed pycld3-0.16

$ python
Python 3.8.1 (default, Jan  3 2020, 20:52:38) 
[Clang 11.0.0 (clang-1100.0.33.12)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cld3
>>> cld3
<module 'cld3' from '.../python3.8/site-packages/cld3.cpython-38-darwin.so'>

...so, I can only suspect this might be due to OS X minor version upgrade. I'm going to try either re-building the wheel or not distributing it at all.

Ref: https://packaging.python.org/guides/distributing-packages-using-setuptools/#platform-wheels

[bsolomon1124]

Hi @MariusMez , this should be fixed, can you try python -m pip install --no-cache-dir pycld3 (or whatever the 'don't used a cached version' conda equivalent is)?

I uploaded a new version to PyPI which has fresh wheels built for Python 3.8 and uses source distribution for Python 3.7. Both of those are working for me and import okay.

Ref: https://pypi.org/project/pycld3/#files

[MariusMez]

Hi @bsolomon1124 thanks for the new wheels.
I have now a new error when building from source distribution (py 3.7)

Failed to build pycld3
Installing collected packages: pycld3
Running setup.py install for pycld3 ... error
ERROR: Command errored out with exit status 1:
command: /PATH/anaconda3/envs/PROJECT/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/5f/jjk15thj4pd17wq28v0s2zv80000gn/T/pip-install-v2oajar6/pycld3/setup.py'"'"'; file='"'"'/private/var/folders/5f/jjk15thj4pd17wq28v0s2zv80000gn/T/pip-install-v2oajar6/pycld3/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/5f/jjk15thj4pd17wq28v0s2zv80000gn/T/pip-record-e__u4tqk/install-record.txt --single-version-externally-managed --compile
cwd: /private/var/folders/5f/jjk15thj4pd17wq28v0s2zv80000gn/T/pip-install-v2oajar6/pycld3/
Complete output (18 lines):
running install
running build
Running protoc sentence.proto feature_extractor.proto task_spec.proto --cpp_out=/private/var/folders/5f/jjk15thj4pd17wq28v0s2zv80000gn/T/pip-install-v2oajar6/pycld3/src/cld_3/protos/
running build_ext
building 'cld3' extension
creating build
creating build/temp.macosx-10.7-x86_64-3.7
creating build/temp.macosx-10.7-x86_64-3.7/src
creating build/temp.macosx-10.7-x86_64-3.7/src/cld_3
creating build/temp.macosx-10.7-x86_64-3.7/src/cld_3/protos
creating build/temp.macosx-10.7-x86_64-3.7/src/script_span
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/PATH/anaconda3/envs/PROJECT/include -arch x86_64 -I/PATH/anaconda3/envs/PROJECT/include -arch x86_64 -I/usr/local/include/ -I/private/var/folders/5f/jjk15thj4pd17wq28v0s2zv80000gn/T/pip-install-v2oajar6/pycld3/src/ -I/private/var/folders/5f/jjk15thj4pd17wq28v0s2zv80000gn/T/pip-install-v2oajar6/pycld3/src/cld_3/protos/ -I/PATH/anaconda3/envs/PROJECT/include/python3.7m -c pycld3.cpp -o build/temp.macosx-10.7-x86_64-3.7/pycld3.o -std=c++11
warning: include path for stdlibc++ headers not found; pass '-stdlib=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
pycld3.cpp:665:10: fatal error: 'ios' file not found
#include "ios"
^~~~~
1 warning and 1 error generated.
error: command 'gcc' failed with exit status 1
----------------------------------------

Seems a bug related to Xcode version (I've the latest install, with xcode cli updated too)

[bsolomon1124]

This is a tough one for me to debug since weirdly I am not seeing this on my own OS X @MariusMez. The difficulties of C++ extensions ☹️ . I suspect, in setup.py, in addition to

extra_compile_args=["-std=c++11"],

we may want

extra_compile_args=["-std=c++11", "-stdlib=libc++"],

(Which on Mac actually gets passed to clang instead of gcc.)

If you are up for it, would you mind trying that out? You can download https://files.pythonhosted.org/packages/44/15/77e4a9b53b53a6705727420a044c653d27ab49d4b6cd2336f056f134493c/pycld3-0.17.tar.gz, unzip, and change line 78 in setup.py, then pip install /path/to/pycld3.

Thank you for bearing with me here.

[MariusMez]

Thank you for your help @bsolomon1124 :-)

I managed to install it correctly with your recommendation and by adding an extra parameter: extra_link_args (seeing that discussion: pandas-dev/pandas#23424)

# https://docs.python.org/3/distutils/setupscript.html#describing-extension-modules
ext = [
    Extension(
        "cld3",  # Name of the extension by which it can be imported
        sources=SOURCES,
        include_dirs=INCLUDES,
        libraries=LIBRARIES,
        language="c++",
        extra_compile_args=["-std=c++11", "-stdlib=libc++"],
        extra_link_args=['-stdlib=libc++']
    )
]

With that setup, install is a success!

[bsolomon1124]

Related: ebca462