Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update setup.py to python<3.7 to fix "C extension not loaded, training/inferring will be slow. Install a C compiler and reinstall fse" warning #44

Closed
AlexMRuch opened this issue Apr 22, 2021 · 4 comments
Closed

Update setup.py to python<3.7 to fix "C extension not loaded, training/inferring will be slow. Install a C compiler and reinstall fse" warning #44

AlexMRuch opened this issue Apr 22, 2021 · 4 comments

Comments

@AlexMRuch
Copy link

This is related to #18, which was closed but is not solved. Myself and another user (@lucas-ubm) are experiencing this problem on macOS systems, so it is not limited to Windows. I have tried installing gensim through conda to no avail. Any tips would be greatly appreciated.

Error message:

/opt/anaconda3/envs/sbir_covid/lib/python3.8/site-packages/fse/models/base_s2v.py:114: UserWarning: C extension not loaded, training/inferring will be slow. Install a C compiler and reinstall fse.
  warnings.warn(

Here is my machine setup:
macOS: MacBook Pro (15-inch, 2019), Version 11.2.3 (20D91)
Processor: 2.3 GHz 8-Core Intel Core i9
Memory: 32 GB 2400 MHz DDR4

Here is my conda env setup:

# packages in environment at /opt/anaconda3/envs/sbir_covid:
#
# Name                    Version                   Build  Channel
affinegap                 1.11                     pypi_0    pypi
aiohttp                   3.7.4            py38h96a0964_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
appnope                   0.1.2            py38h50d1736_1    conda-forge
argon2-cffi               20.1.0           py38h5406a74_2    conda-forge
async-timeout             3.0.1                   py_1000    conda-forge
async_generator           1.10                       py_0    conda-forge
attrs                     20.3.0             pyhd3deb0d_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
black                     20.8b1                     py_1    conda-forge
bleach                    3.3.0              pyh44b312d_0    conda-forge
boto                      2.49.0                     py_0    conda-forge
boto3                     1.17.55            pyhd8ed1ab_0    conda-forge
botocore                  1.20.55            pyhd8ed1ab_0    conda-forge
brotlipy                  0.7.0           py38h5406a74_1001    conda-forge
btrees                    4.8.0                    pypi_0    pypi
bz2file                   0.98                       py_0    conda-forge
c-ares                    1.17.1               h0d85af4_1    conda-forge
ca-certificates           2020.12.5            h033912b_0    conda-forge
cachetools                4.2.1              pyhd8ed1ab_0    conda-forge
catalogue                 2.0.3            py38h50d1736_0    conda-forge
categorical-distance      1.9                      pypi_0    pypi
certifi                   2020.12.5        py38h50d1736_1    conda-forge
cffi                      1.14.5           py38ha97d567_0    conda-forge
chardet                   4.0.0            py38h50d1736_1    conda-forge
click                     7.1.2              pyh9f0ad1d_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
coverage                  5.5              py38h96a0964_0    conda-forge
cryptography              3.4.7            py38h1fa4640_0    conda-forge
cycler                    0.10.0                     py_2    conda-forge
cymem                     2.0.5            py38h91a8764_1    conda-forge
cython-blis               0.7.4            py38ha1b04c9_0    conda-forge
dataclasses               0.8                pyhc8e2a94_1    conda-forge
datetime-distance         0.1.3                    pypi_0    pypi
decorator                 5.0.7              pyhd8ed1ab_0    conda-forge
dedupe                    2.0.8                    pypi_0    pypi
dedupe-hcluster           0.3.8                    pypi_0    pypi
dedupe-variable-datetime  0.1.5                    pypi_0    pypi
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
doublemetaphone           0.1                      pypi_0    pypi
entrypoints               0.3             pyhd8ed1ab_1003    conda-forge
fastcluster               1.1.28                   pypi_0    pypi
filelock                  3.0.12             pyh9f0ad1d_0    conda-forge
flake8                    3.9.1              pyhd8ed1ab_0    conda-forge
freetype                  2.10.4               h4cff582_1    conda-forge
fse                       0.1.15                   pypi_0    pypi
future                    0.18.2           py38h50d1736_3    conda-forge
gensim                    3.8.3            py38ha048514_4    conda-forge
google-api-core           1.26.2             pyhd8ed1ab_0    conda-forge
google-auth               1.28.0             pyh44b312d_0    conda-forge
google-cloud-core         1.5.0              pyhd3deb0d_0    conda-forge
google-cloud-storage      1.35.1             pyh44b312d_0    conda-forge
google-crc32c             1.1.2            py38haea9d43_0    conda-forge
google-resumable-media    1.2.0              pyhd3deb0d_0    conda-forge
googleapis-common-protos  1.53.0           py38h50d1736_0    conda-forge
grpcio                    1.37.0           py38ha263829_0    conda-forge
haversine                 2.3.0                    pypi_0    pypi
highered                  0.2.1                    pypi_0    pypi
idna                      2.10               pyh9f0ad1d_0    conda-forge
importlib-metadata        4.0.1            py38h50d1736_0    conda-forge
iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
ipykernel                 5.5.3            py38h6c79ece_0    conda-forge
ipython                   7.22.0           py38h6c79ece_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jedi                      0.18.0           py38h50d1736_2    conda-forge
jinja2                    2.11.3             pyh44b312d_0    conda-forge
jmespath                  0.10.0             pyh9f0ad1d_0    conda-forge
joblib                    1.0.1              pyhd8ed1ab_0    conda-forge
jpeg                      9d                   hbcb3906_0    conda-forge
jsonschema                3.2.0              pyhd8ed1ab_3    conda-forge
jupyter_client            6.1.12             pyhd8ed1ab_0    conda-forge
jupyter_core              4.7.1            py38h50d1736_0    conda-forge
jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
kiwisolver                1.3.1            py38hd9c93a9_1    conda-forge
krb5                      1.17.2               h60d9502_0    conda-forge
langcodes                 3.1.0                    pypi_0    pypi
lcms2                     2.12                 h577c468_0    conda-forge
levenshtein-search        1.4.5                    pypi_0    pypi
libblas                   3.9.0                     8_mkl    conda-forge
libcblas                  3.9.0                     8_mkl    conda-forge
libcrc32c                 1.1.1                h1c7c35f_2    conda-forge
libcxx                    11.1.0               habf9029_0    conda-forge
libedit                   3.1.20191231         h0678c8f_2    conda-forge
libffi                    3.3                  h046ec9c_2    conda-forge
libgfortran               5.0.0           9_3_0_h6c81a4c_22    conda-forge
libgfortran5              9.3.0               h6c81a4c_22    conda-forge
liblapack                 3.9.0                     8_mkl    conda-forge
libpng                    1.6.37               h7cec526_2    conda-forge
libpq                     13.1                 h052a64a_2    conda-forge
libprotobuf               3.15.8               hcf210ce_0    conda-forge
libsodium                 1.0.18               hbcb3906_1    conda-forge
libtiff                   4.2.0                h355d032_0    conda-forge
libwebp-base              1.2.0                h0d85af4_2    conda-forge
llvm-openmp               11.1.0               hda6cdc1_1    conda-forge
lz4-c                     1.9.3                h046ec9c_0    conda-forge
markupsafe                1.1.1            py38h5406a74_3    conda-forge
matplotlib                3.4.1            py38h50d1736_0    conda-forge
matplotlib-base           3.4.1            py38h6152e83_0    conda-forge
mccabe                    0.6.1                      py_1    conda-forge
mistune                   0.8.4           py38h5406a74_1003    conda-forge
mkl                       2020.4             h08c4f10_301    conda-forge
more-itertools            8.7.0              pyhd8ed1ab_0    conda-forge
msgpack                   1.0.2                    pypi_0    pypi
multidict                 5.1.0            py38h5406a74_1    conda-forge
murmurhash                1.0.5            py38h91a8764_0    conda-forge
mypy                      0.812            py38h96a0964_2    conda-forge
mypy_extensions           0.4.3            py38h50d1736_3    conda-forge
nbclient                  0.5.3              pyhd8ed1ab_0    conda-forge
nbconvert                 6.0.7            py38h50d1736_3    conda-forge
nbformat                  5.1.3              pyhd8ed1ab_0    conda-forge
ncurses                   6.2                  h2e338ed_4    conda-forge
nest-asyncio              1.5.1              pyhd8ed1ab_0    conda-forge
ninja                     1.10.2               h9a9d8cb_0    conda-forge
nltk                      3.6.2              pyhd8ed1ab_0    conda-forge
notebook                  6.3.0              pyha770c72_1    conda-forge
numpy                     1.20.2           py38had91d27_0    conda-forge
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openjpeg                  2.4.0                h6cbf5cd_0    conda-forge
openssl                   1.1.1k               h0d85af4_0    conda-forge
packaging                 20.9               pyh44b312d_0    conda-forge
pandas                    1.2.4            py38h1f261ad_0    conda-forge
pandoc                    2.13                 h0d85af4_0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
parso                     0.8.2              pyhd8ed1ab_0    conda-forge
pathspec                  0.8.1              pyhd3deb0d_0    conda-forge
pathy                     0.4.0              pyhd8ed1ab_0    conda-forge
persistent                4.7.0                    pypi_0    pypi
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    8.1.2            py38h83525de_1    conda-forge
pip                       21.0.1             pyhd8ed1ab_0    conda-forge
pluggy                    0.13.1           py38h50d1736_4    conda-forge
preshed                   3.0.5            py38h91a8764_0    conda-forge
prometheus_client         0.10.1             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.18             pyha770c72_0    conda-forge
protobuf                  3.15.8           py38ha048514_0    conda-forge
psutil                    5.8.0            py38h96a0964_1    conda-forge
psycopg2                  2.8.6            py38hc775865_2    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
py                        1.10.0             pyhd3deb0d_0    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.7                      py_0    conda-forge
pycodestyle               2.7.0              pyhd8ed1ab_0    conda-forge
pycparser                 2.20               pyh9f0ad1d_2    conda-forge
pydantic                  1.7.3            py38h96a0964_1    conda-forge
pyflakes                  2.3.1              pyhd8ed1ab_0    conda-forge
pygments                  2.8.1              pyhd8ed1ab_0    conda-forge
pyhacrf-datamade          0.2.5                    pypi_0    pypi
pylbfgs                   0.2.0.13                 pypi_0    pypi
pyopenssl                 20.0.1             pyhd8ed1ab_0    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyrsistent                0.17.3           py38h5406a74_2    conda-forge
pysocks                   1.7.1            py38h50d1736_3    conda-forge
pytest                    6.2.3            py38h50d1736_0    conda-forge
pytest-cov                2.11.1             pyh44b312d_0    conda-forge
python                    3.8.8           h4e93d89_0_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python_abi                3.8                      1_cp38    conda-forge
pytorch                   1.8.0           cpu_py38h561cec5_1    conda-forge
pytz                      2021.1             pyhd8ed1ab_0    conda-forge
pyyaml                    5.4.1            py38h5406a74_0    conda-forge
pyzmq                     22.0.3           py38hd3b92b6_1    conda-forge
readline                  8.1                  h05e3726_0    conda-forge
regex                     2021.4.4         py38h96a0964_0    conda-forge
requests                  2.25.1             pyhd3deb0d_0    conda-forge
reverse_geocoder          1.5.1              pyhd8ed1ab_0    conda-forge
rlr                       2.4.5                    pypi_0    pypi
rsa                       4.7.2              pyh44b312d_0    conda-forge
s3transfer                0.4.1              pyhd8ed1ab_0    conda-forge
sacremoses                0.0.43             pyh9f0ad1d_0    conda-forge
scikit-learn              0.24.1           py38hfd19401_0    conda-forge
scipy                     1.6.2            py38h431c0a8_0    conda-forge
send2trash                1.5.0                      py_0    conda-forge
sentence-transformers     0.4.1              pyhd3deb0d_0    conda-forge
setuptools                49.6.0           py38h50d1736_3    conda-forge
shellingham               1.4.0              pyh44b312d_0    conda-forge
simplecosine              1.2                      pypi_0    pypi
six                       1.15.0             pyh9f0ad1d_0    conda-forge
sleef                     3.5.1                h35c211d_1    conda-forge
smart_open                2.2.1              pyh9f0ad1d_0    conda-forge
spacy                     3.0.5            py38hc7464f7_0    conda-forge
spacy-alignments          0.8.3                    pypi_0    pypi
spacy-legacy              3.0.3              pyhd8ed1ab_0    conda-forge
spacy-lookups-data        1.0.0                    pypi_0    pypi
spacy-transformers        1.0.2                    pypi_0    pypi
sqlite                    3.35.4               h44b9ce1_0    conda-forge
srsly                     2.4.1            py38ha048514_0    conda-forge
terminado                 0.9.4            py38h50d1736_0    conda-forge
testpath                  0.4.4                      py_0    conda-forge
thinc                     8.0.3            py38he35c9cc_0    conda-forge
threadpoolctl             2.1.0              pyh5ca1d4c_0    conda-forge
tk                        8.6.10               h0419947_1    conda-forge
tokenizers                0.10.1           py38heb1ef26_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tornado                   6.1              py38h5406a74_1    conda-forge
tqdm                      4.60.0             pyhd8ed1ab_0    conda-forge
traitlets                 5.0.5                      py_0    conda-forge
transformers              4.5.1              pyhd8ed1ab_0    conda-forge
typed-ast                 1.4.3            py38h96a0964_0    conda-forge
typer                     0.3.2              pyhd8ed1ab_0    conda-forge
typing-extensions         3.7.4.3                       0    conda-forge
typing_extensions         3.7.4.3                    py_0    conda-forge
urllib3                   1.26.4             pyhd8ed1ab_0    conda-forge
wasabi                    0.8.2              pyh44b312d_0    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
wordfreq                  2.5.0                    pypi_0    pypi
xz                        5.2.5                haf1e3a3_1    conda-forge
yaml                      0.2.5                haf1e3a3_0    conda-forge
yarl                      1.5.1            py38h4d0b108_0    conda-forge
zeromq                    4.3.4                h1c7c35f_0    conda-forge
zipp                      3.4.1              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11            h7795811_1010    conda-forge
zope-index                5.0.0                    pypi_0    pypi
zope-interface            5.4.0                    pypi_0    pypi
zstd                      1.4.9                h582d3a0_0    conda-forge
@AlexMRuch
Copy link
Author

AlexMRuch commented Apr 22, 2021 •
edited

It's also not specific to the version of python

name: sbir_covid
channels:
  - conda-forge
  - defaults
dependencies:
  - python
  - ipykernel
  - pip
  - pandas
  - numpy
  - conda-forge::gensim<4.0
  - pip:
    - fse

@AlexMRuch
Copy link
Author

AlexMRuch commented Apr 22, 2021 •
edited

I solved the issue.

You have to explicitly set python=3.6. That's so weird that the installer didn't pick up that in order to install fse python must be set to 3.6.

This is likely because in your setup.py script you have REQUIRES_PYTHON = ">=3.8" <-- this should be set to 3.6 for conda and pip to circumvent these issues.

@AlexMRuch AlexMRuch changed the title C extension not loaded, training/inferring will be slow. Install a C compiler and reinstall fse. Update setup.py to python<3.7 to fix "C extension not loaded, training/inferring will be slow. Install a C compiler and reinstall fse" warning Apr 22, 2021
@oborchers
Copy link
Owner

This issue should be fixed in version 0.1.17. Please reopen if necessary 👍

@bschulth
Copy link

bschulth commented Nov 4, 2023 •
edited

@oborchers I was interested in trying this package out, but I am on Python 3.11, and was thinking about forking or making a pull request.

Can you describe what the limitations are that currently require this to run on Python < 3.7? And if you think it would be feasible to get it up and running on Python 3.11, or would it be a waste of my time?

Or with all the recent interest in LLMs and embeddings, do you think there is a better alternative in the open source community? I am targetting a large number of PDFs and have had problems getting the embeddings I want using popular embedding models on huggingface, but your approach with SIF seemed like it might get me closer to what I am looking for and to more easily generate tons of embeddings at the sentence or paragraph level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bschulth @AlexMRuch @oborchers