Faiss import error when run in virtualenv by using own built Faiss-python #866

billyean · 2019-06-17T22:09:56Z

Summary

I have built faiss-core and faiss-python by myself. I installed python into my local virtual env and try to import faiss and I got an error, checked egg file, it does have _swigfaiss.so inside. I checked conda swigfaiss.py, it's still using old swig_import_helper, not sure if caused by this you remove it by using swig create python/swigfaiss.py as follows:

7f5b22b#diff-8cf6167d58ce775a08acafcfe6f40966

$ ls faiss-1.5.2-py3.6/faiss
__init__.py	__pycache__	_swigfaiss.so	swigfaiss.py

Platform

OS: centos 7

Faiss version: 1.5.2

Faiss compilation options:

 ./configure  --prefix=/usr --without-cuda --with-blas=/usr/lib64/libblas.so.3 --with-lapack=/usr/lib64/liblapack.so.3
make
sudo make install
make py
cd ~ && rm -rf env && python3 -m venv env
source env/bin/activate
cd ~/faiss && sudo make -C python install

Running on:

CPU
GPU

Interface:

C++
Python

Reproduction instructions

$ python
Python 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 09:07:38)  [GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import faiss
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6.egg/faiss/__init__.py", line 18, in <module>
  File "/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6.egg/faiss/swigfaiss.py", line 13, in <module>
ImportError: cannot import name '_swigfaiss'

The text was updated successfully, but these errors were encountered:

beauby · 2019-06-18T01:53:42Z

Could you paste the output of import faiss._swigfaiss?

billyean · 2019-06-18T05:15:51Z

>>> import faiss._swigfaiss
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6.egg/faiss/__init__.py", line 18, in <module>
  File "/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6.egg/faiss/swigfaiss.py", line 13, in <module>
ImportError: cannot import name '_swigfaiss'

beauby · 2019-06-18T11:20:37Z

And the output of PYTHONPATH=/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6.egg/faiss/ python -c "import _swigfaiss"?

billyean · 2019-06-18T22:25:22Z

I guess this might be related to egg file, when I do

$  PYTHONPATH=/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6.egg/faiss/ python -c "import _swigfaiss"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named '_swigfaiss'

after I unzip the egg file, I don't see error as below

$  PYTHONPATH=/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6/faiss/ python -c "import _swigfaiss"

HoweChen · 2019-06-22T02:47:17Z

Same thing happened to me.

xxllp · 2019-07-05T03:08:14Z

这个问题好麻烦

WIll-Xu35 · 2019-07-15T01:06:35Z

Same problem here.

beauby · 2019-07-15T01:21:57Z

@HoweChen @WIll-Xu35 Please paste your generated makefile.inc.

WIll-Xu35 · 2019-07-15T07:01:13Z

@beauby I met this problem when using pytorch docker environment, which has conda environment and it was really a mess. I gave up on that environment and installed faiss without docker or conda. The installation was successful.

XinliYu · 2019-07-31T01:58:36Z

Same thing here. Compiled from source with anaconda virtual env. My makefile.inc is as below.

Copyright (c) Facebook, Inc. and its affiliates.

This source code is licensed under the MIT license found in the

LICENSE file in the root directory of this source tree.

CXX = g++ -std=c++11
CXXCPP = g++ -std=c++11 -E
CPPFLAGS = -DFINTEGER=int -fopenmp -I/usr/local/cuda/include
CXXFLAGS = -fPIC -m64 -Wno-sign-compare -g -O3 -Wall -Wextra
CPUFLAGS = -mpopcnt -mavx2 -mf16c
LDFLAGS = -fopenmp -L/usr/local/cuda/lib64
LIBS = -lopenblas -lcudart -lcublas
PYTHONCFLAGS = -I/home/ec2-user/anaconda3/envs/pytorch_p36/include/python3.6m -I/home/ec2-user/anaconda3/envs/pytorch_p36/include/python3.6m -I/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/numpy/core/include
SWIGFLAGS = -DSWIGWORDSIZE64

NVCC = /usr/local/cuda/bin/nvcc
CUDA_ROOT = /usr/local/cuda
CUDA_ARCH = -gencode=arch=compute_35,code=compute_35
-gencode=arch=compute_52,code=compute_52
-gencode=arch=compute_60,code=compute_60
-gencode=arch=compute_61,code=compute_61
-gencode=arch=compute_70,code=compute_70
NVCCFLAGS = -I $(CUDA_ROOT)/targets/x86_64-linux/include/
-Xcompiler -fPIC
-Xcudafe --diag_suppress=unrecognized_attribute
$(CUDA_ARCH)
-lineinfo
-ccbin $(CXX) -DFAISS_USE_FLOAT16

OS = $(shell uname -s)

SHAREDEXT = so
SHAREDFLAGS = -shared

ifeq ($(OS),Darwin)
SHAREDEXT = dylib
SHAREDFLAGS = -dynamiclib -undefined dynamic_lookup
SWIGFLAGS =
endif

MKDIR_P = /bin/mkdir -p
PYTHON = python
SWIG = swig
AR ?= ar

prefix ?= /home/ec2-user/faiss
exec_prefix ?= ${prefix}
libdir = ${exec_prefix}/lib
includedir = ${prefix}/include

XinliYu · 2019-07-31T02:06:36Z

Found the solution for me! Just unzip the egg file!
unzip faiss-1.5.3-py3.6.egg

And then a new folder called faiss will be created in the site-package folder. Then import is successful!

mingmingfu · 2019-08-12T03:47:14Z

I guess this might be related to egg file, when I do

$  PYTHONPATH=/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6.egg/faiss/ python -c "import _swigfaiss"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named '_swigfaiss'

after I unzip the egg file, I don't see error as below

$  PYTHONPATH=/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6/faiss/ python -c "import _swigfaiss"

where can I get the egg file? Could you share me the link.

sara-eb · 2019-09-03T11:05:55Z

@beauby I am facing the same issue:

Python 3.5.2 (default, Nov 12 2018, 13:43:14) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import faiss
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/dist-packages/faiss-1.5.3-py3.5.egg/faiss/__init__.py", line 18, in <module>
  File "/usr/local/lib/python3.5/dist-packages/faiss-1.5.3-py3.5.egg/faiss/swigfaiss.py", line 13, in <module>
ImportError: cannot import name '_swigfaiss'

@XinliYu could you please explain more on what you did?

nlp-study · 2019-09-11T02:47:11Z

I guess this might be related to egg file, when I do
$  PYTHONPATH=/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6.egg/faiss/ python -c "import _swigfaiss"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named '_swigfaiss'
after I unzip the egg file, I don't see error as below
$  PYTHONPATH=/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6/faiss/ python -c "import _swigfaiss"
where can I get the egg file? Could you share me the link.

your python dist-packages, such as mine : /usr/local/lib/python3.5/dist-packages

javedqadruddin · 2019-09-25T19:26:46Z

exporting the path to _swigfaiss in the venv's activate script worked for me. Just add this to the end of your activate script:
export PYTHONPATH="${PYTHONPATH}:/path/to/your/env/lib/python3.6/site-packages/faiss/"

Then deactivate and activate again and it should work.

mdouze · 2019-12-02T14:03:14Z

No activity, closing.

deimsdeutsch · 2020-01-01T11:20:45Z

Solution:

unzip the .egg file

create the folder "faiss-1.6.1-py3.5.egg" or whatever your error while import python3 comes up.
mv faiss faiss-1.6.1-py3.5.egg

then check with import faiss

@mdouze
This is definately a bug the egg file is not extracting properly.

sngjuk · 2020-02-10T14:38:50Z

My case, the egg was already extracted. this one may resolve your error.

sudo apt-get install libopenblas-dev

RaulGuo · 2020-04-17T02:19:26Z

for me, it is numpy version problem. I have upgraded numpy version from 1.13.1 to 1.18.2 and it works. Be sure the old version numpy is totally removed, I have uninstalled 1.18.2 numpy and find that numpy is still available in ipython, then I found that 1.18.2 is not the only numpy version installed in my server.

dvlshah · 2020-04-29T06:46:06Z

Just install three things in the following order and your problem Is solved

pip3 install faiss
sudo apt-get install libopenblas-dev
sudo apt-get install libomp-dev

gshguru · 2020-05-17T06:53:43Z

If you have faiss-cpu. Just make sure that faiss is not installed. In my case, these 2 packages were conflicting.

I had installed faiss-cpu using: pip install faiss-cpu --no-cache. Then by mistake installed faiss too in the middle of development. I had to uninstall faiss to get it working

hrxx · 2021-06-29T13:37:48Z

thanks!

gauenk · 2021-11-14T16:33:49Z

The unzip strategy from @XinliYu and @deimsdeutsch work great. An alternative to changing the permissions of the egg file is to use python-pip instead:

cd <path_to_faiss>
cd ./build/faiss/python
python -m pip install .

rather than using the current install directions from step 3 (see below),

python setup.py install

billyean changed the title ~~Faiss import error in~~ Faiss import error when run in virtualenv by using own built Faiss-python Jun 17, 2019

beauby added the install label Jun 18, 2019

mdouze closed this as completed Dec 2, 2019

houssam7737 mentioned this issue Jan 30, 2020

Installation issue - KeyError 'flags' when import faiss #1098

Closed

4 tasks

nscozzaro mentioned this issue Apr 14, 2021

ModuleNotFoundError: No module named 'faiss.swigfaiss_avx2' kyamagu/faiss-wheels#39

Closed

gauenk mentioned this issue Nov 14, 2021

Changing default python install direction #2108

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Faiss import error when run in virtualenv by using own built Faiss-python #866

Faiss import error when run in virtualenv by using own built Faiss-python #866

billyean commented Jun 17, 2019 •

edited

Loading

beauby commented Jun 18, 2019

billyean commented Jun 18, 2019

beauby commented Jun 18, 2019

billyean commented Jun 18, 2019

HoweChen commented Jun 22, 2019

xxllp commented Jul 5, 2019

WIll-Xu35 commented Jul 15, 2019

beauby commented Jul 15, 2019

WIll-Xu35 commented Jul 15, 2019

XinliYu commented Jul 31, 2019

Copyright (c) Facebook, Inc. and its affiliates.

This source code is licensed under the MIT license found in the

LICENSE file in the root directory of this source tree.

XinliYu commented Jul 31, 2019

mingmingfu commented Aug 12, 2019

sara-eb commented Sep 3, 2019

nlp-study commented Sep 11, 2019

javedqadruddin commented Sep 25, 2019 •

edited

Loading

mdouze commented Dec 2, 2019

deimsdeutsch commented Jan 1, 2020

sngjuk commented Feb 10, 2020

RaulGuo commented Apr 17, 2020

dvlshah commented Apr 29, 2020

gshguru commented May 17, 2020 •

edited

Loading

hrxx commented Jun 29, 2021

gauenk commented Nov 14, 2021 •

edited

Loading

Faiss import error when run in virtualenv by using own built Faiss-python #866

Faiss import error when run in virtualenv by using own built Faiss-python #866

Comments

billyean commented Jun 17, 2019 • edited Loading

Summary

Platform

Reproduction instructions

beauby commented Jun 18, 2019

billyean commented Jun 18, 2019

beauby commented Jun 18, 2019

billyean commented Jun 18, 2019

HoweChen commented Jun 22, 2019

xxllp commented Jul 5, 2019

WIll-Xu35 commented Jul 15, 2019

beauby commented Jul 15, 2019

WIll-Xu35 commented Jul 15, 2019

XinliYu commented Jul 31, 2019

Copyright (c) Facebook, Inc. and its affiliates.

This source code is licensed under the MIT license found in the

LICENSE file in the root directory of this source tree.

XinliYu commented Jul 31, 2019

mingmingfu commented Aug 12, 2019

sara-eb commented Sep 3, 2019

nlp-study commented Sep 11, 2019

javedqadruddin commented Sep 25, 2019 • edited Loading

mdouze commented Dec 2, 2019

deimsdeutsch commented Jan 1, 2020

sngjuk commented Feb 10, 2020

RaulGuo commented Apr 17, 2020

dvlshah commented Apr 29, 2020

gshguru commented May 17, 2020 • edited Loading

hrxx commented Jun 29, 2021

gauenk commented Nov 14, 2021 • edited Loading

billyean commented Jun 17, 2019 •

edited

Loading

javedqadruddin commented Sep 25, 2019 •

edited

Loading

gshguru commented May 17, 2020 •

edited

Loading

gauenk commented Nov 14, 2021 •

edited

Loading