.add_faiss_index and .add_elasticsearch_index returns ImportError at Google Colab

[MapleBloom]

Describe the bug

At Google Colab
!pip install faiss-cpu works
import faiss no error
but
embeddings_dataset.add_faiss_index(column='embeddings')
returns

[/usr/local/lib/python3.11/dist-packages/datasets/search.py](https://localhost:8080/#) in init(self, device, string_factory, metric_type, custom_index)
247 self.faiss_index = custom_index
248 if not _has_faiss:
--> 249 raise ImportError(
250 "You must install Faiss to use FaissIndex. To do so you can run conda install -c pytorch faiss-cpu or conda install -c pytorch faiss-gpu. "
251 "A community supported package is also available on pypi: pip install faiss-cpu or pip install faiss-gpu. "

because
_has_faiss = importlib.util.find_spec("faiss") is not None at the beginning of datasets/search.py returns False
when
the same code at colab notebook returns
ModuleSpec(name='faiss', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7b7851449f50>, origin='/usr/local/lib/python3.11/dist-packages/faiss/init.py', submodule_search_locations=['/usr/local/lib/python3.11/dist-packages/faiss'])

But

import datasets
datasets.search._has_faiss

at colab notebook also returns False

The same story with _has_elasticsearch

Steps to reproduce the bug

1. Follow https://huggingface.co/learn/nlp-course/chapter5/6?fw=pt at Google Colab
2. till embeddings_dataset.add_faiss_index(column='embeddings')
3. embeddings_dataset.add_elasticsearch_index(column='embeddings')
4. https://colab.research.google.com/drive/1h2cjuiClblqzbNQgrcoLYOC8zBqTLLcv#scrollTo=3ddzRp72auOF

Expected behavior

I've only started Tutorial and don't know exactly. But something tells me that embeddings_dataset.add_faiss_index(column='embeddings')
should work without Import Error

Environment info

Google Colab notebook with default config

[Akshay-Sisodia]

I can fix this.
It's mainly because faiss-gpu requires python<=3.10 but the default python version in colab is 3.11. We just have to downgrade the CPython version down to 3.10 and it should work fine.

[MapleBloom]

I think I just had no chance to meet with faiss-cpu.
It could be import problem?
_has_faiss gets its value at the beginning of datasets/search.
I tried to call object before import faiss, so _has_faiss took False. And never updated later.

[Akshay-Sisodia]

Yes you can't meet the requirements because faiss-cpu runs only on python3.10 and lower but the default version for colab is python3.11 which results in pip not being able to find wheels for faiss-cpu with python3.11.

…

On Mon, 17 Mar, 2025, 3:56 pm MapleBloom, ***@***.***> wrote: I think I just had no chance to meet with faiss-cpu. It could be import problem? _has_faiss gets its value at the beginning of datasets/search. I tried to call object before import faiss, so _has_faiss took False. And never updated later. — Reply to this email directly, view it on GitHub <#7456 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVUSZMBVD7LEDDUGALOTVN32U2PMBAVCNFSM6AAAAABZDBA426VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMRYHE3TKNRXGI> . You are receiving this because you commented.Message ID: ***@***.***> [image: MapleBloom]*MapleBloom* left a comment (huggingface/datasets#7456) <#7456 (comment)> I think I just had no chance to meet with faiss-cpu. It could be import problem? _has_faiss gets its value at the beginning of datasets/search. I tried to call object before import faiss, so _has_faiss took False. And never updated later. — Reply to this email directly, view it on GitHub <#7456 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVUSZMBVD7LEDDUGALOTVN32U2PMBAVCNFSM6AAAAABZDBA426VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMRYHE3TKNRXGI> . You are receiving this because you commented.Message ID: ***@***.***>

[MapleBloom]

you can't meet the requirements

It is not the case (or I didn't reach this point) because the same code in notebook
importlib.util.find_spec("faiss")
finds faiss. I've mention it.
I think the problem is in the very moment when _has_faiss takes its value and never try again.
(or it couldn't find the path that was easily found when started from my code)

[Akshay-Sisodia]

When you run the first cell containing pip install faiss-cpu does it install it?

…

On Mon, 17 Mar, 2025, 8:01 pm MapleBloom, ***@***.***> wrote: you can't meet the requirements It is not the case (or I didn't reach this point) because the same code in notebook importlib.util.find_spec("faiss") finds faiss. I've mention it. I think the problem is in the very moment when _has_faiss takes its value and never try again. (or it couldn't find the path that was easily found when started from my code) — Reply to this email directly, view it on GitHub <#7456 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVUSZMCCE6BPZCOVAWXKIY32U3MFVAVCNFSM6AAAAABZDBA426VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMRZG4ZTONBRGQ> . You are receiving this because you commented.Message ID: ***@***.***> [image: MapleBloom]*MapleBloom* left a comment (huggingface/datasets#7456) <#7456 (comment)> you can't meet the requirements It is not the case (or I didn't reach this point) because the same code in notebook importlib.util.find_spec("faiss") finds faiss. I've mention it. I think the problem is in the very moment when _has_faiss takes its value and never try again. (or it couldn't find the path that was easily found when started from my code) — Reply to this email directly, view it on GitHub <#7456 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVUSZMCCE6BPZCOVAWXKIY32U3MFVAVCNFSM6AAAAABZDBA426VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMRZG4ZTONBRGQ> . You are receiving this because you commented.Message ID: ***@***.***>

[MapleBloom]

When you run the first cell containing pip install faiss-cpu does it
install it?
…

Yes. It was installed succesfully.
Methods of datasets library that depends on _has_faiss constant didn't start to work.