Import failing on ubuntu

[keithkam-yk]

Hi team! very excited to try this out. I am getting a bit of a bizarre issue, appreciate any insights into the below:

I am simply trying running the examples/01-basic_indexing_and_search.ipynb notebook. This works perfectly fine on my M1 Macbook. However when I try to run this on a remote ubuntu machine with an NVIDIA GPU installed, it hangs on the first cell.

I initially thought perhaps its a networking issue and loading the pre-trained model is simply taking a long time. However the cell still hangs just on the import line. (I have left it running and after an hour it is still stuck).

• The notebook environment works fine for other imported packages, and is responsive (hello world)
• Setting logging level to DEBUG all I get are these (possibly unrelated) logs

TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Popen(['git', 'version'], cwd=<REMOVED>, stdin=None, shell=False, universal_newlines=False)

• This issue persists after creating a fresh poetry environment with only the following installed:

python = "^3.10"
RAGatouille = "^0.0.4b2"
ipykernel = "^6.29.0"

Any help would be greatly appreciated!

[bclavie]

That's another strange issue, and I have to admit it's unclear to me at this stage why there seems to be a very small number of situations where something hangs 🤔
Some things off the top of my head:

• Have you completely removed the possibility that it's a network issue? e.g. if you !wget a large file, does it load fine?
• When you print the dependencies installed in a fresh env, do you also have all the required dependencies installed? (e.g. colbert-ai, etc...)
• This might be related to the unclear CUDA issue reported in Indexing failing: subcommand issues #60 , not sure yet!

[keithkam-yk]

Thanks Benjamin - I've investigated a bit further and this looks like an problem with a dependency and not ragatouille.

• All of the dependencies of ragatouille show up as expected with poetry show
• Importing each of ragatouille's dependencies, they all work fine except for import colbert
• Doing the same with colbert, it hangs on import spacy
• Installing a fresh poetry environment with only spacy (+ ipykernel), import spacy works fine (??)
• Installing a fresh poetry environment with only colbert and the issue returns

Will keep this updated if I fix the issue but closing the issue for now

[okhat]

Great investigation! Btw there’s a chance spacy isn’t really essential for colbert, it may be only used in a utility outside of the actual colbert stuff

[bclavie]

@okhat good shout, it looks like spacy isn't actually used at all anymore, just imported once: https://github.com/stanford-futuredata/ColBERT/blob/bdafbe4ad6aa83bc3db1799ebcac5463d38e45e2/utility/utils/dpr.py#L7 (and not even in a ColBERT-needed file)