Unable to file folder for phrase in wikidump

[tiwari93]

Hi, First of all many thanks for work.

I am trying to test this.
As per documentation I downloaded all 4 tar files (datasets, wikipediadump, pretrained models and phrase index). but while running getting the below mentioned error:

which seems to be finding some phrase folder in wikidump, which is not available at all.

Can u suggest the reason for same.

I have given correct path for all folders.

[jhyuklee]

Hi, could you check if the 'meta_compressed.pkl' is in the right place? It should be inside the 'dump' folder of the phrase index.

[tiwari93]

Thanks for reply but 'meta_compressed.pkl' is under save_dir folder while in error its searching the phrase folder in data_dir

[tiwari93]

Also just to add i am able to load the model, issue is while predicting using model.search

[jhyuklee]

That part of the code (requiring phrase folder) runs when meta_compressed.pkl does not exist or placed in a wrong place. Could you move it to the dump folder? SAVE_DIR/phrase-index/dump

[tiwari93]

Ok got it. Actually I just though to use 1048576_flat_OPQ96_small and downloaded only that phrase index.
Looks like I need to download densephrases-multi_wiki-20181220 (74GB) phrase index dump as meta_compressed.pkl is not in 1048576_flat_OPQ96_small .tar file.

Is it necessary to donload 74 GB file for testing. Do we have any other option to download this pkl file

[jhyuklee]

Oh yes. I'll upload the pkl file separately, soon. In the meantime you have to download the large index that contains the meta_compressed.pkl. Thanks for letting me know!

[tiwari93]

Thank you for your help.
I will do the same and update you once its downloaded.
I should have checked with you earlier as i was struggling with this from last few days. :)
This repo is awesome. I hope i will be able to run it.

[jhyuklee]

Here's the meta_compressed.pkl. I'll update the readme as well. Let me know if the problem is fixed!

[tiwari93]

sure, however i started downloading the bigger one. Will update you for same.

[jhyuklee]

Updated the README as well: https://github.com/princeton-nlp/DensePhrases#3-phrase-index

[tiwari93]

Its working now.
Thank you for your work.