Modifying num_clusters in index-vecs

[light42]

I tried to run index-vecs using custom wikidump, dataset and model, but got this error

Modifying num_clusters flags to 96 doesn't seem to help, the k in error message is still 256.

[light42]

And also, the dump/phrase was made from gen-vecs-parallel instead of gen-vecs. I don't understand how to make gen-vecs choose my custom wikidump as predict_file.

[jhyuklee]

Hi, how big is your custom corpus? Is it separated by multiple files? It seems like the size of the corpus is too small. num_clusters <= 96 should work for the error message.

[light42]

I'm using the latest Indonesian wikipedia dump(idwiki-latest-pages-articles.xml.bz2) from https://dumps.wikimedia.org/idwiki/latest/.
And then I preprocess it into WikiExtractor.
And then I used all of the json output(should be around 1145) to feed into gen_vecs_parallel.

The weird thing is I keep having the exact error message even if I change the num_clusters <= 96. The k in the error message keep showing value 256.

[jhyuklee]

Did you use the pretrained DensePhrases for indonesian? Note that DensePhrases was trained on English Wikipedia.

[light42]

Yes, I'm using pretrained indolem/indobert-base-uncased from huggingface.

After some digging, I have gut feeling that the fifth parameter from IndexIVFPQ constructor have something to do with the error. Since 256 is 2^8.

DensePhrases/build_phrase_index.py

Line 115 in 4f35efe

sub_index = faiss.IndexIVFPQ(quantizer, ds, num_clusters, code_size, 8, faiss.METRIC_INNER_PRODUCT)

Unfortunately this can't be changed at all. It must be set to 8 or else it will break.

^my guess is wrong, I ran it on cpu mode where I could changed the parameter, yet I still got the same error. :(

[jhyuklee]

You should train your pre-trained LMs on QA datasets for phrase retrieval. Is the model trained using our provided training script? (with QA datasets in indonesian?)

[light42]

Yes. In the end I leave fine_quant flag empty and now it worked. Is it okay if I leave it empty?

[jhyuklee]

It means that you are using the flat index, which is not very optimal when you try to use the entire Wikipedia dump. How does it work? How large is your phrase dump?

[light42]

It only generated 1145 files compared to wiki-20181220 which have 5621 files.

[light42]

I find that #15 have similar problem.
I realized making the corpus bigger is the only way to go, but I can't make it bigger unless I search from another sources.
Is there a script where I could preprocess non-wikipedia pages into dumps ready to be used for densephrases?

[light42]

Finally solved, after I changed doc_sample_ration and vec_sample_ratio to 1.0.