Move prebuilt metadata out of Anserini into top-level castorini repo

[lintool]

Prebuilt index metadata currently live in src/main/resources/prebuilt-indexes. Bindings are created in Anserini, and then those symbols are exported into Pyserini. This results in an awkward flow in that new indexes needed to be added to Anserini first.

My proposal is to move all index metadata JSON into a top-level castorini repo called prebuilt-indexes. This would improve the above flow. This would allow Pyserini to directly import prebuilt index symbols.

The downside here is that Anserini fatjar distributions would be "unstable", since it would need to consult the live web to pull in prebuilt index definitions - since they may change after distribution time. But perhaps this isn't so bad since the download URL might go stale regardless.

Thoughts?

[sahel-sh]

+1 to direct import to pyserini
For fatjar stablity we can do what we do for topics and qrels, a hithub submodule for index bindings or including them in the existing one?

[lintool]

I was planning on writing another RFC proposing to move all the eval stuff out. So, in my concrete proposal, we would not have any submodules. Everything (prebuilt index metadata, topics, qrels, etc.) would be pulled in dynamically.

[lilyjge]

I strongly agree! I would like to take this opportunity to also include encoder metadata for prebuilt indexes so you can get the appropriate query encoder model directly. Also, perhaps it would make sense for prebuilt indexes metadata and topics and qrels to live in the same repo?
I'm mostly agnostic on submodule vs pulled in dynamically. However, sometimes I do enjoy the convenience of having topics and qrels where I can see them in the submodule instead of hunting them down in cache and being able to just scroll through them to find the exact name I'm looking for. On the flip side, this means I have topics and qrels both in the submodule and auto downloaded in my cache. Overall this could be solved with documentation of everything available. Just an idea, perhaps we can also take this chance to include corresponding topics/qrels names in the prebuilt index metadata?
Additionally, if everything is pulled in dynamically, I think it would make sense to cache the catalogue locally as a fallback so there's at least some kind of offline support. I think this would also be necessary in order to keep prebuilt indexes working offline (eg to identify whether an index name is part of prebuilt indexes, and if it is, the filename in order to find the local directory it would've been downloaded to)?