A (toy) low-level document indexing and retrieval system

IndexerMcIndexFace is a tiny traditional document indexing and retrieval system that I wrote as an excuse to play with FSTs (using the BurntSushi/fst crate) and Rust's parallelization capabilities (using also the crossbeam crate for message passing)

Features:

• Fully written in Rust
• Uses FSTs for fast access to postings
• Allows fielded documents, and uses the BM25F retrieval model (note: I didn't verify its correctness)
• The indexing stage is paralellized with a threadpool by creating and merging independent indexes
  • (Note that this is a naive implementation, and although it's extremely fast it can be really memory hungry)
• The retrieval stage is parallelized with a threadpool, where in this case it runs a different search for every token

Warnings:

• This is a toy project (e.g: index files are not compressed, the parallelization techniques are naive and resource-hungry...) and the API is very basic.

Usage:

• Simply run cargo run --release. main.rs will create a dummy collection of 1000 files using the MitchellRhysHall/random_word crate, and then will index and perform a randomised moderately sized query.

Possible improvements:

• The use of FSTs opens up many possibilities, as regex-like searches can be easily performed.
• Better parallelization techniques: Right now, each thread will create its own in-memory index, which will be later joined and written to binary files. This means that the memory usage can be very high for bigger collections of documents.
• Better tokenizers.
• N-gram or similar, more elaborate, indexes.
• Alternative retrieval models, phrase queries, etc.