To preserve bandwith, the index is stored in a binary file, using BloomFilters, instead of using a JSON index as Lunr.JS does.
For full details about BloomySearch, please refer to this blog post.
I have a static weblog, generated thanks to [Blogit](https://github.com/phyks/blogit, caution this code is ugly) and, as I only want to have html files on my server, I needed to find a way to enable users to search my blog.
An index is generated by a Python script, upon generation of the pages, and is dynamically downloaded by the client when he wants to search for contents.
Index generation (
pybloom.py: Library to handle bloom filters in Python
stemmer.py: Implementation of Porter Stemming algorithm in Python, from Vivake Gupta.
Example html search form
js/bloom.js: main JS code
js/bloomfilters.js: JS library to use BloomFilters
samples/: samples for testing purpose (taken from my blog articles)
Data from the python script is just the array of bloomfilters bitarray written as a binary file (
data/search_index), which I open with JS. The list of articles is also written in JSON form in a specific file (
Here's the format of the output from the python script:
- [16 bits] : number of articles (== number of bitarrays)
- for each bitarray:
- [16 bits] : length of the bitarray
- […] : the bitarray itself
But I wasn't fully satisfied by the first one, and I found the second one too heavy and complicated for my purpose, so I ended up coding this.
This code is mainly a proof of concept. As such, it is not fully optimized (actually, I just tweaked until the resulted files and calculations could be considered "acceptable"). For those looking for more effective solutions, here are a few things I found while looking for information on the web:
- The stemming algorithm used may not be the most efficient one. People wanting to work with non-English languages or to optimize the overall computation of the index can easily move to a more effective algorithm. See Wikipedia and the stemming library in Python which has C wrappers for best performances.
TLDR; I don't give a damn to anything you can do using this code. It would just be nice to quote where the original code comes from. All the included libraries (pybloom and the stemming library) have their own license.
- "THE NO-ALCOHOL BEER-WARE LICENSE" (Revision 42):
- Phyks (email@example.com) wrote this file. As long as you retain this notice
- you can do whatever you want with this stuff (and you can also do whatever
- you want with this stuff without retaining it, but that's not cool...). If we
- meet some day, and you think this stuff is worth it, you can buy me a
beersoda in return.