Lunr.js as a search for static websites?

[arafalov]

It would be super-amazing to have a Lunr.js run with statically generated websites. I can imagine it running effectively in two modes:

1. Server-side mode during static site generation - when it does indexing, similar to how SASS precompiler works. That would generate a compressed index.
2. Client side which loads the index and does the search.

At the moment, statically generated sites work without search or use Google embedded search. This would open up a lot more options for them, including in presentation and faceting, avoiding generic template elements, etc.

[olivernn]

I think @slashdotdash has already created a plugin for jekyll that uses lunr to provide full text search on jekyll sites - https://github.com/slashdotdash/jekyll-lunr-js-search.

Lunr already allows you to build an index server side and then load this serialised index in the browser to search against.

All that you'd need to do is to be able to convert whatever format the static content is written in (e.g. markdown) into a JSON format that can be loaded into lunr for indexing. You would also need to be able to associate a url with each entry, perhaps as the ref that lunr uses to identify indexed documents.

If you get something up and running let me know, I'm always interested to see where lunr is being used!

[slashdotdash]

@arafalov will happily accept a pull request to extend jekyll-lunr-js-search with the pre-built indexing support.

[arafalov]

I am missing some of the skills to do that at the moment and my current focus is on Solr. :-( But perhaps after that.

[olivernn]

@slashdotdash does it not do this already? I thought it did, sorry! Let me know if you need a hand with adding this feature, I'm not familiar with jekyll plugins in general but am interested in how to generate indexes from ruby for a project where I'm using lunr so perhaps there is some overlap.

There is a simple, server side, index buider being used for the example on http://lunrjs.com/example here that might be of some use - https://github.com/olivernn/lunr.js/blob/master/example/index_builder.js

[arafalov]

That example is still uncompressed though, right?

"body": "basically what I want to do is forward people to a ...."

I was thinking about:

'what': doc1, doc2, doc3
'want': doc3, doc55

Obviously a compressed and optimized version of that. Again, what Lucene does with inverted tree and what lunr.js must be doing after tokenization, right?

[olivernn]

http://lunrjs.com/example/example_index.json basically contains a serialised version of a lunr index, so it contains the inverted index as well as any other data structures that are required for lunr to be able to execute searches. If the actual file size becomes an issue then it can be compressed before being served, but this is outside of the scope of lunr I think.

[arafalov]

Oh, my mistake. I was looking at example_data.json instead of example_index.json. That's exactly what I was thinking about. Looks like the feature is implemented and maybe it just needs to be advertised more. Not sure if Jekyl integration uses the compressed index though.

[olivernn]

Yeah from @slashdotdash comments it appears not.

As for documentation, I definitely agree with you, I need to get around to writing some more guide-like documentation as well as the API docs.

[slashdotdash]

@olivernn was thinking of using node.js and some JavaScript much like your index_builder.js to convert the .json file.

[slashdotdash]

... or even via http://rubygems.org/gems/execjs

[arafalov]

I think Node.js or execjs is the best way to be DRY. And isn't SASS compilation already has a similar pipeline? This could be just 'index compilation' step in the assets pipeline.

[olivernn]

I think execjs is probably the right way to go.

Maybe there is room for a specific lunr-generate command line tool? Something like the following:

$ lunr-generate --fields title:10,body --ref id --pipeline stopWordFilter,stemmer input_data.json > serialised_index.json

This might be a little simpler than having to create a JavaScript file that sets up the index, reads in data and writes out data. It wouldn't be able to deal with any custom pipeline functions that people want to use, but for simple use cases it might be useful.

Not sure lunr-generate is the correct name for it either.

Anyway, just a thought for now…

[brockfanning]

For any Docpad users: https://github.com/brockfanning/docpad-plugin-lunr

[bludrop]

@pburtchaell if you haven't found it yet, there is a plugin being developed at https://github.com/assemble/assemble-contrib-lunr