Implement vector search

[m-i-l]

This is spun off from #96 , to implement a simple vector search using Huggingface's sentence-transformers/all-MiniLM-L6-v2 model.

This would be accessed via a new link below the main search box, to the left of "Browse Sites", called e.g. "Chat". This would take you to a page with a larger box where you enter your search term, and are returned the closest matching chunks of text along with their URLs.

I've got a stand-alone (i.e. not integrated into the indexing process) vector search proof of concept running on local dev, and results are good, suggesting it might be viable from a performance, CPU and possibly even disk space perspective.

Integrating the proof of concept into the indexing process will be fairly involved though:

• It will need to be implemented along with Detect if page content has changed #94 for efficiency, i.e. so embeddings are not repeatedly generated for unchanged content.
• Given that all-MiniLM-L6-v2 only operates on input text less than 256 word pieces, a page will have to be "chunked" with each chunk a separate vector. I've found many giant (often auto-generated) pages that would have tens of thousands or even hundreds of thousands of chunks, and it wouldn't be viable (or even particularly useful) to vectorise all of them, so I'm going to need some sort of sensible limit, e.g. a max of 12 chunks/vectors per page.
• Solr allows only one vector to be stored per page, given DenseVectorField is not currently multivalued (there is a proposal to change this, but nothing confirmed). The best workaround I can think of is to add the vectors for each chunk as child documents, although this will require reviewing and potentially updating all the existing search queries to ensure child documents are not returned for those by mistake.

[m-i-l]

Implemented the creation and indexing of embeddings, using sentence-transformers/all-MiniLM-L6-v2, stored in a field called content_chunk_vector in child nodes called content_chunks. Don't have the interface for querying at the moment, but it'll take 4 weeks to reindex all pages to have embeddings.

Will need to keep a very close eye on memory, CPU and disk utilisation, given that much of the content will be stored in duplicate, and that embeddings are relatively resource intensive to compute.

For reference, on dev, my 23 page site takes 59s to index without embeddings, and 4m 39s to index with embeddings, i.e. indexing takes nearly 5 times longer. On prod, a full listing seems to index at around 4 pages per minute, although basic listings are much faster given the lower number of embeddings.

There are a number of important concessions to try and make the solution sustainable, notably:

• Embeddings are only generated if changes to the page content are detected, as per Detect if page content has changed #94. This does mean that a reindex of the site is no longer a full clean reindex, i.e. indexed content is potentially preserved between indexes, so manual intervention will be required if a clean reindex is required, e.g. because the embedding model changes.
• Basic listings are limited to up to 8 embeddings per page, free trial 12, and full listings 40. Given each embedding contains very very roughly 80-100 words, this will mean that only the start of many pages will be searchable via the vector search. This is in addition to the already-existing indexing page limit, i.e. 50 page max for basic listings. These values are configurable and can be changed later, but need to try to get a balance between useful and sustainable.

[m-i-l]

Vector search has now been implemented. There is a basic interface at https://searchmysite.net/search/chat/ which shows the top 4 content chunks and links to the pages containing them. There's an HTML comment containing the scores. I won't put a link to it in the application though because it hasn't been thoroughly tested, and I may wait until implementing #96 before making public.

[m-i-l]

Getting terrible results on prod, so something must be wrong - reopening for investigation.