Initial Whoosh search work

[kura]

Summary

Provide a short overview ...

Details

Context to describe changes if necessary

Checks

• In case of new feature, add short overview in docs/<corresponding file>
• Tested changes

[kura]

@Linbreux I have created this as a draft since I'm not done yet but this is my initial work on moving search over to Whoosh. I thought I'd create a draft PR so you could have a look to see if you're OK with the approach.

Currently it only indexes documents when the wiki is started and the search index is empty.

TODO:

• Run search.index when a file is created in the wiki
• Run search.update when a file is updated in the wiki - I ended up going with a simpler method of deleting the old version and indexing the new one which is a supported and seems to be how the update method works anyway. It has the advantage of making the code cleaner imo, no special cases in the save method in wiki.py and removes the update method from search.py entirely.
• Run search.delete when a file is deleted in the wiki
• Figure out how to run the 3 above calls when files are created/updated/deleted on the filesystem outside of wikmd
• Write search index to same parent directory as images i.e. /wiki
• • Hide search index directory from directory listings etc
• Add highlights to search results

I am really not sure how to force reindexing files that are created/updated/deleted directly on the filesystem. It seems to me like we'd need a thread running that is monitoring filesystem changes and triggering indexing based on that, and also trigger a reindex every time wikmd is started.

[Linbreux]

Perfect, I'll look into it when I have some free time and maybe add some commits myself. Appreciate the work you put into it!

[kura]

No problem, I was bored and wanted something to do. :D I'll make some more changes this evening or tomorrow to tie up loose ends.

[kura]

@Linbreux I am pretty happy with how this works now.

When the Flask app is started 2 things will happen;

1. A search index will be created, overwriting the existing index. This is done because if the Markdown files were modified on disk before the Flask app was started, without a reindex the file changes would never be picked up.
2. A watchdog thread is started that monitors the wiki directory for file changes

When creating, modifying or deleting a file with extension .md in the wiki directory via the Flask app or on the file system:

1. The watchdog process will notice this change and update the search index, it will insert if the document is new, delete and insert if updated and delete if removed.

The search index directory lives in the wiki directory and is hidden from the results of page listing.

I have also enabled scoring and highlighting, all search results are sorted by score with the highest score first. The results returned also contain a sentence of two with the keyword highlighted. The highlighting itself comes directly from Whoosh and by default makes the highlight bold, this is probably configurable but I kind of like it as it is.

I still need to write some tests for this.

[Linbreux]

Oh alright perfect. I thought it kept some cache in the folder or something :)

[kura]

There is one bug I still need to figure out a solution to, it involves the git_manager and _searchindex. Git tries to commit the search index and it breaks things, maybe the _searchindex should live outside of the Markdown folder like caching does?

[kura]

I see passing tests! 🚀

[Linbreux]

Yes I guess we could put it in the same location as the cache folder!

[kura]

@Linbreux I just submitted a rather large set of changes. When the path was changed to just be a relative path it broke edits and deletes, edit causing duplication in results and delete deleting everything. I have fixed this now by adding an additional ID on the records: filename. This will have the added benefit of making files with the same name in different directories unique.

I have switched the writer used for indexing and deleting to an asynchronous writer because I was running in to lock contention when edit a file (delete + index).

I have entirely rewritten the watchdog so that it is easier to test, it now has a start and stop function that can be called from within Python to trigger the background indexer.

[Linbreux]

@kura Prefect, I noticed that the index does not update when removing a file. This was the only bug I could find for know.

[kura]

@Linbreux you are right, I broke it when rewriting the watchdog code. I'll fix it now.

[Linbreux]

@kura perfect! Do you have some things you'd like to test or are you ready to merge?

[kura]

@Linbreux I would like to write some tests around the search class and watchdog. Give me 24 hours or so to get a few tests in if you wouldn't mind.

[Linbreux]

@kura No perfect, the more tests the better! Just let me know when you'd like to merge

[kura]

@Linbreux I have added a set of tests for Search and Watchdog.

I have also renamed the cache_dir to /dev/shm/wikd/cache so that I could also add /dev/shm/wikmd/searchindex to separate the searchindex from the content so that git didn't try to add them to the local repo.

I'm pretty happy with this now, if you want to give it a once over and re-test.

[kura]

I'd suggest when you're ready to merge that you also use the squash + merge function of GitHub rather than straight up merging. This is a ton of commits and it'll pollute the history so I'd suggest just squash to 1 commit and merging.

Alternatively when you're happy you can prod me and I'll squash locally and force push a squashed commit you can merge instead.

[Linbreux]

Alright, this looks very good! I'll squash them ;) Thanks again for the work you put into it.

[kura]

@Linbreux commented on 6 Sept 2022, 21:06 UTC:

Alright, this looks very good! I'll squash them ;) Thanks again for the work you put into it.

Thanks. And no worries, I've been doing other non-programming stuff at work recently and the lack of coding was driving me a little crazy so I needed to find an outlet. :D

[Linbreux]

@kura Haha I get it. Always welcome to make further adjustments on wikmd!