Fix find_files slowness with SQLite FindIndex (#354).

[false200]

Fixes #354

So basically I added a new file, app/utils/file_index.py, that builds a SQLite index of filenames (FTS5 trigram on the basename). It lives at {base_directory}/.craftbot/findindex.db.

From the outside, find_files behaves the same:

• empty base_directory still goes to home
• the agent can still search wherever it wants
• we still crawl everything, including stuff like $Recycle.Bin
• the response is still just status, matches, and message - all matches, no head_limit

The difference is recursive searches hit the index instead of running os.walk every single time. Non recursive is unchanged - still just a top level glob.

After the first build, watchdog keeps the index updated with a 5 second debounce.

Also added **/.craftbot/ to .gitignore so those runtime DB files don't get committed.

[false200]

@makiroll1125 I've pushed an improvement for this issue. There isn't really a single best solution since each approach has tradeoffs, but this should make search work better than the previous one. Would appreciate a review.

[makiroll1125]

I listed some suggestions regarding search patterns and issues I noticed.

[false200]

I listed some suggestions regarding search patterns and issues I noticed.

I’ve implemented the required changes, please review when you’re free.

[makiroll1125]

I’ve implemented the required changes, please review when you’re free.

New commit here: f71d591
Please take a look when you have time. I made changes regarding prewarming all drives. CraftBot kicks off a background thread at launch that builds the index for every local fixed drive before the user asks. It is also much faster and has great performance.

I also noticed some other issues regarding incorrectly set parameters that was causing CraftBot to rebuild the entire index each time find_files was called. Try running some tests (ask CraftBot to search for files with a specific tag in their title) and let me know how it runs. Assuming no issues, could you pull the commit to your branch so that I can merge the PR?

[false200]

I’ve implemented the required changes, please review when you’re free.

New commit here: f71d591 Please take a look when you have time. I made changes regarding prewarming all drives. CraftBot kicks off a background thread at launch that builds the index for every local fixed drive before the user asks. It is also much faster and has great performance.

I also noticed some other issues regarding incorrectly set parameters that was causing CraftBot to rebuild the entire index each time find_files was called. Try running some tests (ask CraftBot to search for files with a specific tag in their title) and let me know how it runs. Assuming no issues, could you pull the commit to your branch so that I can merge the PR?

Sure!

[false200]

@makiroll1125 I've pulled it to my local branch. Feel free to merge it.

[makiroll1125]

Good to go