RAG

[SilasMarvin]

This PR adds semantic search as a method for retrieving context.

[PrimeTimeTran]

@SilasMarvin,
Crawling could suck for legacy projects as they might have huge codebases. One of my previous companies I worked on a project that was almost 15 years old. It was .NET so I had to restart the server every time I made a change. The feedback loop was painful.

I was on a decent system too. Imagine the user is working on ML project where they're also running a local model which requires compute resources. Might increase their startup time a lot too =).

Regardless, I'd like ask/suggest LSP's workspace get configuration method here.

Do you think we could read configs for the project instead of crawling? As a specification, I think we're free to make it work or add additional APIs if we want; right? We might have to do both long term to build a Co pilot Killer so I'm more interested in understanding your thoughts on the LSP spec here =).