Performance: Initialization takes 1.5 - 3 minutes on large Windows repos

[carson-byte]

I am experiencing significant initialization delays when using Semble on a large enterprise repository (Java/Spring/Angular) on Windows.

Environment:

• OS: Windows (win32 / NTFS)
• Repository Size: ~83,000 total files

Observed Behavior:

• When running a search at the root of the project, the first query takes approximately 3 minutes.
• Even when explicitly scoping the search to a backend subdirectory, the first query still takes 1 minute and 30 seconds.
• Subsequent searches in the same session are fast, but the initial "cold start" delay makes the tool difficult to use as an instant search replacement.

[Pringled]

Hey @carson-byte, thanks for reporting. Couple of questions:

• Are you using the latest version of Semble? (see Updating Semble in the quickstart)
• Do you have a .gitignore set up? By default we ignore some specific dirs, but it might be that there are files being indexed that should be ignored
• Does this happen on every fresh start of Semble, or just the first one (the first time using Semble downloads a few dependencies such as the model and language pack)

[carson-byte]

Are you using the latest version of Semble? (see Updating Semble in the quickstart)

• Yes.

Do you have a .gitignore set up? By default we ignore some specific dirs, but it might be that there are files being indexed that should be ignored

• Yes. .gitignore is setup properly to exclude all the unwanted info. Have been using it for the other tools for a long time.

Does this happen on every fresh start of Semble, or just the first one (the first time using Semble downloads a few dependencies such as the model and language pack)

• Every fresh start.

[stephantul]

For 83000 files an initial indexing time of 1.5 minutes is expected, and not a bug. Nevertheless, if you run this in a Python interpreter from the root of your project, what does it show? For what it's worth, if I run this on a repo with 5000 files it takes about 5 seconds. So 83000 files taking 90 seconds doesn't seem very outlandish.

from collections import Counter

from semble.index import SembleIndex

s = SembleIndex.from_path(".")
files = sorted(s._file_mapping)
print(len(files))
print(len(s.chunks))
languages = Counter([x.language for x in s.chunks])
print(languages)

If you run it in interactive mode, you can walk through files, and take a look at whether something weird got indexed.

[carson-byte]

Understood. But the user experience is bad in this case. Possible to have some enhancement to have option e.g. persistent active semble run in a terminal and MCP connect to it, so only need to warm up once?
Or any other better solution

[stephantul]

Alright, noted. We'll think of a solution for persistency. Closing this for now, feel free to reopen or open another issue if something pops up.

[evgenyop]

@stephantul - I think it's not nice to close an unresolved problem pretending it doesn't exist.
I also have the same issue on every cold start, and Codex times out on Semble MCP calls, despite increased timeouts in the config. We have only large-to-very-large repos in our environment, e.g., 400k lines of code in 2k files. .gitignore is properly configured everywhere. So this issue is very relevant to us.
I understand you run it in Python on a single CPU thread, so the multi-thread processing is not utilised. That'd be a quick and effective win if you could use the multiprocessing module or port the scanning part into a Rust extension.
By the way, I love this tool; it works well otherwise. :-)

[stephantul]

Hey @evgenyop,

Thanks for raising, your issue from the one raised by the OP. 400k lines of code in 2k files should index within seconds, so that's a genuine bug. Please open a different issue, describe what you are seeing, and we'll look at it.

To your other point: I don't think it is rude nor do I pretend this issue doesn't exist. It is unrealistic to expect to be able to index 83000 files within seconds, so this is literally unsolvable for us. Semble uses the fastest existing machine learning models under the hood. Any other code indexing library would take hours to index 83000 files. We purposefully did not add persistence or a client-server model directly: the MCP server has in-memory persistency, the CLI is for one-off queries. Almost all codebases index within seconds. This is the point of the library. If your code base is so large that the fastest models in the world can't index them, then indexing your codebase using models is probably not the right choice.