cannot index vault of 7000 files in 30 folders - 150MB of markdown.

[tyrcho]

• Disable all other plugins besides Copilot (required)
• Screenshot of note + Copilot chat pane + dev console added (required)

Copilot version: 2.9.0

Describe how to reproduce
A clear and concise description of what the bug is. Clear steps to reproduce the behavior

Get a large list of markdown files
I started having this problem with 7000 files for a total of 150MB of markdown.

Open the folder as vault

Install Copilot plugin, enter OpenAI Key.

Set Partitions to 32 or 40 (I thought it would be needed for a large vault)

Switch to vault QA mode

The plugin starts indexing, creating the chunk files, and after about one hour will hang. I had to kill Obsidian on the morning and the indexing would not resume.

Expected behavior
A clear and concise description of what you expected to happen.

I'd like indexing to finish to I can chat with the vault.

Additional context
Add any other context about the problem here.

I have the plugin work really well in other vaults. Actually my largest vault has 10k files and 500MB (includes images).

Any suggestions how I can resolve this ? I've tried already several times to remove the chunk files and reindex this vault with different parameters (which costs a bunch of $ of OpenAI tokens) and it always hangs after indexing a few thousand files.

[tyrcho]

I was wondering why one chunk file is much larger than the others.

Here is how the chunk files look for my other large vault (which works well with copilot)

[logancyang]

The main partition is a full-text search index and not an embedding index, it's not partitionable and that's why it's larger than the others. The main partition's max is around 350MB and there's no way to bump that, unfortunately. The only workaround is to set exclusion filters in QA settings.

This is a known limitation and is listed on our website. The local index has a ceiling at the moment. Our next steps to address this:

1. Separation of full-text and embedding index into 2 separate search tools - keyword search and embedding search. And there will be optimization on the full-text index to make it smaller.
2. A local index has many limitations, one being the Obsidian app's RAM limit (Electron env). The solution is a cloud vector db we host and we will also make it self-hostable.

[tyrcho]

Nice, thanks for answering !

I was wondering if SQLite would be an option instead of local json files ? (I'm pretty noob on vector DBs).
Otherwise a self hosted DB would make perfect sense.

I had missed that limitation on the website, if others wonder :

Copilot Plus does have some limitations subject to the Obsidian plugin environment and the fact that Copilot index is stored locally. For example, Obsidian Electron environment has a limited memory size, so it may have difficulty handling exceptionally large vaults (e.g. an index over 1.5GB). Additionally, the plugin is primarily designed for desktop usage, mobile usage is experimental and feature parity is not guaranteed.