Make embedding batch size configurable instead of hardcoded to 50

[Sogl]

Summary

Embedding batch size is currently hardcoded to 50, which makes local CPU embedding and some OpenAI-compatible embedding providers much less usable than they should be.

It would help a lot if embedding batch size were configurable, ideally via environment variable first.

Current behavior

In open_notebook/utils/embedding.py#L21, embedding requests use a fixed batch size:

EMBEDDING_BATCH_SIZE = 50

This causes two practical problems:

1. Some embedding endpoints reject the request because the batch is too large.
2. Even when the provider accepts it, CPU-only local embedding becomes unnecessarily slow and fragile.

Real example

I tested Open Notebook with:

• local text-embeddings-inference
• model: BAAI/bge-m3
• OpenAI-compatible embedding endpoint
• CPU-only machine

A single book-sized PDF produced:

• 492 chunks
• average chunk size around 1143 chars

With the default batch size 50, the rebuild failed with:

Failed to generate embeddings using model 'BAAI/bge-m3' (batch 1/10, 50 texts):
OpenAI-compatible embedding endpoint error:
batch size 50 > maximum allowed batch size 32

After lowering the embedding batch size to 8, the same source completed successfully:

Successfully embedded source ... 492 chunks in 595.34s

Vector search then worked correctly on the embedded source.

Why this matters

This is not just a performance tweak.

A fixed batch size of 50 makes embeddings:

• fail on providers with lower max batch limits
• much harder to use on CPU-only local setups
• less portable across OpenAI-compatible embedding backends

Suggested fix

Minimum useful fix:

• make embedding batch size configurable via env
• example: OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE

Even better:

• if an embedding request fails because batch size is too large, retry automatically with a smaller batch

Expected outcome

This would make Open Notebook much easier to use with:

• local CPU embedding servers
• self-hosted OpenAI-compatible embedding APIs
• providers with stricter batch-size limits

[lfnovo]

Great report — very well documented, with a real example, logs and a concrete proposal. This is clearly a real limitation that affects local CPU setups and providers with stricter batch limits.

The minimum fix (env var OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE) is straightforward and we'd welcome a PR. There's also a related upstream issue in Esperanto about per-provider batch limits, but that doesn't block the env var fix here.

Would you be interested in contributing the change? The relevant file is open_notebook/utils/embedding.py — happy to review a PR!

[Sogl]

@lfnovo Done: #742