It's too slow #5986
Replies: 3 comments
|
I forgot to add that during indexing it failed two separate times with LadyBug and I lost a day because backups weren't being done aggressively. |
|
Hey, thanks for the feedback! Could you please share your setup? We made a lot of serious improvements on ingestion speed which is why I am quite surprised to see this feedback now. Happy to help where possible. Reference discussion from discord: https://discord.com/channels/1120795297094832337/1534973774141722776 |
|
When indexing a moderate documentation set (200+ files) in Cognee, a multi-day runtime and Python CPU spikes usually indicate two specific pipeline bottlenecks: 1. Chunk Granularity vs LLM Extraction Round-TripsBy default, unstructured text is split into small chunk windows. In If 244 documents yield ~2,500 chunks, and each LLM extraction takes ~1.5s over the network sequentially or with low concurrency, the math quickly blows up to hours or days:
2. Eliminating Python CPU Spikes (Embeddings & Tokenization)Python CPU spikes during ingestion are almost always caused by local embedding inference (such as
3. Checkpointed Ingestion (Preventing Lost Days on Failure)Running all 244 documents in a single massive Instead, process documents in batches of 20 to 30: import cognee
DOCS = [...] # list of 244 file paths
BATCH_SIZE = 25
for i in range(0, len(DOCS), BATCH_SIZE):
batch = DOCS[i : i + BATCH_SIZE]
print(f"Ingesting batch {i // BATCH_SIZE + 1} ({len(batch)} files)...")
await cognee.add(batch, dataset_name="documentation")
await cognee.cognify(datasets="documentation", chunk_size=1024)
print(f"Batch {i // BATCH_SIZE + 1} committed and persisted to graph store.")Processing in batches commits the graph nodes and vector indices to disk incrementally after each batch. If a rate limit or network glitch occurs on document 180, documents 1 through 175 remain safely persisted and verified. |
Uh oh!
There was an error while loading. Please reload this page.
This is just random feedback. I use the superpowers skill and I just wanted to index my documentation. There's about 244 documents, and at the rate of congee processing it would have taken 4 days to complete with ChatGPT Astra. There were also a lot of cpu spikes with python. On the bright side, the results for another project where indexing completed was much better than graphify.
All reactions