Incremental ingestion support

[yonikremer]

I have about 6k docs which need to be read (and understood) in some order. In order to extract the correct ideas from document B, you have to memorise the basics on doc A. How would you do that?

[yulinlina]

You probably don’t want “read 6k docs into one prompt”. You want incremental, dependency-aware ingestion:

1. Model docs as a DAG or ordered sequence.
2. Ingest prerequisite docs first.
3. Store distilled memory items, not just raw chunks.
4. When ingesting a dependent doc, retrieve the prerequisite memories as context.

For example:

type DocMeta = {
  id: string;
  source: string;
  prerequisites: string[]; // doc IDs that must be ingested first
  hash: string;
};

type MemoryItem = {
  id: string;
  docId: string;
  type: "summary" | "concept" | "fact" | "rule";
  text: string;
  concepts: string[];
  embedding?: number[];
};

Ingestion flow:

async function ingestDoc(doc: DocMeta) {
  await ensurePrerequisitesIngested(doc.prerequisites);

  // Retrieve already-memorized basics needed for this doc
  const priorContext = await memory.search({
    docIds: doc.prerequisites,
    types: ["summary", "concept", "fact"],
    topK: 20,
  });

  const chunks = await splitDocument(doc.source);

  const items = await extractMemoryItems(chunks, {
    priorContext,
    prompt: `
      Using the prior context, extract concepts, facts, and rules
      from this document. Resolve terms using prior context where possible.
    `,
  });

  await memory.upsert(items);
  await markIngested(doc.id, doc.hash);
}

For 6k docs, use a persistent queue:

while (true) {
  const ready = await getDocsWherePrereqsDone(limit);
  if (!ready.length) break;
  await Promise.all(ready.map(ingestDoc));
}

Key details:

• Store doc.hash so re-ingestion is idempotent.
• If a prerequisite changes, invalidate/reprocess dependents.
• For each doc, store at least:
  • short summary
  • key concepts
  • important facts/rules
  • provenance: docId, section, page, URL
• When answering questions about doc B, retrieve both:
  • relevant chunks from B
  • prerequisite concepts from A

If the reading order is not known in advance, do a first pass to extract concepts per doc, then infer dependencies from citations, headings, term overlap, or an LLM-generated prerequisite graph.

[yulinlina]

Yes—treat this as **incremental,

[yulinlina]

For 6k documents where later docs depend on earlier ones, I’d avoid “ingest everything at once” and build an incremental, stateful ingestion pipeline.

The basic idea:

1. Store each document’s extracted knowledge as memory entries with metadata: docId, order, prerequisites, version.
2. Process documents in dependency/order sequence.
3. Before ingesting document B, retrieve relevant memories from document A and include them as context for extraction.
4. Save a checkpoint after each document/batch so the process can resume.

Example pattern in TypeScript:

type Doc = {
  id: string;
  order: number;
  prerequisites: string[];
  content: string;
};

async function ingest(docs: Doc[]) {
  const sorted = docs.sort((a, b) => a.order - b.order);
  const lastCheckpoint = await getCheckpoint();

  for (const doc of sorted) {
    if (doc.order <= lastCheckpoint.order) continue;

    // Pull memories from prerequisite docs first
    const priorContext = await memory.search({
      query: doc.content.slice(0, 500),
      filter: {
        docId: { $in: doc.prerequisites },
      },
      limit: 20,
    });

    const extracted = await llm.extract({
      document: doc.content,
      priorContext,
      instructions: `
        Extract durable facts, concepts, and rules from this document.
        Use the prior context to resolve terms and dependencies.
      `,
    });

    await memory.upsert(
      extracted.map(fact => ({
        text: fact,
        docId: doc.id,
        order: doc.order,
        version: doc.version ?? 1,
      }))
    );

    await saveCheckpoint({ docId: doc.id, order: doc.order });
  }
}

Important details:

• Make ingestion idempotent: use docId + hash or docId + sectionId as memory keys.
• If documents form a dependency graph, process them topologically instead of by numeric order.
• Keep concurrency low across dependent documents; parallelize only independent branches.
• Store a short “summary memory” per document plus atomic facts, then retrieve both when processing dependent docs.

This gives you incremental ingestion without requiring the model to reread all previous documents every time.

[yulinlina]

Treat this as ordered/incremental ingestion with dependency-aware memory, not as “embed 6k docs and search later”.

A pattern that works well:

1. Model documents as an ordered graph

   • Each doc has id, order, dependsOn, contentHash, and status.
   • If B requires A, set B.dependsOn = [A].

2. Ingest in topological/order sequence

   • For each document, first retrieve memories from its prerequisite docs.
   • Use those memories as context while extracting/summarizing the current doc.
   • Store distilled memories back with metadata.

3. Store hierarchical memory

   • summary: high-level doc summary
   • concept: atomic facts/rules
   • chunk: raw passages if needed

4. Checkpoint progress

   • Persist lastProcessedOrder or per-doc status so ingestion can resume.
   • Use idempotent upserts keyed by doc/version.

Example TypeScript-ish flow:

type DocMeta = {
  id: string;
  order: number;
  dependsOn: string[];
  text: string;
};

async function ingestDoc(doc: DocMeta) {
  // 1. Load prerequisite knowledge

[yulinlina]

For 6k documents where document B depends on document A, don’t rely only on vector search. You need an explicit ingestion pipeline that preserves order and stores distilled “memory” from earlier docs.

A practical pattern:

1. Model documents as ordered/dependent items

   • If linear: store order.
   • If graph-like: store dependsOn: string[] and topologically sort.

2. Persist ingestion state
   Track each doc as pending | processing | done | failed, plus a content hash, so you can resume safely.

3. Ingest incrementally
   For each document:

   • retrieve summaries/concepts from prerequisite docs,
   • pass them as context to the extractor/LLM,
   • store both chunks and higher-level distilled memories.

Example TypeScript sketch:

type Doc = {
  id: string;
  text: string;
  dependsOn: string[];
};

async function ingestDoc(doc: Doc) {
  const prereqMemories = await memory.search({
    query: "core concepts required by this document",
    filter: {
      docId: { $in: doc.dependsOn },
      kind: "summary",
    },
    limit: 20,
  });

  const context = prereqMemories
    .map(m => m.content)
    .join("\n\n");

  const summary = await llm(`
Using this prerequisite knowledge:
${context}

Extract the essential concepts from the following document.
Document:
${doc.text}
  `.trim());

  await memory.upsert({
    id: `summary:${doc.id}`,
    content: summary,
    metadata: {
      docId: doc.id,
      kind: "summary",
    },
  });

  await markIngested(doc.id

[yulinlina]

For 6k ordered documents, avoid trying to “read everything at once.” Treat ingestion as an incremental, stateful pipeline where each document is processed only after its prerequisite knowledge has been stored.

A practical design:

1. Model documents as ordered items or a dependency graph.
2. Store ingestion state so the process can resume.
3. When processing document B, retrieve relevant stored knowledge from document A first.
4. Extract structured memory from B and upsert it.

Example TypeScript-style pipeline:

interface Doc {
  id: string;
  seq: number;
  text: string;
  checksum: string;
  dependsOn?: string[]; // e.g. ["doc-a"]
}

async function ingestDoc(
  doc: Doc,
  memory: MemoryStore,
  llm: LLMClient,
) {
  // Skip if already processed with same checksum
  if (await memory.isProcessed(doc.id, doc.checksum)) {
    return;
  }

  // Retrieve prerequisite knowledge, not full docs
  const priorMemory = await memory.search({
    filter: {
      docId: { $in: doc.dependsOn ?? [] },
    },
    limit: 50,
  });

  const chunks = chunkText(doc.text);

  for (const chunk of chunks) {
    const extracted = await llm

[yulinlina]

For 6k documents with dependencies, don’t ingest them as one flat corpus. Treat ingestion as an incremental DAG pipeline.

Recommended approach

1. Model each document as a node

   interface DocNode {
     id: string;
     content: string;
     version: string;
     prerequisites: string[]; // doc IDs that must be ingested first
   }

2. Ingest in topological order
   Only ingest B after all prerequisites of B are marked complete.

3. Store extracted memory, not just chunks
   For each document, extract:

   • key concepts
   • rules/constraints
   • summaries
   • dependencies on prior docs

4. When ingesting a dependent doc, retrieve prerequisite memory first
   Use prerequisite metadata filters rather than generic similarity search if possible.

Sketch

async function ingestDoc(doc: DocNode, memory: MemoryStore) {
  // 1. Fetch prerequisite knowledge
  const prerequisiteContext = await memory.search({
    filter: {
      sourceId: { $in:

[yonikremer]

@yulinlina Right now, the ingestion (of documents) doesn't use the existing memories in order to help understand new memories. I will patch this in my fork (you can this if you want)