integration of human authored documentation

[Manu343726]

Hi,

I'm considering adopting hindsight, but there's a feature/use case I couldn't find. So before opening any issue I would like to ask here in case I missed something:

Say I had an existing human authored source of documentation, a wii for example. That wiki is alive and is maintained by humans alone. How would you integrate the knowledge of that wiki in a hindsight instance? What I would like is to have hindsight sync/import the "human wiki" from time to time, integrating the knowledge there into its memory, banks, semantic graph, etc.

Even in the case I didn't miss anything and there's really no feature per se for this, I would love to know what current users do in situations like this.

Thank you.

[Denimworld12]

There's no wiki connector, but the sync you're describing is a supported pattern rather than something you have to work around. The piece that makes it work is that a document has a caller-supplied id, and re-retaining that id is defined behaviour.

Give each wiki page a stable document_id — the page slug or URL works — and retain it on every sync:

POST /v1/default/banks/{bank_id}/memories
{
  "content": "<page markdown>",
  "document_id": "wiki/engineering/deploy-runbook",
  "context": "internal wiki",
  "tags": ["wiki", "engineering"],
  "metadata": {"url": "...", "revision": "1482", "author": "..."}
}

update_mode decides what happens when that id already exists. "replace" is the default and is what you want for a wiki: it "deletes old data and reprocesses from scratch", so an edited page doesn't leave its old facts behind. "append" exists for the other case, where new content is a continuation rather than a correction.

Deleted pages are the third case, and there's a dedicated endpoint:

DELETE /v1/default/banks/{bank_id}/documents/{document_id}

which cascades to the memory units extracted from it and their temporal, semantic and entity links. So a sync loop is really just three branches — new/changed pages retain, removed pages delete, everything else left alone.

The part that surprised me, and the reason periodic re-sync is practical: re-retaining an unchanged page is cheap. The internal force_reextract flag is documented as "re-run extraction even when the content is byte-identical to what is already stored", it defaults to False, and it is not accepted on the public retain API at all — it's set only by reprocess_document. So byte-identical content does not pay for extraction again. There is also a delta path that rewrites only the chunks that changed, so editing one section of a long page doesn't re-extract the whole thing (it applies to the first sub-batch, which is the whole document for anything that isn't oversized).

That means you can afford to just re-push the whole wiki on a schedule and let the content hash sort out what actually changed, instead of building change detection on your side. If you do want to skip unchanged pages anyway, keep the wiki revision in metadata and compare before you call.

Two things worth setting deliberately while you're there:

tags — the wiki is one source among others in the bank, and tags are what let you scope recall to it or exclude it later. Cheap to add now, annoying to backfill.

observation_scopes — this controls how facts from your pages get consolidated into observations. The default ("combined") runs one pass with all tags together. If you want wiki knowledge consolidated as its own body rather than blended with, say, chat history in the same bank, that's the knob.

One thing to keep separate in your head: knowledge pages are documents Hindsight writes about the bank, so they're not where your human-authored wiki goes — your wiki is input, retained as documents; knowledge pages are output. Easy to conflate since both end up looking like a wiki.

For auditing the sync afterwards, GET /v1/default/banks/{bank_id}/documents returns memory_unit_count per document. A page sitting at 0 retained fine but produced no facts — usually a mission too narrow for that page's content, and POST /documents/{document_id}/reprocess re-extracts from the stored text without re-uploading once you've widened it.