19 Knowledge Pack

19 — Knowledge Pack

Updated: 2026-10-07

🌐 English · Français

Status (2026-10-07) : the export / import tools work on a local file and the release builder now produces the pack ; no pack has been published yet (no GitHub Release carries one).

The Nokido Knowledge Pack is an optional, pre-vectorized snapshot of the Nokido source code chunks (modules forge_*.py, docs, curated skills) shipped as a single NPZ file with BGE-M3 1024D embeddings already computed. Fresh installs (and forks) can import it in one command and get instant semantic search over the Nokido codebase without running the local embedder on cold chunks.

🎯 Why ship a knowledge pack?

A fresh clone only has the text seeds (seed/*.jsonl, 11 files on 2026-09-29). Those seeds cover lessons, ADRs, biblio, system rules — but not the Nokido code itself. The user has two options out of the box :

• Wait for the local embedder (NokidoLlamaEmbed, :8099 — the old brain_worker :5557 is disabled) to vectorize the forge modules : 586 app/forge_*.py + 840 tools/forge_*.py on 2026-09-29.
• Run BM25-only semantic search (text keyword match, no embedding similarity). Loses 60% of the value of the Nokido anti-duplication workflow described in CLAUDE.md §3.

The knowledge pack ships the embeddings pre-computed, so the user gets (1) zero waiting time, (2) immediate semantic search over the Nokido codebase, (3) instant ability to run the anti-duplication rule before creating a new forge_*.py.

Forks especially benefit : they inherit the same vectorized base. If they add their own modules, those get embedded locally on top of the pre-shipped pack.

📦 What's in the pack

One rule decides what enters the pack : a chunk is kept only if its text (whitespace normalized) appears word for word in a file published in the public repository, read at the promoted commit. Its content is therefore already public : nothing private can leak, even if a chunk's source label is wrong, and nothing stale gets in. On top of that, a chunk is dropped if it matches a secret pattern, a private identity, or a RED signal of the egress gate (public profile). The pack is licensed like the repository, AGPL-3.0-or-later.

Format nokido-knowledge-pack/2 (NPZ archive, no pickled objects : the importer loads it with allow_pickle=False, so a downloaded pack can never run code) :

Field	Type	Description
embeddings	float16[N, 1024]	BGE-M3 dense vectors (read back as float32)
meta	uint8 (UTF-8 JSON)	manifest, chunk_ids, sources (published path), domains
texts	uint8 (UTF-8 JSON)	the chunk texts, verbatim

Domains read : nokido_doc, doctrine, forge_core, nokido_code, docs, policy_rules, policy_roles, policy_security. Memory, sessions, third-party documentation and watch material are never read.

📊 Pack size (measured 2026-10-07)

Against the public repository at 51d10ef : 29,695 chunks, 27.8 MB of text and about 30 MB of float16 vectors before compression. forge_knowledge_pack_export.py --dry-run prints the real count per domain (read / kept / dropped per reason) before anything is written. The vectors are BGE-M3 : the pack needs the pinned bge-m3-Q8_0.gguf embedder (pack local-llm) to answer queries.

🚀 Export workflow (maintainer-side)

# 1. Make sure your local RAG base is populated and embedded
#    (i.e. you've been using Nokido for a while)
python tools/forge_knowledge_pack_export.py --dist <public repo clone> --verbose --dry-run
# Shows stats : read / kept / dropped per reason, per domain

# 2. Generate the pack (the release builder does this with --with-assets)
python tools/forge_knowledge_pack_export.py --dist <public repo clone> --version 0.1.0
# → writes data/nokido_knowledge_pack_v0.1.0.npz
# → prints SHA256 (record it!)

# 3. Attach to GitHub Release
gh release create v0.1.0 \
    --title "Nokido v0.1.0 + Knowledge Pack" \
    --notes-file CHANGELOG.md \
    data/nokido_knowledge_pack_v0.1.0.npz

The exporter is idempotent — re-run it any time you want to refresh the pack with newer code chunks. Each release ships its own version.

📥 Import workflow (user-side)

Option A — Latest pack from GitHub Releases (one-liner)

python tools/forge_knowledge_pack_import.py --download

Downloads latest release asset, verifies SHA256 (manifest-embedded), imports into the local RAG base. Its default repository is Nokido-labs/nokido, the public showcase that carries the releases ; until it is public, use Option C.

Option B — Specific version

python tools/forge_knowledge_pack_import.py --download --version 0.1.0

Option C — Local file (air-gapped or dev)

python tools/forge_knowledge_pack_import.py \
    --from data/nokido_knowledge_pack_v0.1.0.npz

Option D — Verify before importing

python tools/forge_knowledge_pack_import.py --download \
    --expect-sha256 abc123def456...

🧪 What happens on import

Per chunk in the pack :

State in local DB	Action
Chunk doesn't exist	INSERT (text + embedding + metadata)
Chunk exists, embedding NULL	UPDATE embedding only (keep local text changes)
Chunk exists, embedding present	SKIP (don't overwrite local) — use --overwrite to force

The importer preserves any local modifications you've made — your forked edits to forge_*.py won't be reverted by the pack.

After import, the hub's index needs a reload. The importer calls POST /api/rag/reload, but the current hub exposes no such route (checked 2026-09-29) : restart the hub after an import.

🔄 Updating the pack

Maintainers re-export and re-release periodically (suggested cadence : once per minor version bump). Users re-import to get newer chunks :

python tools/forge_knowledge_pack_import.py --download --overwrite

The --overwrite flag is needed only if you want upstream chunks to replace your locally-edited versions. Default behavior preserves local edits.

🛡️ Security

The pack contains :

• ✅ Nokido source code chunks (the same text as the repository).
• ✅ Pre-computed BGE-M3 embeddings (one-way representation, can't reconstruct original text exactly).
• ❌ NO conversation logs, NO agent_messages, NO sessions, NO secrets, NO PII.

The export sanitization is append-only — chunks containing secrets are dropped (not redacted) to err on the safe side. The SHA256 in the manifest lets users verify they got the file the maintainer signed.

❓ FAQ

Q. Why not ship the pack inside the repo (data/ directory)?

A pack weighs several MB and grows with the code : too heavy for git history. GitHub Releases is the idiomatic distribution channel for "binary blob attached to a tagged release", and keeps the repo lean for users who only want the source.

Q. Why not host on HuggingFace Datasets?

Both work. GitHub Releases is simpler (no separate account needed, versioned alongside the code) but HF Datasets has better discoverability. We may publish a mirror on HF eventually — open an issue if you want to drive that.

Q. Can I fork and ship my own knowledge pack?

Yes — re-run forge_knowledge_pack_export.py on your fork's RAG, attach to your fork's GitHub Release. The importer --repo flag lets users target any fork.

Q. Will the pack ever contain user data?

No. The export filter explicitly drops agent_messages, conversation_log, shared_prompt_log, and any chunk matching PII heuristics. The only domains exported are nokido_code, nokido_docs, curated_skills, policy_rules — all source-code- equivalent content.

Q. What if the BGE-M3 model version changes upstream?

The manifest pins model: "BAAI/bge-m3". If you use a different embedder locally, the imported vectors won't align with yours. There is no one-shot rebuild flag today (forge_embed_auto_trigger.py only takes --max-passes) : plan a re-embed with your local model before mixing vectors from two embedders.

Q. Can the pack be malicious?

Only via the text content (Nokido source code chunks). The embeddings are just numpy float32 arrays — no executable payload possible. Verify SHA256 against the manifest before importing, and only download from the project's official release page.