Repository navigation
Layers and Presets
A knowledge layer holds the documents of one domain: a standard, a product's documentation, a team's runbooks. Each layer has its own chunk settings, its own routing patterns and a description. Searches can target one layer, several, or none. With none, the question is routed by each layer's patterns.
There are three ways to build a layer:
| Way | Best for |
|---|---|
stratakb add <folder> |
One folder, one layer, from a terminal. The quickest |
| From your MCP client | No shell at hand |
| A preset | A layer you keep under version control, with routing patterns and several source folders |
uv run stratakb add path/to/docs # create the layer, or sync it
uv run stratakb add path/to/docs --name team_docs # pick the layer name
uv run stratakb add path/to/docs --json # the report as JSON, for scriptsIt creates one layer for the folder, named after it, and ingests what it finds. The description comes from the folder's README unless you pass --description. One folder belongs to at most one layer.
Running it again syncs the folder: it re-embeds only the files that changed, adds new ones and removes files that were deleted.
flowchart LR
cmd["stratakb add <folder>"] --> plan["Plan: which layer,<br/>which files, what changed"]
plan -->|"broad path or<br/>large run"| ask{"Confirmed?<br/>prompt, --allow-broad, --yes"}
ask -->|"no"| stop["Refuse, nothing written<br/>(exit 2)"]
ask -->|"yes"| apply
plan -->|"otherwise"| apply["Create or sync the layer,<br/>ingest, audit"]
apply --> report["Report and exit code"]
-
Selection. Inside a git work tree, the list comes from git: tracked files plus untracked files that git does not ignore; submodules are left out. Elsewhere the folder is walked and only the root
.gitignoreapplies (the report notes nested ones it did not apply). In both cases it leaves out symbolic links, hidden files and folders (names starting with.), and the foldersnode_modules,venv,__pycache__,dist,build,target,site-packages,bower_componentsandvendor. -
.strataignore. A.strataignorein the folder's root, in.gitignoresyntax, excludes more. A!pattern there brings back a path the built-in list left out. - Policy. Each remaining file is checked before it is opened; the first match wins:
| Reason | Skipped when |
|---|---|
secret |
The name looks like a secret (see Secret names) |
path_too_long |
The path inside the folder is longer than 500 characters |
build_file |
A lock file or tool config (package-lock.json, uv.lock, go.sum, tsconfig*.json, requirements*.txt, *.min.js ...) or a CI workflow in .github/workflows/
|
code |
Source code (.py, .js, .ts, .java, .go, .rs, .sql, .sh, .css ...) |
unsupported |
No parser for the format (see Supported formats) |
too_large |
Bigger than --max-file-size megabytes (default 50) |
The report lists every skipped and excluded file with its reason.
Some runs need a yes first. At a terminal the command asks; in a script it refuses, writes nothing, and names the flag.
| Run | Flag |
|---|---|
| A drive root, your home folder, a folder above it, or a system folder | --allow-broad |
| Adds, updates or removes more than 2,000 files, adds or updates more than 500 MB, or removes at least 10 files that are more than half of the layer's files | --yes |
| Code | Means |
|---|---|
0 |
Done |
1 |
Some files failed (the rest are stored), or an unexpected error (internal_error) |
2 |
Refused or a usage error; nothing written |
3 |
The database is unreachable, or its schema can't be updated |
130 |
Interrupted; stored files stay, and the next run carries on |
stratakb add has no role check: whoever can reach the database with DATABASE_URL can add a folder. Each run writes one audit entry; see Roles and Audit.
Files whose names look like secrets are skipped without being opened, on every route: stratakb add, strata_ingest_documents and presets. A file of that kind that was ingested earlier is removed. The names are matched case-insensitively:
| Kind | Patterns |
|---|---|
| Environment files |
.env, .env.* (but not .env.example) |
| Keys and certificates |
*.pem, *.key, *.p12, *.pfx, id_rsa*, id_ed25519*
|
| Password stores and tool credentials |
*.kdbx, .npmrc, .pypirc, .netrc, kubeconfig*, *service-account*.json
|
| Anything named after secrets |
*secret*, *credential*, for example client_secret.json or secrets.yaml
|
Check what you add. A document in a document format (
.md,.rst,.adoc,.txt) whose name only mentions secrets or credentials, such ascredentials.txtorsecrets.md, is ingested on purpose, so that a guide about credentials can be found. The name check cannot see what such a file contains. Keep real secrets out of the folders you add, or list them in.strataignore.
| Format | Split along |
|---|---|
| Pages, then size | |
| DOCX | Paragraphs, then tables as text rows, grouped into about 3,000 characters, then size |
Markdown (.md, .markdown), HTML (.html, .htm) |
H1 to H3 sections |
Plain text: .txt, .rst, .adoc
|
Size |
| CSV | One chunk per row (the first 10,000 rows), each value labelled with its column header |
| JSON | One readable text per file, then size |
| XSD | One chunk per complex type or element group |
| XML (other than XSD) | One readable text per file, then size |
| OpenAPI and AsyncAPI YAML | One chunk per operation |
| Other YAML | One text per file, then size |
Each stored chunk is the document's own text as its parser extracted it. Where the chunk sits ([layer > file > headings] for Markdown and HTML, [layer > file > Page N] otherwise) is added only to the text that is embedded, so semantic search keeps the context while results and keyword search see only the document's words. A Markdown or HTML section holds its own heading once; the heading chain is in the chunk's section_path metadata. Chunks stored by earlier versions keep their bracketed prefix until their file is ingested again.
The server also keeps each document's extracted text, once per version, and every chunk records where it sits in that text. The version goes up only when the text changes, and an old version is removed once no chunk uses it.
flowchart LR
f["Files"] --> p["Parse<br/>per format"]
p -->|"no text"| s["Skip"]
p --> c["Chunk<br/>(the document's own text)"] --> x["Add context for the embedding only<br/>[layer > file > headings or Page N]"] --> e["Embed"] --> v[("Store the document text<br/>and its chunks")]
- Empty files and files with no text are skipped (
no_text), not reported as failures. - A PDF that looks scanned (fewer than half its pages have text, or under 200 characters per page on average) is stored for the text it has and flagged with
probably_scanned. Run it through an OCR tool such asocrmypdfand ingest the result. Built-in OCR is on the roadmap.
As an admin:
-
strata_create_knowledge_layerwith aname, adisplay_nameand adescription. -
strata_ingest_documentswith the layer name and an absolute path to a file or folder.
Run strata_ingest_documents again after you edit the documents. It syncs the folder and re-embeds only what changed:
flowchart LR
f["Each file<br/>in the folder"] --> n{"Secret-looking<br/>name?"}
n -->|"yes"| k["Skip, never opened"]
n -->|"no"| h{"Same content<br/>as last time?"}
h -->|"yes"| s["Skip"]
h -->|"new"| a["Add"]
h -->|"changed"| u["Replace its chunks"]
g["Tracked file no longer<br/>in the folder"] --> r["Remove its chunks"]
- Other folders ingested into the same layer are left alone. Ingesting a single file never removes other files; it removes that file only when it is now empty or has a secret-looking name.
- A file that fails to parse keeps its previous chunks and is listed under
errors; the rest of the run carries on. - The response reports
files_added,files_updated,files_unchanged,files_removed,files_skippedandfiles_flagged. - "Same content" means the same bytes. A change of the layer's
chunk_configor of the embedding model is not detected: passreplace_all=Trueto rebuild the whole layer. - Sources are stored by their path relative to the folder, for example
guides/readme.md.
This route and presets read every file in a supported format and skip only secret names and files with no text. The rest of the policy above (build files, code, size, path length, .strataignore, the built-in folder list) applies to stratakb add only.
A preset is a YAML file you can keep under version control and rebuild at any time:
name: runbooks
display_name: Ops Runbooks
description: Incident runbooks, on-call procedures and failover guides.
chunk_config: {chunk_size: 1200, chunk_overlap: 150}
routing_patterns: # questions matching any of these search this layer first
- '\brunbooks?\b'
- '\b(failover|incident|on-?call)\b'
answer_mode: grounded # or extractive; grounded when left out
sources:
- name: runbooks
path: runbooks # relative to the preset file
- name: postmortems
path: postmortems
download: https://wiki.example.com/postmortems # shown when the folder is missinguv run python scripts/apply_preset.py presets/runbooks.yaml --dry-run # parse only
uv run python scripts/apply_preset.py presets/runbooks.yaml # build
uv run python scripts/apply_preset.py presets/runbooks.yaml --docs-root /srv/docsThe strata_apply_preset tool does the same from a client. Applying a preset again deletes the layer's chunks and ingests every source again. The layer's description, chunk_config and routing_patterns are set only when the layer is created; to change them, delete the layer with strata_delete_knowledge_layer and apply the preset again. Folders that are missing are skipped and reported with their download link.
flowchart TD
y["preset.yaml"] --> l["Create the layer if it is new,<br/>then clear its chunks"]
l --> s{"For each source:<br/>folder exists?"}
s -->|yes| i["Ingest"]
s -->|no| k["Skip, report the download link"]
The answer_mode says how the layer's questions are to be answered: grounded (the default), where your client writes the answer from the cited passages, or extractive, for layers where paraphrase is not acceptable, answered with the document's own sentences. Any other value is refused. Applying a preset again sets the mode of an existing layer, and strata_list_knowledge_layers shows it. The server records the mode but does not act on it yet.
The description is used twice: it tells people what the layer holds, and it tells the query rewriter which terms the layer uses. "SEPA rulebooks: SCT, SCT Inst, SDD Core; reason codes such as AC04 and AM05" helps strata_ask far more than "Payments docs".
Routing patterns are Python regular expressions, matched without regard to case. Write them for the words that only appear in questions about that layer, such as identifiers, product names and acronyms. Patterns that match common words send every question to the layer. You can check routing by asking a question without layers and looking at which layers the response lists. A layer made with stratakb add has no routing patterns; use a preset when you want them.
flowchart LR
q["Question<br/>(no layers given)"] --> m["A routing pattern matches:<br/>0.7"]
a["Learned affinity<br/>x 0.3"] --> s["Score per layer"]
m --> s
s -->|">= 0.2"| l["Search those layers"]
s -->|"no layer reaches 0.2"| all["Search all layers"]
A routed search that finds fewer results than asked for, or only low-confidence ones, is run again across all layers; see When routing finds too little.
The repository ships three presets:
| Preset | What it holds |
|---|---|
examples/peps/ |
Python Enhancement Proposals. The quickest to try: git fetches eight PEPs in a few seconds, with no account |
examples/sepa/ |
SEPA payments. Includes a 12-code sample, so it works before you download anything |
examples/iso20022/ |
ISO 20022 messaging. Needs the downloaded schemas and documents |
To try the PEPs:
git clone --depth 1 --filter=blob:none --sparse https://github.com/python/peps.git examples/peps/source
git -C examples/peps/source sparse-checkout set --no-cone /peps/pep-0008.rst /peps/pep-0020.rst /peps/pep-0257.rst /peps/pep-0484.rst /peps/pep-0572.rst /peps/pep-0634.rst /peps/pep-0695.rst /peps/pep-0723.rst
uv run python scripts/apply_preset.py examples/peps/preset.yaml --dry-run # parse only, no database
uv run python scripts/apply_preset.py examples/peps/preset.yaml # create the layer and ingestIn Git Bash on Windows, put MSYS_NO_PATHCONV=1 in front of the sparse-checkout command. Then ask from your client, for example "How long should a line of Python be?". examples/README.md has the PEPs options (all PEPs, updates, using stratakb add instead) and lists where to download the SEPA rulebooks and ISO 20022 documents.
Update EMBEDDING_BACKEND, EMBEDDING_MODEL and EMBEDDING_DIMENSIONS (and EMBEDDING_BASE_URL for a model server), then re-embed every layer at once with vector_store.re_embed_all_layers(). Re-embedding uses the stored chunk text, so the source files don't need to be parsed again. It runs in one transaction and resizes the shared vector(...) column once when the dimensions change; if anything fails, every layer and the column stay as they were, and searches wait while the column is resized. Re-embed even when the dimensions match: vectors from two models cannot be compared. vector_store.re_embed_layer() re-embeds one layer at the column's current size only, because every layer shares the column. strata_admin_health reports a column of another size under embedding.
Getting started
Using StrataKB
Reference
Help