Layers and Presets

Layers and Presets

A knowledge layer holds the documents of one domain: a standard, a product's documentation, a team's runbooks. Each layer has its own chunk settings, its own routing patterns and a description. Searches can target one layer, several, or none. With none, the question is routed by each layer's patterns.

There are three ways to build a layer:

Way	Best for
stratakb add <folder>	One folder, one layer, from a terminal. The quickest
From your MCP client	No shell at hand
A preset	A layer you keep under version control, with routing patterns and several source folders

Add a folder

uv run stratakb add path/to/docs                      # create the layer, or sync it
uv run stratakb add path/to/docs --name team_docs     # pick the layer name
uv run stratakb add path/to/docs --json               # the report as JSON, for scripts

It creates one layer for the folder, named after it, and ingests what it finds. The description comes from the folder's README unless you pass --description. One folder belongs to at most one layer.

Running it again syncs the folder: it re-embeds only the files that changed, adds new ones and removes files that were deleted.

flowchart LR
    cmd["stratakb add &lt;folder&gt;"] --> plan["Plan: which layer,<br/>which files, what changed"]
    cmd -->|"STRATA_LOCAL_ONLY on and embedding<br/>leaves the machine, or embedding<br/>not set up (unknown backend, no key)"| stop
    plan -->|"broad path or<br/>large run"| ask{"Confirmed?<br/>prompt, --allow-broad, --yes"}
    ask -->|"no"| stop["Refuse, nothing written<br/>(exit 2)"]
    ask -->|"yes"| apply
    plan -->|"name taken, folder overlaps<br/>another layer, nothing to add"| stop
    plan -->|"otherwise"| apply["Create or sync the layer,<br/>ingest, audit"]
    apply --> report["Report and exit code"]

Loading

Which files it reads

1. Selection. Inside a git work tree, the list comes from git: tracked files plus untracked files that git does not ignore; submodules are left out. Elsewhere the folder is walked and only the root .gitignore applies (the report notes nested ones it did not apply). In both cases it leaves out symbolic links, hidden files and folders (names starting with .), and the folders node_modules, venv, __pycache__, dist, build, target, site-packages, bower_components and vendor.
2. .strataignore. A .strataignore in the folder's root, in .gitignore syntax, excludes more. A ! pattern there brings back a path the built-in list left out.
3. Policy. Each remaining file is checked before it is opened; the first match wins:

Reason	Skipped when
secret	The name looks like a secret (see Secret names)
path_too_long	The path inside the folder is longer than 500 characters
build_file	A lock file or tool config (package-lock.json, uv.lock, go.sum, tsconfig*.json, requirements*.txt, *.min.js ...) or a CI workflow in .github/workflows/
code	Source code (.py, .js, .ts, .java, .go, .rs, .sql, .sh, .css ...)
unsupported	No parser for the format (see Supported formats)
too_large	Bigger than --max-file-size megabytes (default 50)

The report lists every skipped and excluded file with its reason. It shows control characters from file names and layer names as escapes such as \x1b or \n, and keeps one line per file, so a file name cannot hide or rewrite what your terminal shows. --json writes DEL, C1 controls and bidi overrides as \uXXXX escapes and stays valid JSON.

Confirmations

Some runs need a yes first. At a terminal the command asks; in a script it refuses, writes nothing, and names the flag.

Run	Flag
A drive root, your home folder, a folder above it, or a system folder	--allow-broad
Adds, updates or removes more than 2,000 files, adds or updates more than 500 MB, or removes at least 10 files that are more than half of the layer's files	--yes

Exit codes

Code	Means
0	Done
1	Some files failed (the rest are stored), or an unexpected error (internal_error)
2	Refused (including local_only and embedding_setup) or a usage error; nothing written
3	The database is unreachable, or its schema can't be updated
130	Interrupted; stored files stay, and the next run carries on

stratakb add has no role check: whoever can reach the database with DATABASE_URL can add a folder. Each run writes one audit entry; see Roles and Audit.

Secret names

Files whose names look like secrets are skipped without being opened, on every route: stratakb add, strata_ingest_documents and presets. A file of that kind that was ingested earlier is removed. The names are matched case-insensitively:

Kind	Patterns
Environment files	.env, .env.* (but not .env.example)
Keys and certificates	*.pem, *.key, *.p12, *.pfx, id_rsa*, id_ed25519*
Password stores and tool credentials	*.kdbx, .npmrc, .pypirc, .netrc, kubeconfig*, *service-account*.json
Anything named after secrets	*secret*, *credential*, for example client_secret.json or secrets.yaml

Check what you add. A document in a document format (.md, .rst, .adoc, .txt) whose name only mentions secrets or credentials, such as credentials.txt or secrets.md, is ingested on purpose, so that a guide about credentials can be found. The name check cannot see what such a file contains. Keep real secrets out of the folders you add, or list them in .strataignore.

Supported formats

Format	Split along
PDF	Pages and the file's outline (bookmarks), then headings and numbered clauses, then size
DOCX	Paragraphs and tables (as text rows) in document order, grouped into about 3,000 characters, cut at heading styles, headings and numbered clauses, then size
Markdown (.md, .markdown), HTML (.html, .htm)	H1 to H3 sections, cut further at deeper headings and numbered clauses (see Structure and clauses); a # line inside a fenced code block is not a heading, headings may be indented up to three spaces or underlined with === / ---, and front matter (a leading --- block of key: value lines) is left out of the text; its updated, date and supersedes keys are read (see Document dates). An HTML page's first <title> outside an inline <svg> names the section path without being repeated in the text; an SVG's <title> stays in the text
Plain text: .txt, .rst, .adoc	Headings and numbered clauses, then size
CSV	One chunk per row (the first 10,000 rows; a longer file is flagged truncated), each value labelled with its column header; a column with no header name, or a row's extra values, is labelled by position (Column 4); a repeated header name gets a suffix (Name (2))
JSON	One readable text per file, then size
XSD	One chunk per complex type or element group
XML (other than XSD)	One readable text per file, then size
OpenAPI and AsyncAPI YAML	An overview chunk, then one chunk per OpenAPI operation, or per AsyncAPI channel and operation; an OpenAPI schema lists its first 15 properties (a longer one is flagged truncated)
Other YAML	One text per file, then size

Each stored chunk is the document's own text as its parser extracted it. Where the chunk sits ([layer > file > headings] for Markdown and HTML, [layer > file > Page N] otherwise) is added only to the text that is embedded, so semantic search keeps the context while results and keyword search see only the document's words. A Markdown or HTML section holds its own heading once; the heading chain is in the chunk's section_path metadata. Chunks stored by earlier versions keep their bracketed prefix until their file is ingested again.

The server also keeps each document's extracted text, once per version, and every chunk records where it sits in that text. The version goes up only when the text changes, and an old version is removed once no chunk and no receipt uses it.

Structure and clauses

Chunks follow the document's structure. Each document gets a structure map of its headings and numbered clauses, each with a path such as 7 Termination > 7.2 Termination for cause. Most of it needs no configuration: Markdown, reStructuredText and AsciiDoc headings, numbered headings (4.3.2, 7., 7(2)(b)) and keyword headings (Article 12, Section 3, Annex II). PDF and Word files also carry structure outside their text, and the map reads it first: each entry of a PDF's outline starts a section on its page, and each Word paragraph styled as a heading (Heading 1 to Heading 9, or any style with an outline level) starts a section at that depth. The headings and clauses found in the text nest inside them. A numbered line counts as a clause only when it continues the numbering and follows a blank line, a heading or the end of a sentence, so a numbered list inside a clause, a date or an amount is not one. In Markdown, a run of 1. 2. lines with no subclause among them is an ordered list and adds nothing. Table-of-contents entries are skipped. A preset can add its own section starts with structure_patterns.

A chunk never crosses a section or a PDF page. A heading with no text of its own before its first subsection (7. Termination directly followed by 7.1 …) joins that subsection's chunk and carries its path, unless a page starts at the subsection. A section longer than chunk_size is split, and every piece keeps the section's path, which each chunk stores as clause_path and each strata_ask citation shows. A document with no detectable structure is split by size. Chunks stored before this was added get a clause path when their file changes, or when the layer is rebuilt with replace_all=True.

flowchart LR
    o["PDF outline,<br/>Word heading styles"] --> d
    t["Document text"] --> d{"Headings, numbered clauses<br/>or the layer's structure patterns?"}
    d -->|yes| m["Structure map"] --> s["Cut at each section and page start;<br/>a bare heading joins its first subsection"] --> c["Chunks with<br/>clause_path"]
    d -->|no| z["Cut at page starts,<br/>split by size"] --> c2["Chunks without<br/>clause_path"]

Loading

flowchart LR
    f["Files"] --> p["Parse<br/>per format"]
    p -->|"no text"| s["Skip"]
    p --> c["Chunk<br/>(the document's own text)"] --> x["Add context for the embedding only<br/>[layer > file > headings or Page N]"] --> e["Embed"] --> v[("Store the document text<br/>and its chunks")]

Loading

• Empty files and files with no text are skipped (no_text), not reported as failures.
• A PDF that looks scanned is stored for the text it has and flagged with probably_scanned. Run it through an OCR tool such as ocrmypdf and ingest the result. Built-in OCR is on the roadmap. A page counts as scanned when it holds an image and under 200 characters of text, and a file is flagged when more than half its pages count (exactly half is not). A page whose images cannot be read counts as holding one. A PDF with no text at all is skipped as no_text, and the scanned flag wins over truncated.
• A CSV file over 10,000 rows is stored for its first 10,000 rows and flagged with truncated; the result's count says how many rows were left out, as does the server log (csv_truncated). Split the file to make every row searchable.
• An OpenAPI file lists the first 15 properties of each schema and says how many more there are. When a schema is cut this way, the file is flagged with truncated, and count holds the properties left out over the whole file, because those properties are not searchable.
• A PDF whose outline has more than 10,000 entries or goes deeper than nine levels is stored with its full text and flagged with truncated.

Document dates

Each stored document version records a date, which every citation shows. In every format but PDF and Word, a top-level updated key in the front matter wins, then a date key, as YYYY-MM-DD, optionally followed by a time. Otherwise, and for PDF and Word files, the date is the file's modification time as a UTC date, which a checkout or a copy can change. A value that is not a real date is ignored. An unchanged file stored before dates were recorded gets its date on the next ingest, without being embedded again.

What a document says it replaces

A document says it replaces another with a supersedes key in its front matter, naming one file relative to its own folder, like a Markdown link:

---
supersedes: failover-2024.md
---

Citations of the named file then carry replaced_by, the path of the file that says so (see Dates and labels). The rules:

• PDF and Word files cannot declare it, but can be named by it. One key names one file. The named file need not exist yet.
• The path is matched exactly, letter case too, within the same layer and source root, so it does not reach across a preset's source groups. A chain names only the next file.
• When several files name the same one, the citation gives the first by path.
• A value that cannot name a file in the folder (one that leaves the source root, an absolute path, a drive, a URL, a list, the file itself, an empty or over-long value) is ignored and never fails an ingest.
• It is only a declaration: anyone who can add a file to the folder can label another file.
• A file that held the key before it was recorded takes effect when the file changes or the layer is rebuilt.

From your MCP client

As an admin:

1. strata_create_knowledge_layer with a name, a display_name and a description.
2. strata_ingest_documents with the layer name and an absolute path to a file or folder.

Run strata_ingest_documents again after you edit the documents. It syncs the folder and re-embeds only what changed:

flowchart LR
    f["Each file<br/>in the folder"] --> n{"Secret-looking<br/>name?"}
    n -->|"yes"| k["Skip, never opened"]
    n -->|"no"| h{"Same content<br/>as last time?"}
    h -->|"yes"| s["Skip"]
    h -->|"new"| a["Add"]
    h -->|"changed"| u["Replace its chunks"]
    g["Tracked file no longer<br/>in the folder"] --> r["Remove its chunks"]

Loading

• Other folders ingested into the same layer are left alone. Ingesting a single file never removes other files; it removes that file only when it is now empty or has a secret-looking name.
• A file that cannot be read (locked, or no permission) or that its parser rejects keeps its previous chunks and is listed under errors with its reason; the rest of the run carries on. The error names the file by its path inside the folder, never the full path on disk.

Outcome	Reasons
added, updated, unchanged	none
flagged: stored, and also counted as added or updated	probably_scanned, truncated
skipped: not stored	secret, no_text
removed	deleted, not_selected, secret, no_text
failed: keeps its earlier chunks	read_error (could not be read), parse_error (the parser rejected it), embed_error (embedding or storing failed)

• The response reports files_added, files_updated, files_unchanged, files_removed, files_skipped and files_flagged. Each failed file is listed in errors with its reason and a short message. Each flagged file is listed in flagged, for example {"file": "codes.csv", "reason": "truncated", "count": 1234}, where count is null when the parser did not count what it left out. Both lists are null when empty. stratakb add prints flagged files in a Flagged: section after Excluded:, and leaves the section out when there are none.
• "Same content" means the same bytes. A change of the layer's chunk_config or of the embedding model is not detected: pass replace_all=True to rebuild the whole layer.
• Sources are stored by their path relative to the folder, for example guides/readme.md.

This route and presets read every file in a supported format and skip only secret names and files with no text. The rest of the policy above (build files, code, size, path length, .strataignore, the built-in folder list) applies to stratakb add only.

Presets

A preset is a YAML file you can keep under version control and rebuild at any time:

name: runbooks
display_name: Ops Runbooks
description: Incident runbooks, on-call procedures and failover guides.
chunk_config: {chunk_size: 1200, chunk_overlap: 150}
routing_patterns:            # questions matching any of these search this layer first
  - '\brunbooks?\b'
  - '\b(failover|incident|on-?call)\b'
structure_patterns:          # extra section starts, matched at the start of a line
  - '^(?P<label>RB-\d{3})\b'
answer_mode: grounded        # or extractive; grounded for a new layer when left out
archived_patterns: ['archive/', '*-2024.md', '!keep.md']   # quoted: YAML reads a leading * or ! itself
sources:
  - name: runbooks
    path: runbooks            # relative to the preset file
  - name: postmortems
    path: postmortems
    download: https://wiki.example.com/postmortems   # shown when the folder is missing

uv run python scripts/apply_preset.py presets/runbooks.yaml --dry-run    # parse only
uv run python scripts/apply_preset.py presets/runbooks.yaml              # build
uv run python scripts/apply_preset.py presets/runbooks.yaml --docs-root /srv/docs

The strata_apply_preset tool does the same from a client. Folders that are missing are skipped and reported with their download link.

To change a layer, edit its preset and apply it again; there is no need to delete the layer first. The layer takes the preset's display_name, description, metadata_schema, chunk_config, routing_patterns, structure_patterns, answer_mode and archived_patterns (the last two only when the preset gives them), the result lists the settings that changed (layer_changes), and the chunks are rebuilt from the sources. A layer with the same name that was not made by this preset (made with stratakb add, strata_create_knowledge_layer or another preset) is refused and left unchanged: rename the preset's name, or delete that layer. A layer created before origins were recorded is taken over by the preset. strata_apply_preset with dry_run=true also lists the settings that would change (layer_check: would_create, would_update, would_claim for a layer with no recorded origin that already matches the preset, unchanged, refused, or unknown when the database does not answer within 5 seconds); the script's --dry-run only parses the files and needs no database.

flowchart TD
    y["preset.yaml"] --> e{"Layer exists?"}
    e -->|no| c["Create the layer"]
    e -->|"yes, made by this preset<br/>or with no recorded origin"| u["Update the settings that differ<br/>(none when up to date)"]
    e -->|"yes, made another way"| r["Refuse: nothing changed"]
    c --> l["Clear the layer's chunks"]
    u --> l
    l --> s{"For each source:<br/>folder exists?"}
    s -->|yes| i["Ingest"]
    s -->|no| k["Skip, report the download link"]

Loading

structure_patterns are for sections named by identifiers no generic rule can know, such as the SEPA rulebook attributes AT-44. These regular expressions are matched at the start of a line, past any indent and against its first 200 characters. Every line one matches starts a section one level below the innermost open heading or clause, labelled by the pattern's named group label or by the whole match. A pattern that does not compile, that matches an empty line, or that repeats a repeated part (such as (\w+\s?)*) is refused when the preset is loaded. The check is strict and also refuses some harmless patterns, such as (x+y)*z. A layer that already holds an unsafe pattern, from an older version or a restore, skips it at search and at ingest and logs unsafe_structure_pattern with the layer; the form a question runs, the label group alone, is checked too. The same patterns serve exact lookups: asking strata_search or strata_ask to "show AT-R051" finds that section in the layers searched. The SEPA preset ships one for rulebook attribute definitions, and the ISO 20022 preset one for message identifiers.

The answer_mode says how the layer's questions are to be answered: grounded (the default), where your client writes the answer from the cited passages, or extractive, for layers where paraphrase is not acceptable, answered with the document's own sentences. Any other value is refused. Applying a preset again sets the mode of an existing layer when the preset gives one (a preset without answer_mode leaves it alone), and strata_list_knowledge_layers shows it. strata_create_knowledge_layer takes the same answer_mode. When a passage an answer returns comes from an extractive layer, strata_ask answers with the document's own sentences and a receipt; see Extractive answers. The SEPA and ISO 20022 example presets set answer_mode: grounded, so applying them again resets a layer that was set to extractive some other way. Change the value in the preset to keep it.

Writing a good description

The description is used twice: it tells people what the layer holds, and it tells the query rewriter which terms the layer uses. "SEPA rulebooks: SCT, SCT Inst, SDD Core; reason codes such as AC04 and AM05" helps strata_ask far more than "Payments docs".

Routing patterns

Routing patterns are Python regular expressions, matched without regard to case. Write them for the words that only appear in questions about that layer, such as identifiers, product names and acronyms. Patterns that match common words send every question to the layer. You can check routing by asking a question without layers and looking at which layers the response lists. A layer made with stratakb add has no routing patterns; use a preset when you want them. Only a question's first 2,000 characters are read. A pattern that does not compile or that repeats a repeated part (such as (\w+\s?)*) is refused when the layer is created or updated or the preset is loaded, and skipped with a warning if a layer already holds one.

Routing, structure and archived patterns are all refused when longer than 500 characters, when they hold a NUL or a lone surrogate (shown escaped in the message), or when Python warns about them, such as the POSIX class [[:alpha:]], which a later Python may read as a nested set. A preset's YAML error suggests quoting a value only when it starts with * or ! on that line.

Archived files

A layer can hold a list of path patterns saying which of its files are archived. Citations of those files then carry "archived": true (see Dates and labels). The label is worked out on every ask, so changing the patterns needs no new ingest, and it changes no ranking.

• Set them with strata_set_archived_patterns (admin) or the preset key archived_patterns. A preset without the key leaves the layer's patterns alone, and [] clears them; so does an empty list passed to the tool.
• Patterns are written as in .gitignore, not as regular expressions: archive/ for a folder, *-2024.md, !keep.md to take a file back. The last match wins, so the order counts.
• Each pattern is matched against the file's path relative to the folder ingested, letter case included. In a preset with several source groups, rulebooks/archive/ matches nothing: write archive/.
• A pattern is refused when it cannot compile, when it matches nothing (a comment, /, an open [), when it holds a line break, or when it has more than three wildcards (** as a path segment counts as one, any other star one each).
• Viewers and operators see the label, but not the patterns: layer listings show them to admins only.
• This is not the layer's archived status, which covers the whole layer.

flowchart LR
    q["Question<br/>(no layers given)"] --> m["A routing pattern matches:<br/>0.7"]
    a["Learned affinity ≥ 0.6<br/>(counted as 0.8) x 0.3"] --> s["Score per layer"]
    m --> s
    s -->|">= 0.2"| l["Search those layers"]
    s -->|"no layer reaches 0.2"| all["Search all layers"]

Loading

A routed search that finds fewer results than asked for, or only low-confidence ones, is run again across all layers; see When routing finds too little.

Examples

The repository ships three presets:

Preset	What it holds
examples/peps/	Python Enhancement Proposals. The quickest to try: git fetches eight PEPs in a few seconds, with no account
examples/sepa/	SEPA payments. Includes a 12-code sample, so it works before you download anything
examples/iso20022/	ISO 20022 messaging. Needs the downloaded schemas and documents

To try the PEPs:

git clone --depth 1 --filter=blob:none --sparse https://github.com/python/peps.git examples/peps/source
git -C examples/peps/source sparse-checkout set --no-cone /peps/pep-0008.rst /peps/pep-0020.rst /peps/pep-0257.rst /peps/pep-0484.rst /peps/pep-0572.rst /peps/pep-0634.rst /peps/pep-0695.rst /peps/pep-0723.rst
uv run python scripts/apply_preset.py examples/peps/preset.yaml --dry-run   # parse only, no database
uv run python scripts/apply_preset.py examples/peps/preset.yaml             # create the layer and ingest

In Git Bash on Windows, put MSYS_NO_PATHCONV=1 in front of the sparse-checkout command. Then ask from your client, for example "How long should a line of Python be?". examples/README.md has the PEPs options (all PEPs, updates, using stratakb add instead) and lists where to download the SEPA rulebooks and ISO 20022 documents.

Changing the embedding model

Update EMBEDDING_BACKEND, EMBEDDING_MODEL and EMBEDDING_DIMENSIONS (and EMBEDDING_BASE_URL for a model server), then re-embed every layer at once with vector_store.re_embed_all_layers(). Re-embedding uses the stored chunk text, so the source files don't need to be parsed again. It resizes the shared vector(...) column once when the dimensions change; if anything fails, every layer and the column stay as they were. It works in two phases: first it embeds the chunks a batch at a time while searches and ingests carry on, and then, still without a lock, the chunks added meanwhile (up to three extra passes). Then one transaction locks the table, resizes the column if needed and writes the new vectors. Ingests wait for that transaction, and searches too when the column is resized; on a large knowledge base, plan a resize for a quiet moment. Each statement of that transaction may take up to 10 minutes (locked_timeout=600, in seconds); pass a larger value for a very large knowledge base. re_embed_layer() stops with an error if another process resized the column while it ran. Re-embed even when the dimensions match: vectors from two models cannot be compared. vector_store.re_embed_layer() re-embeds one layer at the column's current size only, because every layer shares the column. strata_admin_health reports a column of another size under embedding.

Moving a layer

scripts/backup_restore.py backs up the whole database, or exports and imports a single layer. A layer export holds its configuration, file tracking, stored document text and chunks, so adding the same folder again after an import finds its files unchanged. Receipts are only in a full backup. An export keeps the folder's absolute path. When the folder sits elsewhere on the target machine, run import-layer there with --root <folder>. An import stops, changing nothing, when another layer holds the folder it would use (it names that layer; --root with a free folder gets past this), and when the file has a newer export format than the script reads. Before running anything, import-layer drops routing and structure patterns that do not compile or that repeat a repeated part, and names each one ahead of the confirmation (the 500-character, lone-surrogate and Python-warning checks are not applied on import). The current export format is 5. Older formats still import: a file without dates gets them when the folder is added again, one without supersedes values gets them when a file changes or the layer is rebuilt, and one without archived patterns leaves the layer with none.

Import only files you made or trust. restore and import-layer feed the file to psql with the database owner's rights, so a crafted file can do anything in the database.