Repository navigation
v0.12.0
Upgrading: this release removes the bundle's ollama service; the platform embeds in process.
Back up (./byodt backup), save .env (cp .env .env.before-<version>), keep the previous bundle
tarball, and make room on disk for the model (about 1.26 GB) and the new images: data/ollama can
only go once the upgrade has worked. Take the new bundle and set PLATFORM_VERSION. In .env,
remove (or comment out) the EMBEDDING_MODEL=embeddinggemma, EMBEDDING_SIMILARITY_THRESHOLD and
OLLAMA_IMAGE lines: byodt refuses the first and notes the others. Optionally run
./byodt models pull to download the model ahead, then ./byodt update. Once the new start works,
sudo rm -rf data/ollama reclaims 0.6–1 GB; byodt never deletes it. The first start after the
upgrade re-ingests the MITRE data once (about a minute, with the platform down meanwhile),
downloads the model once (about 1.26 GB, unless models pull placed it), and embeds every class
once, at about 0.2 s per class on a 4-vCPU host: seconds for the bundled modules (about 80 classes,
15–20 s), a few minutes with cloud content packages (about 1,200 classes); matching uses names
meanwhile. It then re-evaluates every element's findings once, in the background while the
platform serves, because no element records yet which class policy its findings came from:
findings the current class policies no longer emit are deleted with their dispositions (see
Fixed, A module update re-evaluates the findings of the classes it changed). Memory: the
platform uses about 0.8–0.9 GiB more, which the removed ollama container used before, so the
host's total is about the same. To roll back, unpack the previous bundle tarball over the bundle,
restore the saved .env, then run ./byodt update, ./byodt restore <backup> and
./byodt restart platform; anything created since the upgrade is lost. See the operator guide's
Operations, Upgrading from a release with the ollama service. Dethereal: 0.4.15 is published
after the platform; it also works with 0.11.1. 0.4.14 was never published: its changes ship in 0.4.15.
Added
byodt models pull(byodt 1.2.0) downloads and verifies the embedding model's weights into
data/modelsahead of the first start. It runs the platform's own code in a one-off platform
container, with no database, selects the model with the platform's rule (a refused
EMBEDDING_MODELis refused there too), and states the model's licence.Ctrl-Cstops it without
keeping a partial file. The platform image carries it asdist/scripts/models-pull.js, which exits
0 when the weights are in place, 1 on a refusal or failure, 130 when stopped.byodt up,byodt updateandbyodt models pullcheck.envbefore anything starts.
EMBEDDING_MODEL=embeddinggemma, however it is quoted, exported or followed by a comment, is refused
with an instruction to remove the line.OLLAMA_*,EMBEDDING_URLand
EMBEDDING_SIMILARITY_THRESHOLDlines get a "no longer used, you can remove the line" notice. A
leftoverdata/ollamais reported with its size andsudo rm -rf data/ollamaonce the new start
has worked; adata/ollamathat is a symbolic link is reported as one, without naming or measuring
its target.byodtnever edits.envand never deletesdata/ollama.console-initchecks the platform's embedding model. Before the schema and the ingest, it
refuses a platform model (EMBEDDING_MODEL, else the platform's default) other than the one the
release's MITRE data was embedded with, and the platform then does not start.byodt upand
byodt updatereport a start thatconsole-initstopped with'byodt logs console-init' says why, under podman-compose too, whoseup -dexits 0 when a dependency failed.- The platform logs the model's licence before a download. EmbeddingGemma is licensed under the
Gemma Terms of Use; the platform logs what it downloads, how big
it is and under which licence, once, before the download. - New embedder query settings.
EMBEDDING_QUERY_BACKLOG(an integer, 100 to 100000, default 200)
caps the query texts that may wait on the embedder.EMBEDDING_QUERY_TIMEOUT_MS(1 to 600000,
default 10000) andEMBEDDING_QUERY_TIMEOUT_PER_ITEM_MS(0 to 60000, default 250) set a match's
query-embed time limit: the base plus the per-text allowance for each text. A value outside its
range stops the platform from starting. See Security, Query embeds are bounded. VectorDisabledReasongainsEMBEDDER_BUSY.matchMitreTechniquesreports it when the
embedder was too busy to embed the request's queries in time; the deterministic tiers answered. The
technique picker's caption for it reads Semantic search is busy right now — showing name-based
matches. Try again in a moment. dt-core'sVectorDisabledReasontype includes it.ReindexResult.alreadyRunning. True when anotherreindexClassEmbeddingswas already running:
the call returned at once and re-embedded nothing./healthcounts embedder overloads. Theembedderblock now carriesqueryTimeouts(query
embeds that ran past their time limit) andqueryRefusals(query embeds refused because the
backlog was full), counted since the start. They do not change the overall status.
Changed
- Breaking: the platform embeds in process by default.
EMBEDDING_MODELnow defaults to
embeddinggemma-300m-retrieval-v1(EmbeddingGemma 300m, Google), which runs inside the platform:
its weights, about 1.26 GB, are downloaded once from Hugging Face intoEMBEDDING_MODEL_DIRand
verified by sha256. LeaveEMBEDDING_MODELunset. - Breaking: refused embedding models. With
EMBEDDING_ENABLED=truethe platform refuses to start,
before it connects to the database, onEMBEDDING_MODEL=embeddinggemma(the old HTTP model: remove
the line), onembeddinggemma-2-text-retrieval-v1(not available in this release: its MITRE vectors
are not published yet), and on any unknown id. With embedding off nothing is refused.
EMBEDDING_URL,EMBEDDING_API_KEY,EMBEDDING_DIMENSIONSandEMBEDDING_SIMILARITY_THRESHOLD
are ignored, with one warning each; they never stop a start.EMBEDDING_THRESHOLD_CLASSES,
_MITREand_COMPONENTS,EMBEDDING_MODEL_DIRandHF_TOKENare unchanged. - The bundle keeps the model's weights on the host. The platform gets
EMBEDDING_MODEL_DIR=/models,
bound to./data/models, so a recreate does not download them again.byodtcreatesdata/models
onup,updateandrestart, with the sticky bit, and refuses adata/*runtime directory that
another local account owns, a link or a plain directory. The platform refuses a weight file when
anything on its path below the models directory is a symbolic link, is owned by an account other
than its own, root or the models directory's owner, or is writable by accounts other than its owner
(a sticky directory excepted). No other local account can then swap the model between its sha256
check and its load, or plant a directory before a download, except root and, under rootful Docker,
a host account with the platform's uid (1001). Weights copied in for an offline host are copied as
the account that runs./byodt: files copied as root pass only under rootful Docker, and rootless
Podman refuses them. The bundle also passesEMBEDDING_THREADS(default 4) to the platform. - A shutdown during the model download stops at once. The platform aborts a download in flight,
or its retry wait, removes the partial file, and loads nothing once the shutdown has begun. - The shipped MITRE vectors are made with the platform's built-in profile. The MITRE ATT&CK,
D3FEND and ATLAS vectors are now embedded withembeddinggemma-300m-retrieval-v1, and the
mitre-frameworksmodule is now version 1.2.0. Its embeddings exporter defaults to the manifest's
profile and never skips: without a provider, or on any failure, it exits non-zero and writes nothing. - Dethereal's
manage_countermeasuresno longer updates or deletes a SYSTEM countermeasure (Dethereal 0.4.15). A
SYSTEM countermeasure (or one with nocreatedBy) is derived from its control's class policy, and
the platform rewrites it from that policy on each save of the control.updatenow refuses any
update of one: links,exposure_idsand plain fields alike. The error names the countermeasure and
says to supersede it into a USER copy (in the platform, Customize as an editable copy on the
countermeasure) and update the copy.deleterefuses one too, since the next save would re-create
it; the error says to change the control's attribute answers so the policy no longer derives it, or
to supersede it into a USER copy. USER countermeasures update and delete as before;create,
getandlistare unchanged.
Removed
- Breaking: the HTTP (Ollama- or OpenAI-compatible) embedder. The platform no longer calls an
external embedding endpoint. - Breaking: the bundle's
ollamaservice.byodt1.2.0 removes the oldollamacontainer on
up,update,restart,downanddestroy, found by its compose project and service labels;
its image anddata/ollamaare kept.byodt versionno longer listsOLLAMA_IMAGE.
Fixed
- A control reference no longer attaches a control by part of its name. Import and model update
attached a control whose name only contained, or was contained in, the name a model file gave, so
a reference to "Encryption at rest for S3 buckets" could attach a control called "Encryption", and
"Firewall" whichever firewall control came first. A control is now attached by its id, by its
exact name, or by a name that matches a single control when case and surrounding spaces are
ignored. A name that matches no control, or that several controls share, attaches nothing: the
import or update result lists it underunresolvedControls, and a warning names it, with the ids
of the controls a shared name matched so the reference can be pinned to one. Each entry under
unresolvedControlsnames, asowner, the model or element the reference came from. Dethereal
users get this fix with Dethereal 0.4.15.
Check your model files: a control that a name-only reference attached only by part of its name
("Firewall" for a control called "Perimeter Firewall") is detached from its element on the next
model update. Dethereal keeps the reference in the model files and warns about it; give it the
control's id or its full name to attach the control again. - A model update no longer removes a control because its name is shared. When a name in the
model file matches several controls and the element already has one of them, the update keeps it
and warns, where it used to replace the element's controls without it. - An import attaches a control by name only against the full control catalog. When the
catalog of controls in folders cannot be fetched, a name could look unique among the controls
outside folders and attach the wrong one; such a reference is now reported as unresolved, and only
an id attaches. - Dethereal keeps an unresolved control reference in the local model files. A push rewrites
the files from the platform, which dropped any control reference the platform could not attach,
including a name several controls share. The reference now stays where it was written, so it can
be pinned to a control id, and the next push reports it again. - A malformed answer from a remote knowledge-graph service is no longer read as an empty graph.
An answer without a result list, with a malformed entry, or missing one of the keys it was asked
for was taken to mean the graph held nothing for those keys, and was cached as such. It is now a
service fault, like a truncated answer: the query fails and nothing is cached. - "Add to Issue board" from a module's finding links what it names. A finding raised on a model
with no element put an empty id on the issue board's clipboard, so adding it to an issue failed
and linked nothing, without a word. Only the ids given are carried now, and the issue card says
when elements could not be linked. - A module's own finding description is the issue's description. An issue raised from a module
finding that carries a description was described as "Exposure: ". The description
is now used as given; a finding passed by name only is still described as "Exposure: ". - Changing the embedding model re-embeds the installed classes. At start the platform skipped a
module whose content hash matched its last complete install. Only modules written in code, which
ship no hash, noticed a change of embedding model; every file module and every published package
module was skipped, its classes kept vectors from the old model, and the class vector tier stayed
off. Each install now records the embedding model its class vectors are made with, and a change of
model, of dimensions, or switching embedding on or off reinstalls every module at the next start
so its classes are embedded again. The start also checks the classes' own vectors: a module whose
searchable classes hold vectors of another model, or none, is installed again even when its record
matches, for example after an older platform version installed it again. The first start after
the upgrade reinstalls every module once, because no module has that record yet; later starts
skip them again while they are unchanged. With the embedding model unchanged, that reinstall
reuses the vectors the modules ship for it and embeds only classes that have none; it takes as
long as a full install of the modules. A manualreindexClassEmbeddingsrecords the model too, so
the next start does not reinstall those modules. - Classes are embedded when the embedder becomes ready, without a restart. A module installed
before the embedder was ready kept its stored class vectors and was marked partial, and only the
next start installed it again. That happened when an in-process model finished its download or load
after the module install. When the embedder becomes ready, the platform now installs the waiting
modules again, with vectors; modules already embedded with the active model are skipped. The module
install at start also waits up to 60 seconds for an in-process model whose files are already in
place; a download is never waited for. Module installs and module resets now run one at a time. - A save keeps a finding's links in line with its class policy. A SYSTEM exposure or
countermeasure that the class policy still emits kept every link an earlier save wrote, so a
technique, mitigation or D3FEND reference the policy had dropped stayed linked. A save now deletes
every link of the writer's own types (EXPLOITED_BYon exposures;RESPONDS_WITHand the eight
COUNTERMEASURE_<VERB>types on countermeasures) to a MITRE or regulatory node that is not among
the finding's resolved references in that save. A link added by hand to a SYSTEM finding is
removed too. A link the policy keeps stays once: its parallel copies are deleted. An edge to a
MITRE or regulatory node that lacks the property it is keyed by is left alone. USER findings,
including copies made with Customize, are never touched. Ajustificationthe policy no
longer gives is cleared from the edge, where it used to stay. A
target named twice under one link type is written once, and the mention with a justification
wins. A dispositioned SYSTEM finding whose links change is flagged stale, as an attribute change
flags it, and the save'sstaleFlippedCountincludes it.
Upgrading: the first start re-evaluates every element once in the background (see A module
update re-evaluates the findings of the classes it changed), which brings its links in line. An
element of a remote module, or of a class without apolicyHash, is not re-evaluated and keeps its
old links until it is saved again; saving it with no changes (setInstantiationAttributeswith
{}) is enough. That first re-evaluation or save can mark as stale any saved result that
fingerprints the removed links. USER copies made before this release keep the links they were
copied with. - A binding change removes the findings a kept class no longer emits.
changeElementBinding
deleted only the findings of the classes it removed, so a Control that kept a class while its
class set changed also kept that class's countermeasures, even ones the class no longer emits.
After the upsert of each class the element is bound to, the binding change now runs that class's
obsolete-finding cleanup, the same one a save runs, and counts what it deletes in
deletedDerivedCountermeasures/deletedDerivedExposures. - A module update re-evaluates the findings of the classes it changed. An element's findings
followed the class policy of its last save, so after a module update it kept findings the new
policy no longer emits, lacked the ones it now emits, and kept old links until someone saved it.
Each class's metadata now carriespolicyHash, the sha256 of the policy source its evaluation
compiles (the hash of the empty source for a class with no policy; none for an issue class or a
policy that failed to register), and the install writes it on the class node; dt-module's class
metadata types declare it. Every save (setInstantiationAttributes, and so import) and every
changeElementBindingrecords on the element the class'spolicyHashas read before the
evaluation, in an internal property that is not in the GraphQL schema or in exports. It records
it only when the module that evaluated reports the same hash through the new optional
DTModule.getPolicyHash(a module without it is taken at the stored hash), so an instance still
holding an older policy leaves the binding unrecorded, and a later sweep re-evaluates it. After an
install that wrote at least one module (start-up, catch-up or module reset), and once in the
background after start, the platform finds the
bindings whose record differs from their class'spolicyHash, or that have none, and saves each
again with no attribute changes, through the save's own path and under the element's lock. New
findings are created, findings the policy no longer emits are deleted with their dispositions,
and links are reconciled, withdispositionStaleset on a link change, exactly as on a save. USER
findings, including copies made with Customize, and findings from other sources are not
touched. One sweep runs at a time; any number of installs during a sweep add one more after it.
The sweep and the element lock are per platform instance: run one platform instance (the bundle
does). Several instances each sweep, which is safe for the records but repeats the work.
An element that fails keeps its findings, is retried once at the end of the sweep, and is tried
again by the next sweep; a sweep never fails an install. Bindings of a class without a
policyHashare skipped and counted; a remote module's classes carry none, so its bindings, which
evaluate only with a caller's token, are skipped as such.
Each sweep that found anything logs one summary line (Re-evaluated findings after a class policy change, a warning when an element failed) with the counts and up to three ids each of the failed
and the skipped elements.
Upgrading: no element has a record yet, so the first start re-evaluates every element once in
the background after start. Findings the current class policies no longer emit are deleted then,
with their dispositions. A USER copy made with Customize is kept, even when its original is
deleted. A module whose content did not change is still reinstalled once when its classes'
policyHashis not recorded on their class nodes, so that re-evaluation covers its elements too.
Security
- Query embeds are bounded, so a flood of matches cannot stall the embedder. The in-process
embedder serves one request at a time, and any signed-in client could queue unbounded query work
behind it throughmatchClassesandmatchMitreTechniques, so every match waited on it. The
embedder now counts the query texts posted to it and not yet answered, and refuses at once a query
embed that would take the count pastEMBEDDING_QUERY_BACKLOG(default 200). A match's query embed
also has a time limit:EMBEDDING_QUERY_TIMEOUT_MS(default 10 s) plus
EMBEDDING_QUERY_TIMEOUT_PER_ITEM_MS(default 250 ms) per text. The model cannot be stopped
mid-run, so a timed-out embed keeps its place in the count until the model finishes it, and its
result is dropped. On a refusal or a timeout the match answers from the name tiers with
vectorAvailable: false, andmatchMitreTechniquesreportsvectorDisabledReason: EMBEDDER_BUSY.
Neither is retried text by text, and the text-by-text fallback after a failed batch stops at the
first one. Document embeds (module install, catch-up,reindexClassEmbeddings) are neither counted
nor timed. The warm-up is a query embed of one text: it is counted, has no time limit, and runs
before any match can reach the embedder.matchClassescuts each element's composed query text
to 2,000 characters. A secondreindexClassEmbeddingswhile one runs returns at once with
alreadyRunning: trueand embeds nothing. Each refusal or timeout logs one warning per request,
with counts and no query text, and/healthcounts both (see Added). - The MITRE embeddings build refuses headers and statements outside the shape it writes. The
05and09embeddings files open with//header comments that carry the weights' repository,
revision and dtype from the embedder helper's ready line, so a value holding a line break could
end its comment and put a statement into the file. The exporter now refuses a ready line whose
repo,revisionordtypedoes not fully match its plain ASCII format (a trailing newline
fails it), and refuses a header line that is not printable before it opens the file. dt-module's
validateProfilerefuses aruntime.dtypeoutside[a-z0-9]+. Both embeddings file checks (the
module's packaging check and the platform's CI test) now match each statement, with its comments
dropped, against the exact shape the exporter writes: theUNWINDof the rows, oneMATCHand
oneSETofembeddingandembeddingModel, so a statement with a clause before itsUNWIND,
between its rows and itsMATCH, or after itsSETis refused. The shipped files pass both.