Skip to content

v0.12.0

Choose a tag to compare

@github-actions github-actions released this 16 Oct 16:40
· 37 commits to main since this release
cdd8c8f

Upgrading: this release removes the bundle's ollama service; the platform embeds in process.
Back up (./byodt backup), save .env (cp .env .env.before-<version>), keep the previous bundle
tarball, and make room on disk for the model (about 1.26 GB) and the new images: data/ollama can
only go once the upgrade has worked. Take the new bundle and set PLATFORM_VERSION. In .env,
remove (or comment out) the EMBEDDING_MODEL=embeddinggemma, EMBEDDING_SIMILARITY_THRESHOLD and
OLLAMA_IMAGE lines:
byodt refuses the first and notes the others. Optionally run
./byodt models pull to download the model ahead, then ./byodt update. Once the new start works,
sudo rm -rf data/ollama reclaims 0.6–1 GB; byodt never deletes it. The first start after the
upgrade
re-ingests the MITRE data once (about a minute, with the platform down meanwhile),
downloads the model once (about 1.26 GB, unless models pull placed it), and embeds every class
once, at about 0.2 s per class on a 4-vCPU host: seconds for the bundled modules (about 80 classes,
15–20 s), a few minutes with cloud content packages (about 1,200 classes); matching uses names
meanwhile.
It then re-evaluates every element's findings once, in the background while the
platform serves, because no element records yet which class policy its findings came from:
findings the current class policies no longer emit are deleted with their dispositions
(see
Fixed, A module update re-evaluates the findings of the classes it changed). Memory: the
platform uses about 0.8–0.9 GiB more, which the removed ollama container used before, so the
host's total is about the same. To roll back, unpack the previous bundle tarball over the bundle,
restore the saved .env, then run ./byodt update, ./byodt restore <backup> and
./byodt restart platform; anything created since the upgrade is lost. See the operator guide's
Operations, Upgrading from a release with the ollama service. Dethereal: 0.4.15 is published
after the platform; it also works with 0.11.1. 0.4.14 was never published: its changes ship in 0.4.15.

Added

  • byodt models pull (byodt 1.2.0) downloads and verifies the embedding model's weights into
    data/models ahead of the first start. It runs the platform's own code in a one-off platform
    container, with no database, selects the model with the platform's rule (a refused
    EMBEDDING_MODEL is refused there too), and states the model's licence. Ctrl-C stops it without
    keeping a partial file. The platform image carries it as dist/scripts/models-pull.js, which exits
    0 when the weights are in place, 1 on a refusal or failure, 130 when stopped.
  • byodt up, byodt update and byodt models pull check .env before anything starts.
    EMBEDDING_MODEL=embeddinggemma, however it is quoted, exported or followed by a comment, is refused
    with an instruction to remove the line. OLLAMA_*, EMBEDDING_URL and
    EMBEDDING_SIMILARITY_THRESHOLD lines get a "no longer used, you can remove the line" notice. A
    leftover data/ollama is reported with its size and sudo rm -rf data/ollama once the new start
    has worked; a data/ollama that is a symbolic link is reported as one, without naming or measuring
    its target. byodt never edits .env and never deletes data/ollama.
  • console-init checks the platform's embedding model. Before the schema and the ingest, it
    refuses a platform model (EMBEDDING_MODEL, else the platform's default) other than the one the
    release's MITRE data was embedded with, and the platform then does not start. byodt up and
    byodt update report a start that console-init stopped with 'byodt logs console-init' says why, under podman-compose too, whose up -d exits 0 when a dependency failed.
  • The platform logs the model's licence before a download. EmbeddingGemma is licensed under the
    Gemma Terms of Use; the platform logs what it downloads, how big
    it is and under which licence, once, before the download.
  • New embedder query settings. EMBEDDING_QUERY_BACKLOG (an integer, 100 to 100000, default 200)
    caps the query texts that may wait on the embedder. EMBEDDING_QUERY_TIMEOUT_MS (1 to 600000,
    default 10000) and EMBEDDING_QUERY_TIMEOUT_PER_ITEM_MS (0 to 60000, default 250) set a match's
    query-embed time limit: the base plus the per-text allowance for each text. A value outside its
    range stops the platform from starting. See Security, Query embeds are bounded.
  • VectorDisabledReason gains EMBEDDER_BUSY. matchMitreTechniques reports it when the
    embedder was too busy to embed the request's queries in time; the deterministic tiers answered. The
    technique picker's caption for it reads Semantic search is busy right now — showing name-based
    matches. Try again in a moment.
    dt-core's VectorDisabledReason type includes it.
  • ReindexResult.alreadyRunning. True when another reindexClassEmbeddings was already running:
    the call returned at once and re-embedded nothing.
  • /health counts embedder overloads. The embedder block now carries queryTimeouts (query
    embeds that ran past their time limit) and queryRefusals (query embeds refused because the
    backlog was full), counted since the start. They do not change the overall status.

Changed

  • Breaking: the platform embeds in process by default. EMBEDDING_MODEL now defaults to
    embeddinggemma-300m-retrieval-v1 (EmbeddingGemma 300m, Google), which runs inside the platform:
    its weights, about 1.26 GB, are downloaded once from Hugging Face into EMBEDDING_MODEL_DIR and
    verified by sha256. Leave EMBEDDING_MODEL unset.
  • Breaking: refused embedding models. With EMBEDDING_ENABLED=true the platform refuses to start,
    before it connects to the database, on EMBEDDING_MODEL=embeddinggemma (the old HTTP model: remove
    the line), on embeddinggemma-2-text-retrieval-v1 (not available in this release: its MITRE vectors
    are not published yet), and on any unknown id. With embedding off nothing is refused.
    EMBEDDING_URL, EMBEDDING_API_KEY, EMBEDDING_DIMENSIONS and EMBEDDING_SIMILARITY_THRESHOLD
    are ignored, with one warning each; they never stop a start. EMBEDDING_THRESHOLD_CLASSES,
    _MITRE and _COMPONENTS, EMBEDDING_MODEL_DIR and HF_TOKEN are unchanged.
  • The bundle keeps the model's weights on the host. The platform gets EMBEDDING_MODEL_DIR=/models,
    bound to ./data/models, so a recreate does not download them again. byodt creates data/models
    on up, update and restart, with the sticky bit, and refuses a data/* runtime directory that
    another local account owns, a link or a plain directory. The platform refuses a weight file when
    anything on its path below the models directory is a symbolic link, is owned by an account other
    than its own, root or the models directory's owner, or is writable by accounts other than its owner
    (a sticky directory excepted). No other local account can then swap the model between its sha256
    check and its load, or plant a directory before a download, except root and, under rootful Docker,
    a host account with the platform's uid (1001). Weights copied in for an offline host are copied as
    the account that runs ./byodt: files copied as root pass only under rootful Docker, and rootless
    Podman refuses them. The bundle also passes EMBEDDING_THREADS (default 4) to the platform.
  • A shutdown during the model download stops at once. The platform aborts a download in flight,
    or its retry wait, removes the partial file, and loads nothing once the shutdown has begun.
  • The shipped MITRE vectors are made with the platform's built-in profile. The MITRE ATT&CK,
    D3FEND and ATLAS vectors are now embedded with embeddinggemma-300m-retrieval-v1, and the
    mitre-frameworks module is now version 1.2.0. Its embeddings exporter defaults to the manifest's
    profile and never skips: without a provider, or on any failure, it exits non-zero and writes nothing.
  • Dethereal's manage_countermeasures no longer updates or deletes a SYSTEM countermeasure (Dethereal 0.4.15). A
    SYSTEM countermeasure (or one with no createdBy) is derived from its control's class policy, and
    the platform rewrites it from that policy on each save of the control. update now refuses any
    update of one: links, exposure_ids and plain fields alike. The error names the countermeasure and
    says to supersede it into a USER copy (in the platform, Customize as an editable copy on the
    countermeasure) and update the copy. delete refuses one too, since the next save would re-create
    it; the error says to change the control's attribute answers so the policy no longer derives it, or
    to supersede it into a USER copy. USER countermeasures update and delete as before; create,
    get and list are unchanged.

Removed

  • Breaking: the HTTP (Ollama- or OpenAI-compatible) embedder. The platform no longer calls an
    external embedding endpoint.
  • Breaking: the bundle's ollama service. byodt 1.2.0 removes the old ollama container on
    up, update, restart, down and destroy, found by its compose project and service labels;
    its image and data/ollama are kept. byodt version no longer lists OLLAMA_IMAGE.

Fixed

  • A control reference no longer attaches a control by part of its name. Import and model update
    attached a control whose name only contained, or was contained in, the name a model file gave, so
    a reference to "Encryption at rest for S3 buckets" could attach a control called "Encryption", and
    "Firewall" whichever firewall control came first. A control is now attached by its id, by its
    exact name, or by a name that matches a single control when case and surrounding spaces are
    ignored. A name that matches no control, or that several controls share, attaches nothing: the
    import or update result lists it under unresolvedControls, and a warning names it, with the ids
    of the controls a shared name matched so the reference can be pinned to one. Each entry under
    unresolvedControls names, as owner, the model or element the reference came from. Dethereal
    users get this fix with Dethereal 0.4.15.
    Check your model files: a control that a name-only reference attached only by part of its name
    ("Firewall" for a control called "Perimeter Firewall") is detached from its element on the next
    model update. Dethereal keeps the reference in the model files and warns about it; give it the
    control's id or its full name to attach the control again.
  • A model update no longer removes a control because its name is shared. When a name in the
    model file matches several controls and the element already has one of them, the update keeps it
    and warns, where it used to replace the element's controls without it.
  • An import attaches a control by name only against the full control catalog. When the
    catalog of controls in folders cannot be fetched, a name could look unique among the controls
    outside folders and attach the wrong one; such a reference is now reported as unresolved, and only
    an id attaches.
  • Dethereal keeps an unresolved control reference in the local model files. A push rewrites
    the files from the platform, which dropped any control reference the platform could not attach,
    including a name several controls share. The reference now stays where it was written, so it can
    be pinned to a control id, and the next push reports it again.
  • A malformed answer from a remote knowledge-graph service is no longer read as an empty graph.
    An answer without a result list, with a malformed entry, or missing one of the keys it was asked
    for was taken to mean the graph held nothing for those keys, and was cached as such. It is now a
    service fault, like a truncated answer: the query fails and nothing is cached.
  • "Add to Issue board" from a module's finding links what it names. A finding raised on a model
    with no element put an empty id on the issue board's clipboard, so adding it to an issue failed
    and linked nothing, without a word. Only the ids given are carried now, and the issue card says
    when elements could not be linked.
  • A module's own finding description is the issue's description. An issue raised from a module
    finding that carries a description was described as "Exposure: ". The description
    is now used as given; a finding passed by name only is still described as "Exposure: ".
  • Changing the embedding model re-embeds the installed classes. At start the platform skipped a
    module whose content hash matched its last complete install. Only modules written in code, which
    ship no hash, noticed a change of embedding model; every file module and every published package
    module was skipped, its classes kept vectors from the old model, and the class vector tier stayed
    off. Each install now records the embedding model its class vectors are made with, and a change of
    model, of dimensions, or switching embedding on or off reinstalls every module at the next start
    so its classes are embedded again. The start also checks the classes' own vectors: a module whose
    searchable classes hold vectors of another model, or none, is installed again even when its record
    matches, for example after an older platform version installed it again. The first start after
    the upgrade reinstalls every module once
    , because no module has that record yet; later starts
    skip them again while they are unchanged. With the embedding model unchanged, that reinstall
    reuses the vectors the modules ship for it and embeds only classes that have none; it takes as
    long as a full install of the modules. A manual reindexClassEmbeddings records the model too, so
    the next start does not reinstall those modules.
  • Classes are embedded when the embedder becomes ready, without a restart. A module installed
    before the embedder was ready kept its stored class vectors and was marked partial, and only the
    next start installed it again. That happened when an in-process model finished its download or load
    after the module install. When the embedder becomes ready, the platform now installs the waiting
    modules again, with vectors; modules already embedded with the active model are skipped. The module
    install at start also waits up to 60 seconds for an in-process model whose files are already in
    place; a download is never waited for. Module installs and module resets now run one at a time.
  • A save keeps a finding's links in line with its class policy. A SYSTEM exposure or
    countermeasure that the class policy still emits kept every link an earlier save wrote, so a
    technique, mitigation or D3FEND reference the policy had dropped stayed linked. A save now deletes
    every link of the writer's own types (EXPLOITED_BY on exposures; RESPONDS_WITH and the eight
    COUNTERMEASURE_<VERB> types on countermeasures) to a MITRE or regulatory node that is not among
    the finding's resolved references in that save. A link added by hand to a SYSTEM finding is
    removed too. A link the policy keeps stays once: its parallel copies are deleted. An edge to a
    MITRE or regulatory node that lacks the property it is keyed by is left alone. USER findings,
    including copies made with Customize, are never touched. A justification the policy no
    longer gives is cleared from the edge, where it used to stay. A
    target named twice under one link type is written once, and the mention with a justification
    wins. A dispositioned SYSTEM finding whose links change is flagged stale, as an attribute change
    flags it, and the save's staleFlippedCount includes it.
    Upgrading: the first start re-evaluates every element once in the background (see A module
    update re-evaluates the findings of the classes it changed
    ), which brings its links in line. An
    element of a remote module, or of a class without a policyHash, is not re-evaluated and keeps its
    old links until it is saved again; saving it with no changes (setInstantiationAttributes with
    {}) is enough. That first re-evaluation or save can mark as stale any saved result that
    fingerprints the removed links. USER copies made before this release keep the links they were
    copied with.
  • A binding change removes the findings a kept class no longer emits. changeElementBinding
    deleted only the findings of the classes it removed, so a Control that kept a class while its
    class set changed also kept that class's countermeasures, even ones the class no longer emits.
    After the upsert of each class the element is bound to, the binding change now runs that class's
    obsolete-finding cleanup, the same one a save runs, and counts what it deletes in
    deletedDerivedCountermeasures / deletedDerivedExposures.
  • A module update re-evaluates the findings of the classes it changed. An element's findings
    followed the class policy of its last save, so after a module update it kept findings the new
    policy no longer emits, lacked the ones it now emits, and kept old links until someone saved it.
    Each class's metadata now carries policyHash, the sha256 of the policy source its evaluation
    compiles (the hash of the empty source for a class with no policy; none for an issue class or a
    policy that failed to register), and the install writes it on the class node; dt-module's class
    metadata types declare it. Every save (setInstantiationAttributes, and so import) and every
    changeElementBinding records on the element the class's policyHash as read before the
    evaluation, in an internal property that is not in the GraphQL schema or in exports. It records
    it only when the module that evaluated reports the same hash through the new optional
    DTModule.getPolicyHash (a module without it is taken at the stored hash), so an instance still
    holding an older policy leaves the binding unrecorded, and a later sweep re-evaluates it. After an
    install that wrote at least one module (start-up, catch-up or module reset), and once in the
    background after start, the platform finds the
    bindings whose record differs from their class's policyHash, or that have none, and saves each
    again with no attribute changes, through the save's own path and under the element's lock. New
    findings are created, findings the policy no longer emits are deleted with their dispositions,
    and links are reconciled, with dispositionStale set on a link change, exactly as on a save. USER
    findings, including copies made with Customize, and findings from other sources are not
    touched. One sweep runs at a time; any number of installs during a sweep add one more after it.
    The sweep and the element lock are per platform instance: run one platform instance (the bundle
    does). Several instances each sweep, which is safe for the records but repeats the work.
    An element that fails keeps its findings, is retried once at the end of the sweep, and is tried
    again by the next sweep; a sweep never fails an install. Bindings of a class without a
    policyHash are skipped and counted; a remote module's classes carry none, so its bindings, which
    evaluate only with a caller's token, are skipped as such.
    Each sweep that found anything logs one summary line (Re-evaluated findings after a class policy change, a warning when an element failed) with the counts and up to three ids each of the failed
    and the skipped elements.
    Upgrading: no element has a record yet, so the first start re-evaluates every element once in
    the background after start. Findings the current class policies no longer emit are deleted then,
    with their dispositions. A USER copy made with Customize is kept, even when its original is
    deleted. A module whose content did not change is still reinstalled once when its classes'
    policyHash is not recorded on their class nodes, so that re-evaluation covers its elements too.

Security

  • Query embeds are bounded, so a flood of matches cannot stall the embedder. The in-process
    embedder serves one request at a time, and any signed-in client could queue unbounded query work
    behind it through matchClasses and matchMitreTechniques, so every match waited on it. The
    embedder now counts the query texts posted to it and not yet answered, and refuses at once a query
    embed that would take the count past EMBEDDING_QUERY_BACKLOG (default 200). A match's query embed
    also has a time limit: EMBEDDING_QUERY_TIMEOUT_MS (default 10 s) plus
    EMBEDDING_QUERY_TIMEOUT_PER_ITEM_MS (default 250 ms) per text. The model cannot be stopped
    mid-run, so a timed-out embed keeps its place in the count until the model finishes it, and its
    result is dropped. On a refusal or a timeout the match answers from the name tiers with
    vectorAvailable: false, and matchMitreTechniques reports vectorDisabledReason: EMBEDDER_BUSY.
    Neither is retried text by text, and the text-by-text fallback after a failed batch stops at the
    first one. Document embeds (module install, catch-up, reindexClassEmbeddings) are neither counted
    nor timed. The warm-up is a query embed of one text: it is counted, has no time limit, and runs
    before any match can reach the embedder. matchClasses cuts each element's composed query text
    to 2,000 characters. A second reindexClassEmbeddings while one runs returns at once with
    alreadyRunning: true and embeds nothing. Each refusal or timeout logs one warning per request,
    with counts and no query text, and /health counts both (see Added).
  • The MITRE embeddings build refuses headers and statements outside the shape it writes. The
    05 and 09 embeddings files open with // header comments that carry the weights' repository,
    revision and dtype from the embedder helper's ready line, so a value holding a line break could
    end its comment and put a statement into the file. The exporter now refuses a ready line whose
    repo, revision or dtype does not fully match its plain ASCII format (a trailing newline
    fails it), and refuses a header line that is not printable before it opens the file. dt-module's
    validateProfile refuses a runtime.dtype outside [a-z0-9]+. Both embeddings file checks (the
    module's packaging check and the platform's CI test) now match each statement, with its comments
    dropped, against the exact shape the exporter writes: the UNWIND of the rows, one MATCH and
    one SET of embedding and embeddingModel, so a statement with a clause before its UNWIND,
    between its rows and its MATCH, or after its SET is refused. The shipped files pass both.