v0.14.0

The compose bundle can reach the internet through an outbound HTTPS proxy. Each release now
publishes a signed installer, get-byodt.sh, and get-byodt.sh from the release is the way to
install. The npm package @dether.net/byodt is the same installer, once it is published to npm.
./byodt setup is the first run of a new bundle, and ./byodt upgrade moves a deployment to a
newer release in place. byodt and the installer show a third-party notice before the first
download. byodt is 1.4.0.

A control can be applied to a Data element: in the API, in dt-core, in the data dialog and in
Dethereal 0.4.19. The threat report lists a Data element's own controls. A slow remote module now
serves its read-only fallback and stays in service. The settings window no longer saves the wrong
values, drops a class change or shows old attribute values when a save and a load overlap. Database
logins on Memgraph are fast.

Upgrading: this is the first release with get-byodt.sh and ./byodt upgrade. A 0.13.0
deployment cannot upgrade itself: 0.13.0's ./byodt upgrade is another name for update, ignores
the version and stays on 0.13.0. Upgrade it once with this release's installer, naming the
deployment's folder, or by hand as before (see the operator guide's Operations, Upgrading from
0.13.0 and Upgrading by hand):

curl --proto '=https' --tlsv1.2 -fsSL https://github.com/dether-net/dethernety-oss/releases/download/v0.14.0/get-byodt.sh \
  | sh -s -- --upgrade --dir /path/to/your/deployment

From 0.14.0 on, ./byodt upgrade does the next upgrade. On Memgraph with podman-compose, the
graph backup of an upgrade from 0.13.0 is taken by 0.13.0's own byodt, and it fails there: take
the snapshot by hand and add --skip-graph-backup to the installer command above (see Operations,
Upgrading on Podman). Signatures: without a terminal, an upgrade needs --yes. Without
cosign, an upgrade given --yes or run without a terminal refuses the release unless you add
--allow-unsigned; at a terminal, without --yes, it asks first. A signature cosign rejects always
stops the upgrade. Install cosign to have the signature checked. To make a missing signature stop
every upgrade, --allow-unsigned included, set UPGRADE_REQUIRE_SIGNATURE=true in .env.

Operators: a proxy that reached the containers from your shell under Podman, or from the Docker
client's ~/.docker/config.json, no longer reaches them; if the stack needs one, set HTTPS_PROXY
in .env (see Changed). The JWT guard no longer builds an HTTPS agent of its own or loads a
root.crt from beside the file NODE_EXTRA_CA_CERTS names (see Changed). If that variable named
your identity provider's server certificate and its private CA was in the root.crt beside it, the
guard no longer trusts that CA: point NODE_EXTRA_CA_CERTS at the CA itself. In the compose bundle,
add the CA to proxy-ca/proxy-ca.pem, beside any proxy CA already in that file; console-init, the
platform and the console then trust it for all their outbound HTTPS (see Added, the proxy CA
entry). A NODE_EXTRA_CA_CERTS that already named the CA itself is unaffected.

TEMPLATE_OPERATION_TIMEOUT_MS: the compose bundle does not pass it to the platform, so this
concerns dt-ws run outside the bundle, or given the variable through compose.override.yaml.
0.13.0 checked its range but did not use it. dt-ws now waits that long for a module's template or
guide, and raises a value below 20000 to 20000 (see Fixed). Before upgrading, check that the
value is how long you want dt-ws to wait, or remove it to get the default, 30000.

Moved folders: if you moved a deployment's folder to a new path, run ./byodt down --take-over
once in the new folder before you start it: byodt now refuses to act on another folder's
deployment (see Changed). Memgraph: ./byodt up and ./byodt update, the upgrade's own apply
included, set a password of the generated shape again to move it to the faster hash; with the
generated password there is nothing to do (see Fixed). A password of another shape (anything but
lowercase hex of 32 characters or more) keeps the slower hash until you set it again in the
database (see How Memgraph stores it in the BYODt configuration guide).

Module authors: @dethernety/dt-module is 0.14.0. A module whose manifest declares
compatibility.dtModule must declare ^0.14.0 to be packaged against it (see Changed).
dethernety-threat-report is 2.1.0. API callers: Data.controls and Control.supportedData
are new (see Added). Dethereal users: Dethereal 0.4.19 needs this platform: it checks for
Data.controls and stops on an older platform (see Changed). Upgrade the platform first.
Dethereal is published separately from this release.

Added

• REMOTE_MODULE_CALL_TIMEOUT_MS in @dethernety/dt-module (15000): how long a remote module
  waits for one call to its content service before it gives up and serves its read-only fallback. A
  host that limits how long it waits for a remote module's template or guide must allow longer.
• A remote module logs the read-only fallbacks it serves. The warning Serving the read-only fallback names the module, the class, whether it was the template or the guide, and the reason:
  no-credential, denied or unavailable (a call that timed out is unavailable). It is logged
  at most once a minute for each reason, with the count of fallbacks not logged since the last
  warning.
• The compose bundle can reach the internet through an outbound HTTPS proxy. Set
  HTTPS_PROXY=http://host:port in .env, and NO_PROXY for extra hosts to reach directly. Only
  console-init, the platform and the console get the proxy; the database and the front door get the
  proxy variables set empty. The platform's Node uses the proxy (NODE_USE_ENV_PROXY=1) only when
  one is set, for fetch and the core HTTPS agent alike.
• ./byodt reads the proxy from .env only, and checks it. A different HTTPS_PROXY in your
  shell gets a notice and is not used. HTTP_PROXY is not used, and ./byodt warns when .env
  sets it. ./byodt refuses a proxy URL that carries a login, has no numeric port, names a loopback
  address, or names one of the stack's own services.
• A proxy login goes in .env.secrets. Set HTTPS_PROXY_USERINFO=user:password, with the user
  name and the password percent-encoded, and any character outside ASCII as its UTF-8 bytes. Over
  http:// the login crosses the network in clear; use https:// if the proxy offers it. It is
  checked strictly too, and ./byodt masks it in its messages. docker inspect, podman inspect
  and ./byodt compose config show it, as they show the database password, and under
  podman-compose so does the host's process list while containers are created. console-init, the
  console and the platform mask it as ://***@ only where they report a failed fetch: a module
  download, a call to the cloud content service or to the platform, the sign-in keys and the model
  download. Other error text is not masked. In tests, Node's own proxy errors left the login out,
  and no service log carried it.
• NO_PROXY takes exact host names or addresses. Each may have an optional :port, and they
  are separated by commas; CIDR, ranges, wildcards and a leading dot are refused. Do not rely on an
  entry to cover its subdomains: the platform's core HTTPS agent matches the exact name only, so
  list every host, login.example.com and not example.com. ./byodt adds loopback and the
  stack's own service names, including any service in compose.override.yaml.
• A TLS-inspecting proxy's CA goes in proxy-ca/proxy-ca.pem. It must be the only file in
  proxy-ca/: owned by you, readable by every account (chmod 0644), not writable by group or
  others, and not a symbolic link. After replacing it, run ./byodt restart platform console.
  Trusting that CA lets whoever holds its key read and change everything the stack sends over TLS,
  including the signing keys sign-ins are checked against, so it can sign anyone in. Prefer asking
  for the identity provider and the content service to pass through the proxy uninspected.
• Image pulls use the Docker daemon's own proxy setting, not .env. ./byodt up and
  ./byodt update warn before they pull when .env sets a proxy and the daemon has none. A failed
  embedding-model download names the host that failed, never the path or query of its URL.
• Setup warns about a database volume an earlier deployment left. When
  dethernety_memgraph-data or dethernety_neo4j-data, a volume an earlier deployment of the
  bundle left, exists, setup names it and says whether this deployment would mount it or not. If
  it would, its first start would be refused; setting DB_DATA or NEO4J_DATA to a folder avoids
  that. If not, the old graph may still be in it: start the deployment it belongs to again, with
  this one stopped, to use it. The warning gives the volume rm command to run if that graph is
  no longer needed. Nothing is removed for you.
• Each release publishes an installer, get-byodt.sh, signed like the bundle. Run with
  curl --proto '=https' --tlsv1.2 -fsSL https://github.com/dether-net/dethernety-oss/releases/download/vX.Y.Z/get-byodt.sh | sh,
  it downloads that release's bundle over HTTPS (at most 50 MiB), checks the SHA-256 the release
  wrote into the installer and, with cosign installed, the release workflow's signature at that
  exact tag, unpacks it into a new folder (./byodt, or --dir), and runs ./byodt setup. It
  refuses an existing folder, never installs over a deployment, and upgrades one with --upgrade.
  Options: --dir, --upgrade, --yes, --https-proxy, --tarball, --require-signature,
  --allow-unsigned; any other option goes to setup or upgrade. Without a terminal it needs
  --yes. A proxy login comes only from HTTPS_PROXY or https_proxy in the environment:
  --https-proxy refuses one.
• An npm package that wraps the same installer, @dether.net/byodt. Each package version is
  the release of the same version and carries its get-byodt.sh, which it runs with /bin/sh
  (npx <package>@X.Y.Z). It has no dependencies and no install scripts, and stops on Windows (use
  WSL). Publishing it to npm is switched on separately; until then, and after, get-byodt.sh from
  the release is the primary way to install.
• ./byodt setup, the first run of a bundle. It asks for the database, the data storage (on
  Docker Desktop or a Podman machine), the bind address, the port, the outbound proxy (its hosts,
  login and CA), and whether to start, and writes .env (and .env.secrets only for a proxy
  login). Every question has a flag; --yes takes the defaults. Without a terminal and without
  --yes it refuses, and a refusal leaves the folder as it was.
• A third-party notice before the first download. byodt and the installer show which
  third-party software the deployment runs (the database image, nginx, the embedding model), where
  the machine downloads it from, and its licence, and ask you to confirm that you have read it and
  are responsible for using that software under its licence. It is not a licence agreement. Each
  confirmation is a line in third-party-acknowledgement.log; the notice is asked for again only
  for a new text, another database image, or another model. ./byodt notice [--all] prints it,
  ./byodt acknowledge records it. ./byodt compose … never shows it.
• ./byodt upgrade [VERSION] moves a deployment to a newer release. The new release's own
  byodt checks first (edited shipped files, the .env changes, free disk, a running database),
  asks every question at once, backs up into backups/upgrade-<from>-to-<to>-<time>/ (the graph,
  .env, the shipped files), moves the new files in, updates .env, applies the release, waits
  until it is healthy, and prints the rollback steps. An interrupted upgrade resumes with
  ./byodt upgrade, or is dropped with --abandon; one interrupted before it changed anything
  starts over, with the options it was given and any given now. Options: --yes, --move-db,
  --replace-edited, --skip-graph-backup, --require-signature, --allow-unsigned,
  --health-timeout. A signature cosign rejects always stops the upgrade. Without cosign, an
  upgrade given --yes or run without a terminal refuses the release unless --allow-unsigned is
  given, and --yes never implies it; at a terminal, without --yes, it asks first.
• UPGRADE_REQUIRE_SIGNATURE=true in .env makes every upgrade of the deployment require a
  verified signature. An upgrade given --allow-unsigned is refused before anything is
  downloaded. false, in any case, or an empty value leaves it to the flags; any other value, such
  as yes or 1, stops the upgrade before anything is downloaded. An upgrade by an account that
  cannot read .env stops.
• .byodt-release records the release a deployment runs and the SHA-256 of every file it
  shipped. The next upgrade checks local edits against it.
• A control can be applied to a Data element. Data.controls and Control.supportedData read
  and write the SUPPORTS edge from a Control to a Data element. Both fields allow only a nested
  connect and disconnect; a nested create or delete is not offered.
• A data item's controls are edited in its data dialog. The dialog has a Controls tab once the
  data item exists, with the same picker as an element's Controls tab: a pick is saved at once, and
  unticking a control asks before it is removed. In control gaps, a control on a data item counts
  for that data item's own exposures only: not for the elements that handle it, nor theirs for it.
• Dethereal 0.4.19 reads and writes a data item's controls. Each item in data-items.json may
  carry controls, the same { id, name } references boundaries, components and data flows carry.
  Pull writes them, push applies them, get_model_schema and validate_model_json know the field,
  and a reference the push could not attach is written back to the item. On push a present list
  (including []) replaces the item's controls, and an absent key leaves them as they are; on a
  boundary, component or data flow an absent key clears them. To remove a control from a data
  item, write the list without it, or []; never delete the key. manage_controls assign attaches
  data items too.
• The threat report lists a Data element's own controls. In dethernety-threat-report 2.1.0,
  the off-grid Data disclosure in Coverage & Gaps names the controls applied to each Data element,
  as assigned and without a coverage grade. Data exposures stay off the coverage grid.

Changed

• HTTP_PROXY, HTTPS_PROXY and NO_PROXY from your shell under Podman, or from the Docker
  client's config.json, no longer reach the containers. Podman copies them from the environment
  it runs in into every container that does not set them (http_proxy = true in containers.conf,
  its default), and the Docker client does the same with the proxies in ~/.docker/config.json.
  So console-init and the console used to follow such a proxy. The bundle now sets these variables
  for every service, from .env or empty. If the stack needs a proxy, set HTTPS_PROXY in .env.
• The platform's JWT guard fetches the OIDC signing keys (JWKS) with fetch, so with a proxy
  set they go through it like the platform's other calls. The JWKS URI must be https://
  (http:// only on a loopback host), a redirect is refused, and the fetch fails after 30 s or once
  the answer passes 1 MiB. The GraphQL layer fetches the same keys on its own, on Node's core HTTPS
  agent, and goes through the proxy too. The guard no longer builds an HTTPS agent of its own when
  NODE_EXTRA_CA_CERTS is set, and no longer loads a root.crt from the directory that variable
  points into. If your identity provider's certificate is signed by a private CA, add that CA to
  proxy-ca/proxy-ca.pem in the compose bundle; elsewhere, point NODE_EXTRA_CA_CERTS at it.
• ./byodt now refuses to act on another folder's deployment. up, update, restart,
  down, stop, destroy, backup, restore, upgrade, and setup when it starts the stack
  (--start, or a yes at its question), refuse when the deployment's compose project (all bundles
  share the project name dethernety) is running, or ran, from another folder, and they name that
  folder. Before, up in a leftover folder took over another folder's containers, and down or
  destroy there stopped its running deployment. If you moved a deployment's folder to a new path,
  run ./byodt down --take-over (or destroy --take-over) once in the new folder before you start
  it: it removes the containers the old path started.
• The first ./byodt up refuses to mount a database volume an earlier deployment left. When the
  database would use the named volume dethernety_memgraph-data or dethernety_neo4j-data and that
  volume already exists, the first up of a folder stops, and so does setup when it starts the
  stack: that graph's database users were made with another password, so the platform would be
  locked out. Setting DB_DATA or NEO4J_DATA to a folder avoids it (--db-data bind in setup);
  the message gives the volume rm command for a graph you no longer need.
• ./byodt upgrade is a command of its own. It was another name for update. Release 0.13.0's
  byodt keeps the old meaning: upgrade a 0.13.0 deployment with the new release's installer and
  --upgrade.
• The first start of a bundle shows the third-party notice and waits for a confirmation.
  ./byodt up, update, restart and models pull on a deployment that has not run yet stop
  without a terminal unless --yes is given. On a deployment that has run, a due notice is asked
  for at a terminal, where n stops the command; with no terminal and no --yes, it is printed
  with a warning and never stops a command.
• byodt names itself in its messages the way it was run, ./byodt up rather than byodt up,
  so a suggested command works as printed.
• byodt records the notice only in a folder no other account can change. A group-writable
  folder counts as one other accounts can change, unless the group is the operator's own private
  group.
• Dethereal 0.4.19 needs a platform with controls on data elements, which this release adds.
  Before export_model, import_model, update_model, create_threat_model,
  update_attributes, and the list, assign and push-greenfield actions of manage_controls,
  it checks that the platform has Data.controls, and remembers a platform that has it. On an
  older platform those calls stop with "Platform too old for this Dethereal version" and a message
  that names the platform. Upgrade the platform first.
• @dethernety/dt-module is 0.14.0, and its one API change is additive: the new export
  REMOTE_MODULE_CALL_TIMEOUT_MS (see Added). The changes in behaviour are in DtRemoteModule:
  once registered, getMetadata answers from the entry it registered (see Fixed), a served
  read-only fallback is logged (see Added), and an error from the content service masks the login
  of a URL in its message as ://***@. Declaring compatibility.dtModule in manifest.json is
  optional; a module that declares it must declare "compatibility": { "dtModule": "^0.14.0" } to
  be packaged against dt-module 0.14.0. For a 0.x version a caret range pins the minor version, so
  scripts/package-module.js refuses a manifest that still declares ^0.13.0 against dt-module
  0.14.0, and produces no package. dethernety-general, dethernety-coverage-tools and
  dethernety-threat-report declare ^0.14.0.

Fixed

• A slow remote module's class template or guide now shows the module's read-only fallback.
  dt-ws stopped waiting for a template or guide after 10 s, before a remote module's own 15 s call
  timeout let it answer with its fallback, so the settings window showed no attribute form, only
  the warning "Form could not be rendered due to missing or invalid schema". dt-ws now reads
  TEMPLATE_OPERATION_TIMEOUT_MS (default 30000, for local and remote modules alike), which was
  documented but never read. A value outside 5000 to 300000 stops dt-ws at boot, and one below 20000
  is raised to 20000. The template and guide failure logs now name the module and carry timedOut,
  true when dt-ws stopped waiting.
• The settings window no longer shows or saves the previous element's attributes while a new one
  loads. After a change of selection, the Attributes tab kept the previous element's form until
  the new element's class arrived, and an attribute save in that time sent the previous element's
  class for the new element. The tab now shows a progress bar until the new class has loaded, saves
  no attributes before then, and ignores a class or attributes answer that arrives for an earlier
  selection. The class picker is disabled until the class has loaded.
• A save while the settings window loads no longer writes empty fields or the previous
  element's links. Until the element's class arrived, the window held empty name and description
  fields and the previous element's controls and zoning, so a crown-jewel toggle, an added control
  or a zoning save in that time wrote them to the new element. It now fills them from the element at
  once and keeps an edit made while the class loads. A save of the General tab no longer drops an
  unsaved zoning edit before the zoning save that follows it.
• A slow content service no longer takes a registered remote module out of service. The detailed
  health check (GET /health) gives each module's getMetadata 5 s. A DtRemoteModule
  (@dethernety/dt-module) whose portal origin was not pinned at boot, or whose pin was not cached,
  called its content service on every getMetadata call, with a 15 s timeout, so a slow service
  failed the check. The platform then stopped using the module (its templates, guides, evaluation
  and frontend) until a later check passed. Once registered, a remote module now answers
  getMetadata at once from the entry it registered, and pins the portal origin and fetches the
  document at its pin in the background, at most once a minute; a failed document fetch is logged
  without server text. Concurrent calls share one registration, a failed registration is answered
  again for a minute without the network, and a live module document with no name is refused before
  it is cached. A module registered from another pin during an outage keeps serving that entry until
  the background fetch of its own pin succeeds; the next install after that, at the latest the next
  start, installs its own pin's classes.
• "Commit & change" no longer removes an element's class instead of changing it. With unsaved
  General edits, picking a new class or model and choosing "Commit & change" saved the edits first,
  and the form reload that save started cleared the pick before the change ran: a class change
  became a class removal, and a model pick was dropped. The pick is now taken before the edits are
  saved, and the change is cancelled, with a message, if another element was selected or a new edit
  was made while they were saved. Cancelling a class or model change now puts back only the pick,
  instead of also discarding the other unsaved edits.
• Saving General and Attributes together no longer shows old attribute values. The General
  save's form reload ran during the attribute save and read the attributes before that save had
  committed, so the form showed the old values over the new baseline and the next save sent them
  back. The settings window now reloads the form once, after all of its saves, and only when no edit
  is left unsaved. A failed save, or an edit made while a save was in flight, keeps its edit, as
  does an unsaved attribute edit when a link is added or the crown jewel is toggled.
• Database logins on Memgraph are fast, so a burst of new connections no longer stalls the
  database. Memgraph's default password hash, bcrypt, holds a lock for each login. The compose
  bundle now starts Memgraph with --password-encryption-algorithm=sha256-multiple. ./byodt up
  and ./byodt update set the database password again to the value it already has, when it has
  the generated shape, so an existing deployment moves to the new hash too; .env.secrets is
  unchanged, and a failure only warns. Neo4j is unaffected. If you set your own password, use a
  long random value, which the faster hash relies on: on Memgraph, byodt warns when it is
  shorter than 32 characters. See How Memgraph stores it in the BYODt configuration guide.
• ./byodt backup and ./byodt restore now work on Podman with podman-compose for a Memgraph
  database. They copied the graph snapshot with compose cp, which podman-compose does not have,
  so they failed with "Failed to copy the snapshot out of the db container." (backup) or "Could not
  copy the snapshot into the db container." (restore). An upgrade backs up the graph first, so an
  upgrade on such a deployment stopped there, with nothing changed. byodt now copies the snapshot
  with the container engine's own cp (docker cp or podman cp) on the project's database
  container, which both engines have. Neo4j deployments were not affected. The snapshot a manual
  ./byodt backup writes is now readable by its owner only (it kept the container file's mode
  before). An upgrade from 0.13.0 on Podman still runs 0.13.0's own backup and fails there; take a
  snapshot of the graph by hand with podman cp and upgrade with --skip-graph-backup.
• getControlsAssignedModels now counts a control applied to a Data element or to a whole
  model. It reached a Model only through the components, boundaries and data flows the control
  is applied to, so a control applied only to a Data element or to a Model read as assigned to no
  model. It now also reaches the Model that contains the Data element, and the Model the control
  is applied to directly.

Security

• The image workflow no longer leaves its token in the checkout. Every job in docker.yml
  checks out with persist-credentials: false, so the GITHUB_TOKEN of a job that can push images
  is no longer stored in .git/config, where any code the job runs, a dependency's install script
  included, could read it.
• The verify command in the image workflow's summary names the exact signer. It printed
  --certificate-identity-regexp '^https://github.com/<repository>/', which a signature from any
  workflow run of the repository would pass. It now prints --certificate-identity with the
  workflow file and the ref of the run that signed the image.