Skip to content

v0.6.0 — Credentials out of .env, with ergosum

Latest

Choose a tag to compare

@axeldunkel axeldunkel released this 10 Nov 22:04
· 9 commits to main since this release
Immutable release. Only release title and notes can be modified.
c0a209f

ToolMesh 0.6.0 takes the credentials out of .env. The new credential store command runs a command such as sops -d or vault kv get and keeps what it prints in memory only. The credentials no longer appear in .env, in the process environment that Docker writes into the container configuration on disk, or in the log. With ergosum, the key that opens them is released only on the machine it is bound to, at its place. A copied disk, VM image or backup therefore holds no usable secret either.

This release has a security focus throughout. Every REST backend is now bound to its own credentials and hosts, and a series of security issues is fixed. Details are in the security advisory GHSA-7ppc-qvgq-3rgf. Thanks to Krynex Labs for reporting the issues behind it and for reviewing the fixes.

Upgrading from 0.5.x: read the behavior changes below first. Some backends may be refused at load until their entry in backends.yaml gets one more line, and the login page now asks for a confirmation.

Credentials out of .env

  • Credential store command (CREDENTIAL_STORE=command). ToolMesh runs TOOLMESH_CREDENTIAL_COMMAND without a shell and reads its output as dotenv or JSON. TOOLMESH_CREDENTIAL_JSON_PATH selects the object, for example data.data for vault kv get -format=json. The values live in memory only, and the .env holds only the store settings. TOOLMESH_AUTH_PASSWORD, TOOLMESH_API_KEY, REDIS_URL and env: remappings come from the output too.
  • ergosum as the anchor.
    • With sops, ergosum releases the age key that decrypts the credential file only on the bound machine (ergosumd secret get, ergosum v0.2.0 or later).
    • With Vault or OpenBao, ergosum gives the machine a short-lived JWT for its login. No AppRole secret ID or other long-lived secret lies on the disk.
  • Short-lived tokens. TOOLMESH_CREDENTIAL_RELOAD_IN, or a RELOAD_IN line in the output, runs the command again before a token expires. A failed refresh keeps the previous values and retries with backoff.
  • stdio backends receive only the variables their new credential_env entry in backends.yaml names, instead of the whole environment.
  • Monitoring: toolmesh_credential_store_runs_total{result} and toolmesh_credential_store_last_success_timestamp_seconds follow the runs.

The embedded store, which reads CREDENTIAL_* from the environment, works as before.

Documentation: Credential stores, with examples for sops, Vault/OpenBao and short-lived tokens.

Credential and host binding for DADL backends

  • Credentials per backend. A backend reads only the credentials of its entry's namespace CREDENTIAL_<ENTRY>_*. A credential name belongs to the configured entry with the longest matching namespace, so github does not read the credentials of github-enterprise. The entry's env: mappings and the new credentials: list add further names. This holds for both credential stores.
  • Hosts per backend. Credentials, OAuth token requests and session logins go only to the base_url host and port, other hosts of its registrable domain on the default port, and the hosts in the entry's new allowed_hosts: list (host, *.domain, optionally :port). An https base_url stays on https. Session cookies follow the same binding.
  • Checked at load. A backend whose DADL reaches further is refused when it is loaded, at startup and on reload, and the log line backend refused names the exact line to add. The other backends keep running. ToolMesh checks again before every request that carries a credential.
  • Stricter DADL parsing. Credential names consist of letters, digits and _; parameter defaults of the form credential:<name> and placeholders in the scheme, host or port of an absolute tool path are refused.
  • Trust. A DADL is code you install: it decides which requests its backend sends with that backend's credentials. ToolMesh keeps the credentials of different backends apart; the host binding is a second line of defense and does not make a malicious DADL safe. Load DADLs only from sources you trust, as you would a program. On shared SaaS domains such as atlassian.net or zendesk.com, other customers' tenants count as the same registrable domain.

Documentation: Configuration and DADL.

Behavior changes: read before upgrading

  • Backends can be refused at load until their entry has the lines the log names (see above). REST entries need a name. After upgrading, search the log for backend refused.
  • The OAuth login page names the client and the redirect host and requires a confirmation tick. Scripted logins must send confirm_host=<redirect host>. TOOLMESH_OAUTH_REDIRECT_HOSTS optionally restricts the redirect hosts, for example claude.ai,claude.com. Redirect URIs with a fragment, without a host or with a non-ASCII host are refused.
  • Tokens follow the current user configuration. Refresh and every access token are checked against the users as loaded at startup: after a restart, removed users lose access at once, including running sessions, and changed roles and plans apply to existing tokens. A password change ends the user's sessions. POST /revoke (RFC 7009) revokes a token pair. Changes to users.yaml still take effect only after a restart.
  • Tokens from before the upgrade carry no password fingerprint yet. They are bound at their next refresh, so a password change made after the upgrade does not end them. Revoke such tokens, or clear the stored tokens while ToolMesh is stopped, as described in Tokens from before the upgrade.
  • X-Forwarded-For is honored only from TOOLMESH_TRUSTED_PROXIES, by default loopback only. Behind a reverse proxy in another container, on another machine or behind a CDN, list the proxy's address; until then all clients share the proxy's address for the login, registration and API key limits, and ToolMesh warns about it. For a proxy container, give it a fixed address in the Compose network and list that address alone, not the network as the example in .env.example and docs/configuration.md suggests: the network's gateway belongs to it, and with Docker Desktop, rootless Docker or a userland proxy, direct connections to ToolMesh's published port come from that gateway and could then choose their address. Once a proxy container stands in front of ToolMesh, publish ToolMesh's port on 127.0.0.1 only, or not at all.
  • Failed API keys are throttled per client address: after 50 failures within the login window (TOOLMESH_APIKEY_MAX_FAILURES_PER_IP), further keys from that address get 429 for the rest of the window, a correct key included.
  • DADL arguments are checked against their declared types before the policies run. "true" or "false" as strings for a boolean, fractions or non-numeric strings for an integer and scalars for arrays or objects are refused with invalid_input. Numeric strings for numeric parameters are sent as numbers ("007" becomes 7), and policies see DADL defaults filled in.
  • Requests stay where the configuration points.
    • Redirects to another origin drop credentials, custom headers, the Referer and session cookies, never go from https to http, and do not re-send a request body.
    • OAuth token and session login requests follow no redirects.
    • Path parameter values . and .. are refused.
    • Transport errors name only scheme and host.
    • MCP-over-HTTP backends stay on the origin of their configured URL: a redirect or a legacy SSE endpoint pointing elsewhere is refused.
  • Private addresses.
    • With allow_private_url, private addresses are reachable only for the destinations the configuration names (base_url, token_url, absolute tool paths), not for redirect targets.
    • Without it, carrier-grade NAT (including Tailscale addresses), benchmarking, documentation, multicast, reserved, NAT64 and 6to4 ranges count as private too.
    • IPv6 addresses with a zone suffix and the unspecified address are refused.
  • Blobs can only be deleted by the user who created them.
  • execute_code runs at most TOOLMESH_CODE_MAX_CONCURRENT scripts at once (default 16), and one user at most half of them. Further runs wait up to 10 seconds, then fail.
  • Without Redis, /register is limited per client address.
  • TOOLMESH_ISSUER defaults to https://toolmesh.invalid/ when unset; set it to the public URL clients use.
  • Only the statuses in a DADL's retry_on are retried, as DADL spec §8 requires. Until now every 5xx was retried, listed or not. Most published DADLs list [429, 502, 503, 504], so a 500 is no longer retried; a DADL that wants it retried lists it in retry_on.
  • Backends with an underscore in their name keep it in audit, telemetry and metrics. web_search_query used to be recorded as backend web, tool search_query; dashboards and alerts that select such a backend by the shortened backend label need the full name.
  • Arguments to MCP backends are checked against the tool's inputSchema. An undeclared argument or a missing required one is refused with invalid_input, naming the closest declared parameter, where an MCP backend used to drop it silently or answer with its own error. A schema that opens its property set on purpose keeps accepting undeclared names.

Upgrade checklist

  1. Start 0.6.0, search the log for backend refused and add the named credentials: or allowed_hosts: lines to the entries in backends.yaml.
  2. Behind a reverse proxy that is not on loopback, set TOOLMESH_TRUSTED_PROXIES to its own address, not to the network it shares with ToolMesh.
  3. Adapt scripted logins to send confirm_host; consider TOOLMESH_OAUTH_REDIRECT_HOSTS.
  4. If you suspect a session was misused, revoke its tokens or clear the stored tokens as described in Tokens from before the upgrade, then change the password.
  5. Check policies that compare DADL parameters with strings such as "false"; they now see the declared type.
  6. Check scripts and agents that pass MCP backends arguments the tool's schema does not declare; they now get invalid_input.

The checklist and all behavior changes are also on the website: Upgrade notes.

Other changes

  • DADL health checks. A DADL can declare under backend.health the cheapest call that proves the API and the credential work. ToolMesh runs it at startup, after a reload and every 15 minutes, reports failing backends and credentials close to expiry on /health, offers the check as <backend>_health, and exports three gauges for alerting.
  • execute_code metrics: toolmesh_codemode_swallowed_errors_total and toolmesh_codemode_scripts_total show how often scripts finish although a tool call in them failed.
  • The file broker can be switched off and bounded: TOOLMESH_FILE_BROKER_ENABLED, TOOLMESH_FILE_BROKER_MAX_TOTAL_BYTES, TOOLMESH_FILE_BROKER_MAX_BYTES_PER_USER.
  • OAuth clients registered at /register expire (TOOLMESH_DCR_CLIENT_TTL, TOOLMESH_DCR_CLIENT_IDLE_TTL).
  • execute_code sandbox: atob, btoa, TextEncoder and TextDecoder (UTF-8), also in composites and unit backends. A failed call's entry in the result listing shows the arguments the script sent, with secret-looking values redacted.
  • The approval page leads with what an approver decides on, comes in English, German, French, Spanish and Japanese, and lets an approver grant a kit for longer than requested, up to the kit's maximum.

Fixes

  • tell_me_my_rights reports the built-in tools (execute_code, discover_tools, upload_file and others) as allowed instead of denied, and a name no tool has as an unknown tool.
  • Tool calls to a backend whose name contains an underscore, such as web_search, are attributed to that backend in the audit trail, in telemetry and in the metrics.
  • The MCP registry entry declares the Authorization header, so registry listings no longer present ToolMesh as needing no authentication.
  • A new installation sends its first telemetry report a few minutes after it starts instead of after 24 hours, so installations that restart more often than once a day report at all; restarts add no reports.
  • The warnings about a failing response transform, a pagination error and a failed file broker upload name the backend and tool they concern.

Security

  • Backend credentials are bound to their backend entry and its hosts (see above).
  • Composite tool calls keep their authorization and pre-execution policies after a reload of backends.yaml.
  • The OAuth login page names the redirect host and requires a confirmation; tokens are checked against the current users and can be revoked.
  • Credentials no longer follow redirects to other hosts; path parameters cannot leave the configured base path; API keys no longer appear in error texts.
  • X-Forwarded-For is honored only from trusted proxies.
  • SSRF checks refuse IPv6 addresses with a zone suffix (such as fe80::1%eth0), which passed the private-address checks, and the unspecified address (0.0.0.0, [::]) as a base URL with allow_private_url: false.
  • MCP-over-HTTP backends send their token only to the configured endpoint origin.
  • Hardening: special-use address ranges, parameter types before policies (also numbers and booleans for string parameters), blob deletion by the owner, registration and API key throttling without Redis, X-Forwarded-For entries with a port, file_url_allowed_hosts on redirects, refresh token expiry, the issuer placeholder, bounded execute_code concurrency.
  • A REDIS_URL that does not parse no longer puts the URL, and with it a password, into the log.

Advisory: GHSA-7ppc-qvgq-3rgf. Full details in CHANGELOG.md.

Docker image: ghcr.io/dunkelcloud/toolmesh:0.6.0, also tagged 0.6 and latest, built for linux/amd64 and linux/arm64. Image tags carry no v prefix.