Skip to content

ADR Centralize secret storage and harden admin credentials

Nabendu Maiti edited this page Nov 13, 2026 · 21 revisions

ADR: Centralize secret storage and harden admin credentials

Date: 2026-10-12

Last revised: 2026-11-13 (see Revision log)

Related issue: https://github.com/device-management-toolkit/console/issues/1067

Related architecture: ADR-Centralize-secret-storage-and-harden-admin-credentials-Architecture

Status: Proposed

Context

Standalone (non-OAuth2) Console deployments stored the administrator username, password, and JWT signing key in plaintext in config.yml or the corresponding environment variables, and compared the submitted login password with a plain ==.

This ADR covers, for deployments that do not use OAuth2/OIDC:

  • local first-run bootstrap of the administrator credentials
  • secure storage of those credentials in the OS keyring
  • password hashing
  • selection of the secret backend (config or keyring) and the plan for changing its default
  • a credential reset and recovery workflow

The JWT signing key is explicitly out of scope. This ADR does not change its existing configuration/environment handling or store it in the dmt-console keyring service; its storage and lifecycle remain as they are today.

OAuth2/OIDC deployments are unaffected: when AUTH_CLIENT_ID is configured, Console never loads or stores standalone admin credentials.

Existing deployments rely on credentials in config.yml or the environment, and some (for example Docker Compose) have no usable OS keyring. The design therefore adds the keyring as an opt-in backend in v1.x and does not change existing behavior until a coordinated major release.

Decisions

1. Authentication mode precedence

Decision: OAuth2/OIDC takes absolute precedence. When AUTH_CLIENT_ID is non-empty, Console uses the OAuth2/OIDC path and ignores all standalone credential sources and settings (keyring, config, AUTH_SECRET_BACKEND).

Rationale: Prevents silent override of configured identity providers and keeps OAuth2 deployments deterministic.

2. Secure credential storage: OS keyring

Decision: The OS keyring is the secure credential store for standalone Console. When the keyring backend is selected, the administrator credentials are stored only in the OS keyring (never in config.yml or the environment), via github.com/zalando/go-keyring, as follows:

  • Service name: dmt-console. This is separate from the existing encryption-key service (device-management-toolkit), which is unchanged and not reused.
  • Entries: two entries, admin-username (the username) and admin-password-hash (the PBKDF2 hash from Decision 3, which embeds its own salt and iteration count). No plaintext password and no other secret, such as a signing key, is stored.
  • Startup states:
    • Both entries present: Console loads them and proceeds to login verification.
    • Both entries absent: first-run setup runs (Decisions 7 and 8).
    • Only one entry present (partial or corrupted): Console fails at startup with recovery guidance and never overwrites the remaining entry. The operator resolves it by deleting the entries (Decision 6).
  • Keyring failure: if the keyring is missing, locked, access is denied, or no supported backend exists, Console fails with an actionable error. It never falls back to reading or writing plaintext credentials (Decision 5).
  • Login: the submitted password is verified against the stored hash. The stored hash and the plaintext password are never returned, logged, or displayed.

Rationale: The OS keyring is scoped to the OS user, is OS-native, and keeps credentials out of config.yml, separating secret storage from configuration files. Console already uses a keyring for its encryption key, so this adds no new class of dependency (confirm that go-keyring is already a dependency before implementation). Failing closed on partial or unavailable state avoids silently creating or overwriting administrator credentials. The username is stored in the keyring too because it is part of the credential set and then shares the same access controls.

Limitation: The keyring depends on the user session. On Linux the Secret Service API needs a D-Bus session and an unlocked keyring, which headless servers, services, and containers often lack. This is why config remains the v1.x default (Decision 4). Entries are local to the host and OS user, so they are not shared across replicas.

Supported keyrings:

  • Windows: Credential Manager
  • macOS: Keychain
  • Linux: Secret Service API (GNOME Keyring, KWallet)

Entry names, partial-state handling, and failure behavior are specified in the ADR-Centralize-secret-storage-and-harden-admin-credentials-Architecture.

3. Password hashing: PBKDF2-HMAC-SHA256

Decision: Passwords are hashed with PBKDF2-HMAC-SHA256 (600,000 iterations, 16-byte salt, 32-byte hash) before storage in the keyring. The stored representation uses the $pbkdf2-sha256$ prefix to identify the algorithm, with the salt and hash encoded using standard padded Base64. Every new password hash, including initial setup, password changes, and setup after recovery, must use a freshly generated 16-byte salt from crypto/rand. Previously stored salts must never be reused for a new hash, even when the password is unchanged. If random generation fails, the operation fails without replacing existing credentials.

Rationale: The Go standard library provides crypto/pbkdf2 (Go 1.24 or later; confirm against go.mod) so no additional cryptographic dependency is needed. PBKDF2-HMAC-SHA256 is NIST-approved and 600,000 iterations is the current OWASP minimum for it. It provides one-way, deliberately slow hashing. The stored hash is never the plaintext password.

Trade-off: PBKDF2 is not memory-hard, so it resists GPU cracking less well than bcrypt or Argon2. Accepted to avoid a new dependency; the hash format is versioned so the algorithm can change later.

4. Secret backend: AUTH_SECRET_BACKEND, default config throughout v1.x

Decision: Console supports AUTH_SECRET_BACKEND=config|keyring:

  • config (default throughout v1.x): credentials continue to come from config.yml/environment, as today.
  • keyring: opt-in. Credentials come from the OS keyring only (Decision 5).

The default changes to keyring only in a future major release (for example Console v2), after the keyring backend and the UI first-run setup are fully implemented. That change is coordinated with the installer release and accompanied by advance notice and migration guidance. Timing is decided in Engineering Sync.

Rationale:

  • Existing customers keep the behavior they rely on, with no forced change or credential migration during v1.x.
  • Docker Compose and other managed deployments with no usable OS keyring continue to work.
  • Customers can opt into the keyring while the backend and UI setup are completed.

The config backend stores credentials in plaintext. This is the existing behavior and is not made safer by this ADR. Whether config mode logs a reduced-security warning is an implementation detail; it must not be exposed through unauthenticated endpoints.

5. Keyring mode is isolated from config and environment

Decision: When AUTH_SECRET_BACKEND=keyring, Console checks the keyring only. It does not read, import, or use admin credentials from config.yml, .env, or environment variables, does not prompt to migrate them, and does not modify or scrub config.yml. There is no fallback between backends in either direction.

If the keyring is missing, locked, denied, or unsupported, Console fails with an actionable error. If the required keyring entries are absent, first-run setup runs (Decisions 7 and 8).

Rationale: Removing fallback and migration logic reduces code paths and makes behavior predictable. Existing credentials are left untouched, and no operation modifies customer configuration without the customer's action. This supersedes the earlier design that migrated legacy plaintext values after operator confirmation (-confirm-legacy-migration), which is not implemented.

Consequence: An operator who opts into the keyring must set up credentials through first-run setup and should remove the old plaintext values from config.yml themselves. Console does not do this.

6. Credential recovery: manual keyring deletion

Decision: To reset credentials and re-run first-run bootstrap, operators must manually delete both keyring entries (admin-username and admin-password-hash) from the OS keyring using OS-native tools.

For Phase 2 authenticated standalone password changes, Console must revoke server-side all previously issued JWTs authenticated with the old credentials, including the current session. After a successful change, those tokens must be rejected even if unexpired, and the operator must sign in with the new password. Console must not report success until the password update and revocation are effective. If either fails, report the failure and fail closed; do not silently leave old tokens usable with the changed password. JWT signing key storage and OAuth2/OIDC token lifecycle remain unchanged.

Rationale: Avoids Console-managed recovery tooling (such as console -clean), which would add security responsibilities; it can be revisited if customers ask for it. Phase 2 UI provides authenticated password change for operators who know the current password. Exact per-OS commands are in the Architecture document.

7. Phase 1 first-run setup: command-line input (temporary)

Decision: Phase 1 (before Phase 2 UI) uses a simple CLI-based setup for customers who opt into keyring: on first startup with no keyring entries, Console prompts the operator to enter a username and password via interactive CLI input. Console validates the input, hashes the password with PBKDF2, and stores both in the keyring. This temporary path is documented for customers who opt in before the UI is available. It is removed in Phase 2 (Decision 8).

Rationale: Gets first-run initialization working without waiting for the Phase 2 UI. Minimal implementation surface. The operator chooses their own credentials.

8. Phase 2 first-run setup: unified bootstrap service

Decision: Phase 2 first-run setup uses a single bootstrap service shared by Console in headless mode and Console with the UI, not two independent code paths. Entering the first-time username and password at the command line (Decision 7) is removed in Phase 2; first-run setup is no longer an interactive prompt on startup.

  • Shared service and API: Both modes use the same loopback setup REST API, a thin wrapper over the bootstrap service. Console with the UI checks setup status and submits the chosen username and password over HTTPS from the setup form, then returns to sign-in after successful setup. In headless mode there is no UI, so the operator, on the Console host, calls the same setup API directly (for example with curl). Only the Console backend process reaches the keyring; neither the UI nor the API client does.
  • Temporary UI bootstrap token: When no credentials are present, Console prints the complete local setup URI to its CLI, with a short-lived, single-use token in the URL fragment, for example https://localhost:<port>/setup#token=<setup-token>. When Console opens a browser locally, it opens this same URI; otherwise the operator opens the printed URI. The setup page reads the token into memory and immediately removes the fragment using history.replaceState. The operator enters only the new username, password, and password confirmation. The UI submits the credentials and token together in the HTTPS request body. The backend validates the token and credentials before storing the username and password hash. There is no separate token-to-cookie step. On success, the token is invalidated, the UI shows confirmation, clears setup state, and returns to sign-in with the user logged out. On failure, the UI shows an error without reporting setup success. The token is not an authenticated user session and is never written to application logs or included in URL query parameters. Headless setup continues to use the loopback API directly.
  • Credential handling in headless mode: The password must not be passed on the curl command line, where it is visible in the process list and shell history. Document supplying the request body from a file or standard input (for example curl --data @-) and never logging the request body.
  • Loopback-only exposure: The setup endpoint is served only on a loopback listener (127.0.0.1, ::1), never on the public listener and never through a reverse proxy. The restriction comes from which listener serves the route, not from client IP or forwarded headers, because a reverse proxy on the same host also connects from loopback.
  • Serialization: The service takes a lock that works across processes, rechecks the keyring state under that lock before writing, and reports success only after the write is durable. Keyring errors are reported as setup failures; there is no plaintext fallback.
  • Error contract: If a non-loopback listener is enabled, setup requests reaching it are rejected with 403 SETUP_LOCAL_ONLY (the route is not served there). A local setup request after initialization receives 409 SETUP_ALREADY_COMPLETED.
  • After initialization: The sign-in page replaces setup. Setup is reachable again only after an authorized credential reset or manual keyring deletion (Decision 6).
  • Partial keyring state: Same as Decision 2. Console does not overwrite the remaining entry and gives recovery guidance.
  • API specification: The setup API is specified in OpenAPI.

Rationale: A single implementation avoids code duplication and keeps behavior consistent between headless mode and the UI, and giving headless deployments the same setup path as the UI without an interactive prompt. Binding to loopback prevents remote unauthenticated users from claiming the first admin account. The lock and recheck prevent concurrent setup from racing, and a durable-write check avoids reporting success for credentials that were not stored.

Security limitation: The CLI-displayed token authorizes initial local browser setup. It does not authenticate a local user or process that can access the CLI output and loopback listener. The complete setup URI is a secret: terminal capture, browser launch arguments, browser history, or extensions may expose it. The fragment avoids sending the token in the initial HTTP request; it does not eliminate these local exposure risks.

Open point: Whether the username and hash become a single versioned keyring record in Phase 2 remains to be validated.

Sequence diagrams (before and after initialization) and the remote-browser cases are in the Architecture document.

Consequences

Phases

  • Existing customers see no change in v1.x unless they opt into AUTH_SECRET_BACKEND=keyring.
  • No plaintext-to-keyring migration exists. Plaintext values in config.yml are never read, copied, or scrubbed by keyring mode.
  • The stored hash format is $pbkdf2-sha256$<iterations>$<base64(salt)>$<base64(hash)>.
  • Phase 1 first-run setup prompts the operator for a username and password on the command line on first startup (no auto-generation).
  • Phase 2 removes the command-line prompt. First-run setup is reachable through a loopback-only REST API, used by the UI or, in headless mode, called directly by the operator (for example with curl).
  • Username and password hash are the only credentials stored in the keyring.
  • Local keyring entries are not shared across replicas.
  • The default backend change in v2 is a breaking change that needs advance notice and migration guidance.
  • In config mode (the v1.x default), credentials remain in plaintext.

Alternatives considered

  • Keyring as the default in v1.x with automatic or confirmed migration. Rejected: forces a change on existing customers and breaks Docker Compose.
  • bcrypt or Argon2 for password hashing. Stronger against GPU cracking but adds a dependency; PBKDF2 was chosen (Decision 3).
  • Console-managed reset (console -clean). Rejected for now (Decision 6).
  • Dedicated secret-file provider for Docker/Kubernetes. Deferred; config remains the backend for these deployments until it ships.

Phases

Phase 1 (planned): Backend credential storage, PBKDF2 generation and verification, command-line first-run setup (temporary), and standalone login verification. No browser-based credential screens.

Phase 2 (planned, design specified): First-run setup form, standalone sign-in form, authenticated password-change form. Uses the Phase 1 keyring and PBKDF2 backend, and replaces the Phase 1 command-line first-run prompt with the bootstrap service (Decision 8). Detailed design is in the Architecture document.

Open items

These are not decided by this ADR and need a decision before or during implementation:

  • Phase 1 first-run setup when unattended. The Phase 1 command-line prompt does not work for Console running as a service with no terminal. Define the behavior (for example fail with guidance) until Phase 2 provides the setup API.
  • Single versioned keyring record. Whether Phase 2 replaces the two entries with one record to avoid partial-write states.

Issue 1067 Coverage

  1. Support secure storage and retrieval of Console credentials for deployments without OAuth2. Opt-in OS keyring backend, designed. Planned (Phase 1).
  2. Provide a way to disable plaintext storage of sensitive credentials in config/config.yml. With AUTH_SECRET_BACKEND=keyring, Console does not read admin credentials from config.yml; operators remove the old values themselves. Plaintext remains the v1.x default. Partially addressed.
  3. Provide a customer-usable workflow for setting, updating, and removing stored credentials. Phase 2 first-run setup and authenticated password change are designed. Phase 1 recovery requires manual OS keyring deletion. The UI workflow is a design target; implementation is future work. Partially addressed.
  4. Support the deployment environments currently supported by Console. Windows, macOS, and Linux via OS keyring are covered. Docker/Kubernetes managed deployments keep using the config backend; a dedicated secret-file provider is deferred. Partially addressed.

Revision log

  • 2026-10-12: Initial ADR.
  • 2026-10-21: Added AUTH_SECRET_BACKEND, confirmed legacy migration, and the Phase 2 bootstrap service design.
  • 2026-10-26: Dropped console -clean; manual keyring deletion is the recovery path.
  • 2026-10-27: Accepted Ganesh's 10/26 proposal: config is the v1.x default, keyring mode is isolated, and the legacy migration (Decision 5) was removed. Decision numbering is unchanged; the old "Legacy plaintext migration" decision is replaced by "Keyring mode is isolated".
  • 2026-11-13: Added a complete local setup URI printed to the CLI, carrying a short-lived token in its fragment and used when opening the local browser. The UI submits the token with the new credentials. Success returns the user to sign-in, logged out; failure shows an error. Loopback-only exposure remains required.
  • 2026-11-13: Adopted standard padded Base64 for the stored salt and hash fields in response to Divneil's review feedback.
  • 2026-11-13: Required a fresh crypto/rand salt for every new password hash, with no reuse and explicit failure on random-generation errors.
  • 2026-11-13: Required server-side revocation of existing standalone JWTs on authenticated password changes.

Review feedback resolution

  1. Migration consent (Ganesh 10/14 #1). Superseded. Migration was first gated on explicit confirmation, then removed entirely by the 10/26 proposal (Decision 5). Nothing is migrated, so nothing is migrated without consent.
  2. UI setup delivered with the backend (Ganesh 10/14 #2). Not adopted. Phase 2 UI remains deferred; the command-line path in Decision 7 covers opt-in customers until then.
  3. Docker Compose / managed deployments (Ganesh 10/14 #3). Adopted. AUTH_SECRET_BACKEND lets managed deployments keep using config; with the 10/26 change that is also the default.
  4. Drop console -clean (Ganesh 10/14 #4). Adopted (Decision 6).
  5. Headless vs. UI bootstrap detail (Sudhir 10/15 #1). Adopted (Decision 8). The single-versioned-record refinement remains open (see Open items); browser setup reads the token from the CLI-displayed setup URI and submits it with the new credentials as specified in Decision 8.
  6. Default backend and keyring isolation (Ganesh 10/26). Adopted (Decisions 4 and 5). Point 3, coordination of the default change with v2 planning and the installer, is an Engineering Sync action.

Suggestions

  • Give the keyring-mode operator guidance clear docs. Document that keyring mode ignores config.yml credentials and that the operator must remove old plaintext values.
  • Add an explicit regression test for the setup endpoint. A request that reaches the public listener, even if path-identical to the setup endpoint, must not be served. This is a security property, not just a routing detail.
  • Do not expose backend mode on unauthenticated endpoints. If surfacing reduced-security (config) mode, do it in an authenticated admin view.

Ganesh Review Feedback - 10/14/2026

  1. How will existing binary deployments be migrated? Will credentials be moved from config.yml or the environment into the OS keyring only with the user's consent? Prompting or migrating credentials without the user's prior consent does not feel like the right design. What happens if they decline?

Resolved 10/27/2026: Superseded by the 10/26 feedback. There is no migration. See Decision 5.

  1. Could the UI setup flow be delivered with the backend changes? The UI is the primary way customers expect to set up and use Console, so it could let them create their credentials during first-run setup. Credentials should be printed to the CLI only for headless deployments, where no UI is available. Could the UI flow be included with the backend changes rather than deferred to a later phase?

Not adopted (10/21/2026): Phase 2 UI remains deferred. The Phase 2 UI design is specified in the Architecture document.

  1. How will standalone Console be supported in Docker Compose? The ADR says Console will fail to start if the OS keyring is unavailable, while Docker/Kubernetes secret-file support is deferred. Does this mean Docker Compose users will be unable to run Console with standalone authentication under this design? If not, how will Console access and persist their admin credentials? We have a customer using this deployment, and I want to make sure this change does not disrupt them.

Resolved 10/27/2026: AUTH_SECRET_BACKEND=config is the default throughout v1.x (Decision 4), so Docker Compose deployments are unaffected.

  1. In my opinion, the -clean option is not needed at this point. Users can delete credentials directly from the OS secret store. Adding a Console-managed recovery path gives us additional security responsibilities; we can revisit it if customers ask for it.

Adopted (10/26/2026): console -clean is not implemented. Operators manually delete both keyring entries using OS-native tools.

Ganesh Review Feedback - 10/26/2026

Keep AUTH_SECRET_BACKEND=config as the default throughout v1.x, and change the default only after the keyring backend and UI setup are fully implemented.

  1. Keep AUTH_SECRET_BACKEND=config as the default throughout v1.x. Customers can opt into keyring. Make it the default only with a future major release, such as Console v2, coordinated with the installer release and accompanied by advance notice and migration guidance.
  2. Keep keyring mode isolated to the keyring. When keyring is selected, Console checks the keyring only and does not read, import, or use credentials from config.yml or environment variables, nor prompt to migrate them. When required entries are absent, use first-run credential setup in the UI. Since UI setup is deferred to Phase 2 and keyring is not yet in production or preview, continue with config as the v1.x default and document the temporary Phase 1 setup path for customers who opt into keyring.
  3. Coordinate the default change with major-version release planning. Use Engineering Sync to determine timing and coordinate Console with the installer, possibly together with discovery and the rpc-go download feature.

Adopted 10/27/2026. See Decisions 4 and 5.

Review Comments - Divneil (29/Oct/2026)

Overarching comment: If Phase1 is temporary and code will be removed. Why do it? Phase2 should be revisited as mentioned below in Loopback-only exposure point.

Response 11/13/2026 - partially addressed: Decision 7 retains the temporary CLI prompt so opt-in customers can initialize credentials before the UI is available. The keyring and hashing backend are reused in Phase 2; only the prompt is removed. Whether to skip the temporary prompt remains unresolved.

  1. Pls define "headless mode" or share the reference to the definition of the same.

Addressed 11/13/2026: Decision 8 defines headless mode as Console running without the UI. The operator calls the loopback setup API from the Console host, supplying the request body from a file or standard input.

  1. Loopback-only exposure: Why is this interface exposed for configuration. My understanding is that the username configuration is:

    • at installation/configuration time before deployment
    • using cli
    • authenticated interface with older password

    The loopback interface introduces a race on multi-user system. The user who accesses the URL first becomes the admin

Response 11/13/2026 - partially addressed: Decision 8 now requires the browser setup form to submit the new credentials together with a short-lived, single-use token carried in the fragment of the complete setup URI printed to the local CLI. Opening the setup page without a valid token cannot complete this browser flow. Setup is serialized and the credential state is rechecked under a cross-process lock. Success invalidates the token and returns the user to sign-in, logged out; failure shows an error. The headless path still needs an explicit authorization requirement before this concern can be considered fully resolved.

  1. JWTs Revocation: In case of password change, the in-flight JWTs have to be revoked because they were issued in response to authentication using username/password

Adopted 11/13/2026: Decision 6 requires server-side revocation of all previously issued standalone JWTs on a successful password change, including the current session. Old tokens are rejected even if unexpired, and the operator must sign in again.

  1. PBKDF2 Salt: Pls state the salt will be a new crypto random number on every password storage. It cannot be reused.

Adopted 11/13/2026: Every new password hash uses a fresh 16-byte salt generated with crypto/rand, including when the same password is set again. Salt generation failure aborts the operation without replacing credentials.

  1. $pbkdf2$<iterations>$<hex salt>$<hex hash> could be
    • $pbkdf2 -> $pbkdf2-sha256 for now
    • $<hex salt>$<hex hash> -> $<base64 salt>$<base 64 hash>. base64 is generally accepted format

Adopted 11/13/2026: Decision 3 and the architecture specification use $pbkdf2-sha256$<iterations>$<base64(salt)>$<base64(hash)>, with standard padded Base64 for both fields.

Clone this wiki locally