Per-user isolation of Valkey clients and metrics collectors

[ArgusLi]

RBAC part of #552.

1. Summary

When several people use one Valkey Admin server against the same Valkey endpoint, they share a single Valkey client and a single set of metrics collectors. Whoever connected or retried last decides the Valkey identity that runs everyone's commands. A user reported this: their ACL LIST was rejected with an error naming another user, admin_user. The same sharing lets a restricted user read hot keys, command logs and MONITOR output gathered with someone else's credentials.

We propose to fix this in two phases. Phase 1 gives each Valkey identity its own clients and collectors, so nothing is shared between different users. It is the smallest change that removes the problem, and it costs more processes and connections. Phase 2 brings back shared metrics collection for operator-configured instances, using an admin account and a server-side permission check, while each user keeps their own client for commands.

2. Tenets

1. Valkey's ACL is the source of truth for what a user may do. Valkey Admin does not invent its own permission model.
2. A user never runs a command, or sees data, under another user's Valkey identity.
3. The server enforces access. The frontend may hide what a user can't use, but hiding is never the control.
4. When safety and efficiency conflict, we ship safety first and recover efficiency later.

3. Background

Valkey Admin keeps all connection state in one server process. The maps that matter are:

State	Keyed by today	Shared between users
Valkey client per connection	Endpoint and database (host-port-dbN)	Yes
Cluster client being created	Cluster	Yes
Cluster topology, and the credentials used to spawn collectors	Cluster	Yes, last writer wins
Metrics collector per node	Node (host-port)	Yes
Browser session	Session cookie	No

Release 1.2.0 (#500) stopped a session from reusing a cached client it never connected to. It did not separate two sessions that each connected successfully to the same endpoint. Both sessions then own the same client slot. A second user's connect overwrites the slot without closing the old client, and a retry by the first user flips it back. Collectors are keyed by node alone, so any user who owns a connection to a node reads data gathered under the credentials that spawned its collector. In Kubernetes, that is the sidecar's credentials.

A Valkey ACL checks each command against the identity of the client that sends it. Valkey does not propagate ACLs between nodes: each node loads its own aclfile or valkey.conf users, and runtime changes stay on the node that received them.

4. Customers

We see two kinds of deployment.

Shared-instance teams. Many people with different roles work against a small, stable set of Valkey instances. They want shared, efficient monitoring and want Valkey ACLs (and, from Valkey 9.2, ACL roles) to decide who sees what.

Individual developers. A few people per instance, and instances come and go often. They connect to new endpoints from the UI and care little about shared monitoring.

Both need the same guarantee: what a user does and sees in Valkey Admin matches what their own Valkey user is allowed to do.

5. Goals and non-goals

Goals for Phase 1:

• Commands always run as the Valkey identity the user connected with.
• Users see only metrics data gathered with their own credentials.
• A user can reuse a client or collector only after Valkey accepts their credentials, not just because they name the same username.
• No change to the frontend protocol, to how connections are identified in the UI, or to the metrics collector.

Non-goals for Phase 1:

• A Valkey Admin login. Identity remains "the credentials you connected with".
• Shared metrics collection between users. Phase 2 restores this.
• Fixing how ElastiCache IAM and GCP IAM identities are derived. Those tokens come from the server's own cloud credentials, as today.

6. Proposal

Phase 1: isolate by Valkey identity

Owner identity. Every piece of server state that holds a client, a collector or per-cluster data is keyed by an owner identity: the auth type and the Valkey username the user connected as, with an empty username meaning default. For ElastiCache IAM the identity also includes the region and replication group. Users who authenticate as the same Valkey user on the same connection share one client and one set of collectors. Users authenticated as different Valkey users share nothing. The frontend keeps identifying connections the same way.

Reusing a client. The owner identity is not secret, so it never grants access by itself. A new client authenticates when it's created, as today. To reuse an existing client, the server first checks the presented credentials with a fresh AUTH against that node, and hands over the client only if Valkey accepts them. Valkey Admin never stores or compares secrets. This means:

• After a password change, only the current password gives access to an existing client.
• A disabled user can't reuse a client. Clients that are already open keep working, as Valkey itself behaves.
• A Valkey user with several valid passwords is one identity, with one client and one set of collectors.
• Failed authentications are rate-limited per username, so Valkey Admin can't be used to guess passwords.
• ElastiCache IAM and GCP IAM behave as today: the server mints the token, so authentication succeeds for the named user.
• A client that is replaced or no longer used is closed.

Collectors. Each identity's collectors keep their own metrics history and their own collector settings. Today all collectors share one settings file, so one user's Monitor or retention change would reach every other user's collectors. History is stored under a readable, reversible encoding of the owner identity, so it survives restarts and password changes, and operators can tell whose history is whose. Collectors stop when their owner no longer has a connection to the node. The server caps the number of collector processes it runs. When the cap is reached, a new connection still works for commands and the cluster dashboard, and its metrics panels say the collector limit was reached. The collector itself does not change: the server already controls where each collector stores data and which settings file it reads, and already gives each collector its own registration key.

Browser sessions. Ownership moves from the session cookie to the browser's live connection to the server. That connection may act only on Valkey connections it opened with credentials Valkey accepted. When it drops briefly, the frontend already re-sends the credentials it holds in memory, and the server verifies them and reattaches to the existing client. Valkey clients and collectors are released shortly after the last browser using them goes away. The one capability lost is resuming after a full page reload, or in a new tab, without the password. Web never keeps a real password across a reload, so the user re-enters it, and the frontend already prompts for a missing password. IAM, GCP IAM and passwordless connections reconnect as before. Electron is unaffected when the OS keystore is available. Without one, it prompts after a reload as Web does.

Restarts. A server restart clears clients, collectors, topology and browser bindings, because all of them live in server memory. Users reconnect, and the server rebuilds them under the same owner identities, with nothing to regenerate. A graceful shutdown stops every spawned collector. If the server crashes instead, collectors it spawned keep running indefinitely and can't re-register with the new server, so they waste Valkey and server capacity until someone stops them. In Docker this doesn't arise, because the crash stops the container, so we will move to stop support for bare-metal server deployment. This doesn't affect the Electron deployment as Electron will close child processes on a crash.

Kubernetes. Each Valkey pod runs one metrics sidecar under one set of credentials, so sidecars can't be made per user. In Kubernetes, Phase 1 disables collector-backed features (dashboard history, hot keys, big keys, command logs, Monitor and collector config) and shows a clear message in their place. Commands and the cluster dashboard keep working through each user's own client. Phase 2 restores these features for Kubernetes through the operator-configured admin account.

What Phase 1 removes:

• Browser sessions: the 24-hour cookie, its expiry timer, and the ownership checks built on it.
• Resuming after a page reload without re-entering the password.
• The single client per endpoint and single collector per node shared by every user.
• The collector settings file shared by every user.
• Connections and collectors that outlive their last browser by up to a day.
• Bare-metal server deployment.

Phase 2: shared collectors for operator-configured instances

Operators configure their Valkey instances before launch, with an admin account used only for metrics collection. Each configured instance gets one shared collector process that polls each primary through a Glide cluster client, matching Glide's default of sending INFO to all primaries. MONITOR runs in per-primary subprocesses that report aggregates to the collector.

Users keep their own client per connection for every command, exactly as in Phase 1. We considered one unified client per instance that runs ACL DRYRUN as the user before each command, and rejected it (Section 8). With per-user clients, Valkey checks every command against the user's own ACL as it runs, including commands inside scripts and transactions, and ACL changes apply immediately.

ACL DRYRUN appears in Phase 2 only to gate aggregates. The shared collector gathers aggregates as the admin account, so Valkey never checks the requesting user's ACL against the commands behind them. ACL DRYRUN asks Valkey whether that user would be allowed to run a command, without running it. Probing with the user's own client doesn't work for every aggregate: MONITOR has no harmless probe, and checking access to all keys needs ACL GETUSER.

A user sees aggregate data only if their own Valkey user could run every command the aggregates rely on (for example INFO, COMMANDLOG GET, MONITOR, SCAN), including access to all keys, on every node. The check uses ACL DRYRUN and ACL GETUSER through the admin account, takes the weakest result across the cluster's nodes, and is cached in the collector until ACL DIGEST changes. The server enforces it.

A user's connection uses a shared collector only when the address the user authenticated to is in the operator's configured node list. Connections the operator didn't configure fall back to Phase 1 per-user collectors, and the UI warns that this mode is inefficient.

Phase 2 depends on a change in Valkey Glide. Today, a command sent without an address, such as INFO, goes to all target nodes, but the reply succeeds only if every node succeeds: one error or timeout fails the whole call, and the results from the other nodes are lost. Phase 2 needs the cluster client to return every node's reply, with a timed-out or failed node included as an error entry alongside the others. Without it, one slow primary blanks metrics for the whole instance.

7. Cost of Phase 1

Phase 1 trades efficiency for safety:

• Collector processes grow from one per node to one per node per distinct Valkey identity, up to the server's cap. Ten users on a cluster with 100 primaries can mean up to 1,000 collector processes on the Valkey Admin host.
• All spawned collectors share one rate limit on the server's collector API, 600 requests a minute by default. That supports about 100 collectors, so we will increase the default ORCHESTRATOR_RATE_LIMIT_MAX.
  • It's important for operators to understand the ORCHESTRATOR_RATE_LIMIT_MAX setting so that they can set the cap smartly, rather than discovering the ceiling in production. A sizing guide will be added for this setting.
• Each user's cluster client holds two connections per node, so every node accepts two connections per active user.
• Metrics history is stored once per user rather than once per node.
• In Kubernetes, collector-backed features are unavailable until Phase 2.
• In Web and Docker, and in Electron without an OS keystore, reloading the page or opening a new tab asks for the Valkey password again for password-authenticated connections.
• Reusing an existing client costs one extra short-lived connection and AUTH per connect or resume.

These costs fall on the deployments with the most users per instance, which are the shared-instance teams Phase 2 is designed for.

8. Alternatives considered

One unified client per Valkey instance, with ACL DRYRUN before each user command. This saves connections, two per node for each user. Rejected for both phases:

• Commands run as the admin account, so any gap in the check gives the user full admin rights.
• Commands inside EVAL, EVALSHA and FCALL run with the admin's rights, whatever the user's ACL says.
• Each command costs an extra round trip, and the check and the command can see different ACL states.
• ACL DRYRUN doesn't check whether the user is enabled.
• Commands that change connection state, such as SELECT, CLIENT or HELLO, would affect every user on the shared client.
• Valkey's CLIENT LIST, ACL LOG and command logs would show the admin account, not the user.
• Connections the operator didn't configure have no admin account to run the check with.

Gate access in the frontend only. The WebSocket API accepts any action from any socket, so hiding UI elements stops nothing. Rejected as a control.

Isolate per browser session instead of per identity. This is simpler, but the same person in two browsers would get two sets of collectors, and people who legitimately share one Valkey user would gain nothing. Rejected in favor of keying by Valkey identity.

An owner tag made from a keyed hash of the credentials. The server would hash auth type, username and secret with a key generated at startup, and reuse a client when the hashes match. It avoids the extra AUTH round trip, but Valkey Admin rather than Valkey decides who may reuse a client. The old password keeps matching after a password change, a disabled user still matches, and a user with two valid passwords gets two clients and two sets of collectors. The key changes on every restart, so data directories named by the tag would be orphaned each time, while persisting the key would make directory names usable for offline password guessing. Rejected in favor of AUTH-gated reuse.

Keep sessions alongside owner identities. This lets a reload or new tab resume without the password. It keeps a 24-hour cookie that grants use of authenticated Valkey clients, a second ownership check that overlaps AUTH-gated reuse, and the expiry sweep and listener that drive teardown. It also keeps a closed tab's clients and collectors alive until the session expires. Session ownership alone is what allowed the reported bug, because both users owned the same connectionId. Rejected: re-entering the password after a reload is an acceptable cost.

Go straight to Phase 2. Phase 2 needs new configuration, a new collector design, a permission check and a Glide change, so it would leave the impersonation open for longer. Rejected as the first step.

Keep serving Kubernetes sidecar data to every user. This keeps metrics working in Kubernetes but leaves the sidecar's credentials deciding what every user sees, which breaks tenet 2. Rejected for Phase 1.

Appendix A. Shared state that Phase 1 scopes by owner identity

Every piece of server state that today is shared between users of the same endpoint: Valkey clients and connects in progress, cluster topology and the credentials used to spawn collectors, the collector registry and each collector's registration key, CPU baselines, config pushes in progress, the record of which browsers are watching each connection, and each collector's history and settings on disk. Per-cluster bookkeeping that is already held per browser connection only needs to carry the owner identity.

Appendix B. ACL facts this design relies on

• Valkey checks each command against the sending client's user, at the time it runs. Disabling a user doesn't close connections that already authenticated.
• ACLs are stored per node. The cluster bus has no ACL messages, and ACL SETUSER isn't replicated.
• ACL DRYRUN checks command, key, channel and database rules, including roles, but not whether the user is enabled.
• SCAN has no key spec, so key patterns don't limit which key names it returns.
• Database rules (db=) arrive in Valkey 9.1.0 and roles (role=) in 9.2.0.

[mbor-vendor]

Thanks for putting this together, Argus. This is really solid! The model makes sense to me and I like shipping safety first, then efficiency in Phase 2. Just one thing on the design I'd like us to sort before Phase 1.

The crashed-server case. On a graceful shutdown the collectors are stopped, but on an ungraceful crash they can keep running with nothing tying their lifecycle to the server. On a plain host that can leave orphaned collectors still holding Valkey connections, and since Phase 1 spawns more of them (one per node per identity), a crash loop could compound. It's a fairly rare case, but there's no cleanup path described, so worth deciding deliberately.

[ArgusLi]

Thanks for the review!

That's a good point you're making. I should be more explicit about not supporting bare-metal server deployment moving forward from phase 1 onwards. When the container or pod stops, the runtime tears down the whole process tree, so collectors are reaped cleanly. Removing bare-metal server deployment also makes overall scaling design less multi-faceted, as we can work with the container model from small Docker deployments to large Kubernetes deployments.

To be clear, the Electron desktop app stays, this only removes the bare-metal server path, so single-developer use is unaffected. I'll make the change in the doc. Thanks!

[ArgusLi]

Merged as part of #581