NATS multi-tenancy limitation / user creation

[sdktr]

hi Team!
Wondering around the options for backend storage, I noticed the support for (kine based) NATS. Great!

The docs state a limitation on the usage of multi-tenancy though. Can someone explain to me what is missing here? Kine-by-nats supports a unique bucket name per cluster (using that in the lab to run multiple clusters on the same NATS KV).

According to the initial issues we're missing user management, which hints at the creation of a TCP specific NATS user? Can someone shine a light on this, how is this done in postgres or other kine backends?

https://github.com/clastix/kamaji/issues?q=nats%20is%3Aissue
@prometherion
@thecodeassassin

[sdktr]

Copilot helped out here:

NATS multi-tenancy is not supported in Kamaji because, unlike PostgreSQL/MySQL, the current NATS integration does not provision per-tenant identities or permissions. In internal/resources/datastore/datastore_storage_config.go, NATS is explicitly special-cased to reuse the datastore’s shared basicAuth username/password instead of generating tenant-specific credentials. The NATS datastore backend (internal/datastore/nats.go) only creates a KV bucket; CreateUser, GrantPrivileges, DeleteUser, etc. are all no-ops, and UserExists/GrantPrivilegesExists always return success. Because all tenant control planes would otherwise share the same NATS credentials, Kamaji enforces a hard single-tenant restriction for NATS in internal/resources/datastore/datastore_multitenancy.go.

By contrast, PostgreSQL follows the normal multi-tenant path: Kamaji generates tenant-specific DB_USER, DB_PASSWORD, and DB_SCHEMA values into the tenant secret, then datastore_setup.go uses the PostgreSQL backend to create the database, create the user, and grant privileges. That is why PostgreSQL works without a special branch here while NATS does not.

To fix NATS multi-tenancy, Kamaji would need a real per-tenant auth and authorization model for NATS. The likely direction is NATS JWT/NKey-based identities: for each tenant, Kamaji would generate tenant credentials and issue a user JWT whose permissions are restricted to that tenant’s JetStream/KV resources. However, this is not something nats.go can do by itself. nats.go is only a client SDK; it can consume credentials, but it does not create trusted NATS users on the server. For JWT-based auth to work, the NATS deployment must be preconfigured to trust an operator/account signing chain, and Kamaji must either hold the trusted account signing key or call an external signer service that does. In other words, a one-time trust/bootstrap setup is required on the NATS side before Kamaji can mint per-tenant credentials.

A full solution would therefore require: extending the DataStore model to represent NATS auth provisioning; implementing real CreateUser/GrantPrivileges/DeleteUser logic in internal/datastore/nats.go; changing datastore_storage_config.go so NATS writes tenant credentials instead of reusing root credentials; restricting each tenant JWT to the subjects/JetStream API access needed for only that KV bucket; verifying that kine can authenticate using the chosen NATS credential format (JWT/NKey/creds file, not just username/password URL auth); and only then removing the single-tenant guard in datastore_multitenancy.go. In short: NATS multi-tenancy is currently unsupported because Kamaji has bucket creation but no tenant-scoped auth model; fixing it means adding trusted per-tenant NATS identity issuance and enforcement.

A likely recommended design
If I were designing this for Kamaji, I would recommend:

Control plane
Keep a shared privileged NATS connection for Kamaji setup tasks
Do not let tenant kine use shared creds
Identity provisioning
Use NATS JWT/NKey user auth
Store the account signing key outside Kamaji if possible
Expose a signer interface to Kamaji
Per-tenant auth
For each tenant:

generate a user NKey
mint a user JWT signed by the trusted account signer
embed permissions limited to the tenant’s KV/JetStream subject space
write creds to a Secret owned by the tenant control plane
Kamaji secret/config changes
For NATS, instead of current DB_USER / DB_PASSWORD, probably store:

NATS_CREDS or NATS_USER_JWT and NATS_USER_SEED
or another format kine can actually consume
Kine integration
confirm kine can authenticate using that format
if not, extend Kamaji/kine integration accordingly
Deletion/revocation
On tenant deletion:

delete bucket
revoke or expire tenant JWT
rotate if necessary
One subtle but important point
With JWT-based NATS auth, “grant privileges” is usually not a separate mutable server-side record the way SQL grants are.

Instead, the privilege set is part of the signed JWT itself.

So conceptually:

PostgreSQL: CreateUser then GrantPrivileges
NATS JWT: IssueSignedUserJWTWithPermissions
This means Kamaji’s Connection interface may be a bit awkward for a true NATS-native implementation. It can still be adapted, but NATS auth provisioning is not a natural fit for the current SQL-shaped abstraction.

[prometherion]

Thanks for the AI report: I'm fully aligned on the findings.

The NATS integration in Kamaji has reached a natural inflexion point. When the feature was contributed by @thecodeassassin, we made a deliberate investment: several calls to share context on the codebase and set the contributor up for ongoing involvement. Unfortunately, engagement dropped off entirely after the merge, leaving the implementation without a path forward.

Usage data from our telemetry server tells a consistent story: NATS sees very limited adoption relative to etcd, MySQL, and PostgreSQL. The original intent was to use NATS support as a starting point for broader community collaboration. That hasn't materialised.

In practice, the driver partially supports our multi-tenancy model, has no current customer demand, and creates incremental technical debt over time. For a project maintained exclusively by our team, that's a combination we can't sustain without trade-offs elsewhere.

We've currently undecided if we have to deprecate and remove the NATS driver, mostly because we have to concentrate our maintenance capacity on the datastores our customers actually rely on and uphold the quality standard Kamaji is built on. If you or your organisation is up to help Kamaji with this support, let me know.