Standalone IAM Setup

This page is the configuration reference for the standalone IAM service — the versitygw iam process — and for the S3 gateways that authenticate against it. For what the service is and why you would run one, see Standalone IAM.

Table of Contents

• Running the service
• Global options
• Private and isolated OIDC providers
• Storage backends
• The private endpoint
• Private protocol versioning
• TLS
• Startup validation and errors
• systemd
• Kubernetes and Docker
• See Also

---

Running the service

The standalone IAM service is a subcommand of the same versitygw binary that runs the S3 gateway:

versitygw [global options] iam [command options]

Important

Global options must be written before the iam subcommand, and iam-specific options after it. This is a property of the CLI parser, not a style preference — versitygw iam --port :7070 --dir /var/lib/vgw-iam fails, while versitygw --port :7070 iam --dir /var/lib/vgw-iam is correct.

A minimal invocation needs a root credential, a listen address, and one storage backend:

versitygw --port 127.0.0.1:7070 --access user --secret pass iam --dir /var/lib/vgw-iam

The subcommand's own options are --dir and the thirteen --vault-* options (Storage backends), the five --private-* options (The private endpoint), --quiet, and these OIDC options:

Option	Environment variable	Default
--disable-oidc-thumbprint-autofetch	VGW_IAM_DISABLE_OIDC_THUMBPRINT_AUTOFETCH	false
--oidc-allow-private-endpoints	VGW_IAM_OIDC_ALLOW_PRIVATE_ENDPOINTS	false
--oidc-allow-insecure-transport	VGW_IAM_OIDC_ALLOW_INSECURE_TRANSPORT	false

See Private and isolated OIDC providers before enabling either endpoint relaxation. Everything else is a global option.

Unless --quiet is set, the service prints a banner listing every address it will serve. The banner is printed before the listeners bind, so an address that is later rejected — a plain-TCP private endpoint, for example — still appears in it before the process exits with the error.

Startup and configuration errors are printed as a single line to stderr and the process exits with status 1. The service shuts down on SIGINT or SIGTERM, printing terminating signal caught, shutting down and giving each listener a 10 second graceful-shutdown window.

Only the versitygw binary carries the iam subcommand. The RDMA build, vgwrdma, does not register it; run the IAM service from versitygw and point vgwrdma at it with --iam-standalone-endpoint like any other gateway.

---

Global options

iam inherits the global option set printed in Global Options. Only a subset affects the IAM service, and only the ones whose behavior differs from the gateway's are worth restating here.

Option	Environment	Behavior on the iam command
--port, -p	VGW_PORT	Listen address of the public IAM API — the AWS Query API that aws iam and aws sts clients talk to. Same address forms as the gateway's --port, repeatable. The default :7070 is also the S3 gateway's default, so two co-located processes must move one of them.
--access, --secret	ROOT_ACCESS_KEY_ID, ROOT_SECRET_ACCESS_KEY	Required — the service refuses to start without them. They define the root account, the only credential never written to a storage backend. Root bypasses identity-policy evaluation entirely, and is the only credential the private endpoint accepts.
--cert, --key	VGW_CERT, VGW_KEY	TLS certificate and key file paths for the public listener; both or neither. They do not apply to the private endpoint, which has its own --private-cert / --private-cert-key pair.
--log-level, --debug	VGW_LOG_LEVEL, VGW_DEBUG	Covers both listeners. The public request log is prefixed vgw-iam and the private endpoint's vgw-iam-private, so the two streams stay distinguishable in one file. See Debugging.
--quiet, -q	VGW_QUIET	Suppresses the per-request access log and the startup banner. Exists both as a global option and as an iam subcommand option; either one enables it.
--health	VGW_HEALTH	Registers an unauthenticated GET route returning 200, ahead of both the rate limiter and the debug-logging middleware, so health traffic is never throttled and never fills a debug log. Only GET is routed — a POST to the path returns 404 — and the path is reserved, so a GET never reaches an IAM action. See HealthCheck.
--socket-perm	VGW_SOCKET_PERM	Applies to file-backed unix sockets given to --port and --webui. It does not apply to --private-ports, which has its own --private-socket-perm.
--cors-allow-origin	VGW_CORS_ALLOW_ORIGIN	Enables the CORS middleware on the public IAM API; without it no browser on another origin can call it. The configured value is echoed on every response without being compared against the request's Origin, so set it to the WebUI's own origin rather than * where you can. A request with no Origin header passes through untouched, so command-line and SDK clients are unaffected.
--region, -r	VGW_REGION	Seeds the region selector on the WebUI login page, and nothing else.
--max-connections, --max-requests	VGW_MAX_CONNECTIONS, VGW_MAX_REQUESTS	Cap concurrent connections and in-flight requests on the public IAM API; both must be positive. A request turned away by the in-flight limiter gets the Throttling error, Rate exceeded., at HTTP 400. --max-requests above --max-connections is allowed but logs a startup warning. Neither limit applies to the private endpoint.

--region does not change the SigV4 signing region of the IAM or STS API, which is fixed at us-east-1. A client that signs for any other region is rejected with SignatureDoesNotMatch / Credential should be scoped to a valid region.

The IAM service can host the WebUI itself, with the same --webui* options as the gateway. Four things differ here; WebGUI has the rest.

• The IAM URLs offered on the login page are derived from this process's own --port addresses unless --webui-iam-gateways overrides them.
• S3 and admin gateway URLs are not auto-detected — the IAM service does not know where your S3 gateways are. Set --webui-gateways to pre-fill them.
• --webui-s3-prefix is a gateway-only option and has no effect on the iam command.
• --webui without --cors-allow-origin forces the CORS origin to * and prints WARNING: WebuiPorts is set but CORSAllowOrigin is not; defaulting to '*'; consider setting it to the WebUI's own origin.

Private and isolated OIDC providers

By default, an OIDC provider must use HTTPS, have no explicit port, resolve only to publicly routable addresses, and present a certificate that verifies by hostname against the system trust store or the provider's registered thumbprint. These defaults protect the IAM service from server-side request forgery (SSRF) and authenticate the identity provider before its signing keys are trusted.

The endpoint policy applies to both outbound phases:

1. the optional certificate-thumbprint handshake during CreateOpenIDConnectProvider; and
2. discovery-document and JWKS retrieval during every AssumeRoleWithWebIdentity cache miss.

Choosing an endpoint policy

Configuration	Accepted endpoint	TLS and thumbprint behavior	Intended use
Default	Public https://host on implicit port 443	Fully verified	Public identity provider
--oidc-allow-private-endpoints	HTTPS at private, loopback, link-local, unspecified, or multicast addresses; explicit ports allowed	Still fully verified, including hostname and registered-thumbprint checks	In-cluster or private-network provider with working TLS
--oidc-allow-insecure-transport	Plain HTTP is allowed; HTTPS is also accepted without certificate verification	HTTPS hostname, CA-chain, and ThumbprintList checks are all disabled	Provider reached over an independently trusted path
Both flags	HTTP or HTTPS at private addresses and explicit ports	No TLS authentication when insecure transport is enabled	Loopback sidecar in the IAM pod

--oidc-allow-insecure-transport does not by itself permit a loopback/private address or explicit port. A typical sidecar URL such as http://127.0.0.1:8080 therefore requires both flags.

Private HTTPS example

Use only the private-endpoint relaxation when the provider has a certificate for its service hostname:

versitygw --port :7070 --access root --secret '<secret>' iam \
  --dir /var/lib/versitygw/iam \
  --oidc-allow-private-endpoints

This permits a URL such as https://spire-oidc.spire.svc.cluster.local:8443. The explicit port becomes part of the stored provider URL, provider ARN, token iss value, and trust policy condition-key prefix, so those values must match it exactly.

If the provider uses an internal CA that is not in the IAM container's system trust store, supply ThumbprintList explicitly when registering it. That thumbprint can authenticate discovery and JWKS TLS connections, but it cannot make the enrollment-time auto-fetch handshake trust an unknown CA.

Loopback sidecar example

For a plaintext discovery provider bound to loopback in the IAM server's own pod:

versitygw --port :7070 --access root --secret '<secret>' iam \
  --dir /var/lib/versitygw/iam \
  --oidc-allow-private-endpoints \
  --oidc-allow-insecure-transport

A plaintext provider has no TLS certificate, so provider creation stores an empty ThumbprintList when one is omitted; there is no thumbprint to auto-fetch. Its http:// scheme is retained in the stored URL, ARN, token issuer, and trust-policy condition keys, keeping it distinct from an HTTPS provider on the same host.

[!DANGER] --oidc-allow-insecure-transport makes the network path the only authentication of the identity provider. It allows HTTP discovery, JWKS, and redirects, and for HTTPS it disables hostname validation, CA-chain validation, and registered-thumbprint pinning. Do not use it across a routable cluster or host network. Bind the provider to loopback in the same pod, include only trusted containers in that pod, and prevent untrusted workloads from entering its network namespace.

SSRF implications

--oidc-allow-private-endpoints removes the address-range block for the enrollment handshake, discovery document, JWKS URI, and every redirect target. It consequently re-enables access to cloud metadata addresses such as 169.254.169.254, loopback services, and other internal control planes.

The DNS-rebinding defense remains: each connection resolves once and dials the selected IP directly. Fetches also retain the 8-second timeout, five-redirect limit, 1 MiB response limit, and JWKS key-count limits. Those controls reduce exposure but do not restore the private-address boundary.

Use these safeguards whenever private endpoints are allowed:

• restrict CreateOpenIDConnectProvider and provider updates to trusted IAM administrators;
• apply egress firewall or Kubernetes NetworkPolicy rules so the IAM process can reach only the intended IdP addresses and ports, explicitly denying metadata and unrelated internal ranges;
• prefer private HTTPS with explicit thumbprint pinning over insecure transport;
• scope each role trust policy tightly by audience and subject;
• monitor provider changes and outbound connection failures.

The detailed URL, thumbprint, redirect, and JWKS behavior is documented in STS and Web Identity.

Global options that are ignored

The parser accepts every global option on the iam command, but most of them configure the S3 gateway and are silently discarded here. These have no effect on the standalone IAM service:

Option group	Options
Admin API	--admin-port, --admin-max-connections, --admin-max-requests, --admin-cert, --admin-cert-key, --admin-access-log
Logging to file	--access-log, --log-webhook-url
S3 behavior	--virtual-domain, --disable-acl, --readonly, --disable-strict-bucket-names, --mp-max-parts, --copy-object-threshold
Website hosting	all --website-*, and --webui-s3-prefix
Event notifications	all --event-*
Metrics	all --metrics-*
Built-in IAM backends	--iam-dir, all --iam-ldap-*, all --iam-vault-*, all --ipa-*, all --s3-iam-*, --iam-cache-*, --iam-debug
Standalone IAM client	all --iam-standalone-*

Passing the gateway's --iam-dir to the iam command is a common mistake: it is accepted and ignored, and the service then fails with no IAM storer config specified because the subcommand's own --dir was never set.

---

Storage backends

The standalone IAM service persists users, roles, OIDC providers and STS sessions in exactly one of two backends, selected with --dir or --vault-endpoint-url. Neither is an error — no IAM storer config specified — and both is an error — multiple IAM storer configs specified: dir, vault. There is no in-memory or default backend, and no migration path between the two, so pick one at deployment time.

File-backed storage

--dir (VGW_IAM_DIR) names a directory holding the entire IAM database as one JSON document:

Path	Contents
<dir>/iam.json	The whole database — created on first start with mode 0600
<dir>/iam.json.backup	A copy of the previous state, rewritten before every mutation; does not exist until the first mutation

The top-level JSON keys are users, accessKeyIndex, userNameIndex, roles, roleNameIndex, oidcProviders and sessions. The two name indexes map a lowercased name to the canonical stored casing — case-insensitive uniqueness with the original casing preserved — and are rebuilt automatically from users and roles if a hand-edited file is missing them.

The service does not create the directory. It must exist and be writable by the process before start, or the service exits with init internal IAM storer: write default iam: open <dir>/iam.json: no such file or directory.

Warning

The file store coordinates writers inside one process only. Every mutation is a read-modify-write of the entire file, with no cross-process locking — two IAM services pointed at the same directory will silently lose each other's updates. Run exactly one versitygw iam --dir process per directory, and never scale it to more than one replica. The Helm chart enforces this by refusing to render an IAM Deployment with replicaCount > 1 and internal storage.

Two further properties are worth knowing before choosing this backend:

• Secrets at rest are not encrypted. iam.json holds every IAM user's secret access key in plaintext. The file is created 0600, but the directory's own permissions are yours to set — put it on a filesystem only the service account can read, and back it up as you would a key store.
• Last-used tracking is synchronous here and rewrites the whole file. Both planes record — S3 requests to a gateway backed by this service included — but repeat use of the same credential, in the same service and region, is coalesced to at most one write a minute. Vault storage records asynchronously and coalesces the same way.

Vault KV v2 storage

Option	Environment	Notes
--vault-endpoint-url	VGW_IAM_VAULT_ENDPOINT_URL	Vault server URL. This is the option that selects the Vault backend
--vault-mount-path	VGW_IAM_VAULT_MOUNT_PATH	KV version 2 secrets engine mount; the client library's default, kv-v2, applies when empty
--vault-secret-storage-path	VGW_IAM_VAULT_SECRET_STORAGE_PATH	Path prefix inside that mount; defaults to iam
--vault-auth-method	VGW_IAM_VAULT_AUTH_METHOD	Auth method mount path; the client library's default is approle
--vault-root-token	VGW_IAM_VAULT_ROOT_TOKEN	Root-token authentication
--vault-role-id, --vault-role-secret	VGW_IAM_VAULT_ROLE_ID, VGW_IAM_VAULT_ROLE_SECRET	AppRole authentication; both must be set together
--vault-namespace	VGW_IAM_VAULT_NAMESPACE	Fallback namespace for both uses
--vault-auth-namespace	VGW_IAM_VAULT_AUTH_NAMESPACE	Namespace for the AppRole login; overrides --vault-namespace
--vault-secret-storage-namespace	VGW_IAM_VAULT_SECRET_STORAGE_NAMESPACE	Namespace for the KV requests; overrides --vault-namespace
--vault-server-cert	VGW_IAM_VAULT_SERVER_CERT	PEM content of the Vault server certificate to verify against
--vault-client-cert, --vault-client-cert-key	VGW_IAM_VAULT_CLIENT_CERT, VGW_IAM_VAULT_CLIENT_CERT_KEY	PEM content of the client certificate and key presented to Vault

--vault-endpoint-url is what actually selects the Vault backend: supplying any other --vault-* option without it leaves the service with no storage configured. Requests to Vault use a 10 second timeout. The resulting KV layout is:

Entity	KV path
Users	<secret-storage-path>/users/<lowercased user name>
Roles	<secret-storage-path>/roles/<lowercased role name>
OIDC providers	<secret-storage-path>/oidc-providers/<base64url of the scheme-stripped URL>
Sessions	<secret-storage-path>/sessions/<session access key id>

Names are lowercased in the path so that case-insensitive name uniqueness becomes an atomic property of Vault's own check-and-set write instead of a check-then-write race; the original casing is preserved inside the stored document. OIDC provider URLs are base64url-encoded because they contain /, which Vault treats as a path separator.

All writes use KV v2 check-and-set. A create is written with cas: 0, so a duplicate name loses the race and surfaces as EntityAlreadyExists. An update is written with the version that was just read; a mismatch is retried up to 3 times and then reported to the caller as ConcurrentModificationException. This is what makes Vault storage safe for multiple concurrent IAM service replicas.

On any Vault 403 the service re-runs the AppRole login once and retries the operation, so a short-lived AppRole token is fine. A root token takes precedence over --vault-role-id / --vault-role-secret with no error raised, despite the option text describing them as mutually exclusive — and it cannot be re-authenticated on a 403, since there are no AppRole credentials to fall back on.

A least-privilege Vault policy for the AppRole needs only these three paths:

path "<mount>/data/<secret-storage-path>/*"     { capabilities = ["create", "update", "read"] }
path "<mount>/metadata/<secret-storage-path>/"  { capabilities = ["list"] }
path "<mount>/metadata/<secret-storage-path>/*" { capabilities = ["delete", "list"] }

Vault Enterprise namespaces resolve in a two-level fallback — the AppRole login uses --vault-auth-namespace if set, otherwise --vault-namespace; the KV requests use --vault-secret-storage-namespace if set, otherwise --vault-namespace — and the auth namespace is only attached when authenticating with AppRole, since a root token carries its own scope. Because AppRole tokens are namespace-scoped, differing resolved namespaces with no root token refuse startup:

approle tokens are namespace scoped. auth namespace "x" and secret storage namespace "y" differ. use the same namespace or authenticate with a root token

The three Vault TLS options take PEM certificate content, not file paths — the opposite of --cert, --key, --private-cert, --private-cert-key and --private-client-ca, all of which are paths. Pass the PEM through the environment variable, or inline it with a shell substitution such as --vault-server-cert "$(cat vault-ca.pem)". TLS configuration is applied only when --vault-server-cert is non-empty: a client certificate supplied without a server certificate is silently ignored, which makes a missing --vault-server-cert a quiet way to lose mutual TLS to Vault. A client certificate set without its key refuses startup with client certificate and client certificate key should both be specified.

Environment variable collisions

Warning

The iam subcommand's storage options reuse the same environment variables as the S3 gateway's built-in IAM backends: --dir reads VGW_IAM_DIR, the same variable as the gateway's --iam-dir, and every --vault-* option reads the same VGW_IAM_VAULT_* variable as the gateway's corresponding --iam-vault-* option. These name different, incompatible stores. Share one environment file between an IAM service and a gateway and you configure both at once — and a gateway that was meant to use the service but is missing VGW_IAM_STANDALONE_ENDPOINT will silently bring up its own built-in IAM from that same variable instead of failing.

The two file stores are especially easy to confuse because they can live in the same directory without conflicting — they use different filenames and unrelated schemas:

	Built-in IAM (--iam-dir on the gateway)	Standalone IAM service (iam --dir)
Files	users.json, users.json.backup	iam.json, iam.json.backup
Top-level schema	{"accessAccounts": {<access key> → account}}	users, accessKeyIndex, userNameIndex, roles, roleNameIndex, oidcProviders, sessions

There is no converter between them. Keep the two environments separate: give the IAM service its own systemd instance or container, with its own environment file.

On the gateway, --iam-standalone-endpoint takes precedence over every built-in backend, so a stray VGW_IAM_DIR in a gateway's environment is harmless as long as the standalone endpoint is also set. It is the missing endpoint, not the stray directory, that produces a gateway quietly serving its own identities.

The option names also differ slightly across the two surfaces even where the environment variables match — the gateway spells three of them with underscores (--iam-vault-server_cert, --iam-vault-client_cert, --iam-vault-client_cert_key) while the iam subcommand uses hyphens throughout. See IAM Vault.

---

The private endpoint

The public IAM API is where administrators create users and where federated clients call STS. It is not how an S3 gateway authenticates traffic: that runs over a separate control channel, the private endpoint, whose whole purpose is that the gateway never holds a user's secret access key. Standalone IAM explains the model; what matters for setup is the routes and how the listener is secured.

Path	Purpose
/private/version	Protocol and build version
/private/derive-signing-key	SigV4 kSigning for one caller
/private/evaluate-policy	Identity-policy and session-policy decisions
/private/resolve-identity	Access key id → principal ARN, batched; backs ACL grantee and bucket-owner validation
/private/resolve-principals	Bucket-policy Principal validation: the gateway posts {"principals": [...]} and the service answers {"invalid": [...]}, listing only the entries that name nothing, so an empty list means the document's principals are all valid

All five routes are POST, and all are signed with the IAM service's root credential. A store fault on /private/resolve-principals is a 500, which the gateway surfaces as an IAM-service error rather than as MalformedPolicy.

Two consequences follow from the gateway holding no secrets. User management through the gateway's Admin API is unsupported — Standalone IAM has the per-command list — and the private endpoint, carrying the whole trust of the deployment, is restricted to a unix socket or mTLS with no plaintext TCP option. It has no health route either; probe it, if you must, with an authenticated POST /private/version, which is exactly what the gateway's startup probe does.

IAM service side

Option	Environment	Notes
--private-ports	VGW_IAM_PRIVATE_PORTS	Private listener address: a unix socket path, or <ip>:<port> / :<port> when mTLS is also configured. Repeatable, and the environment variable is comma-separated
--private-cert, --private-cert-key	VGW_IAM_PRIVATE_CERT, VGW_IAM_PRIVATE_CERT_KEY	Server certificate and private key, as file paths
--private-client-ca	VGW_IAM_PRIVATE_CLIENT_CA	File path of the PEM CA bundle used to verify the S3 gateway's client certificate
--private-socket-perm	VGW_IAM_PRIVATE_SOCKET_PERM	Octal file mode applied to a file-backed unix socket after bind

The private endpoint exists only when --private-ports is set; with no value, no private listener is created at all and no S3 gateway can use this service.

An address is treated as a unix socket path whenever it does not parse as host:port — /run/vgw/iam.sock, ./iam.sock and the Linux abstract-namespace form @vgw-iam are all sockets, while :7443, 127.0.0.1:7443 and [::1]:7443 are TCP. A path containing a colon defeats that heuristic and is misread as TCP, so keep socket paths colon-free.

The three certificate options must be set all together or all left empty, and mTLS is mandatory for a TCP address: every --private-ports address is validated before anything binds, and a TCP address without a complete mTLS configuration aborts startup. Both messages are in the IAM service validation table. The listener negotiates TLS 1.2 or better and requires and verifies a client certificate against the --private-client-ca bundle, but does not examine that certificate's identity beyond the CA check — authorization is the root SigV4 credential the gateway signs every private request with. Both layers must be right.

For a unix-socket private endpoint, leave the three certificate options empty. A unix socket configured together with --private-cert is wrapped in TLS, but the gateway's unix-socket client speaks plain HTTP over the socket and will not connect.

With no --private-socket-perm value, no chmod is performed at all and the socket keeps whatever mode the process umask produced. Set it — 0660 together with a shared group is the usual arrangement — because filesystem permissions are the entire access control for a socket-based private endpoint. It has no effect on TCP or abstract-namespace (@) sockets. Relative socket paths given to --private-ports are not resolved to absolute paths at startup, unlike --port and --webui, so always give --private-ports an absolute path.

S3 gateway side

These nine are global options on the S3 gateway process — versitygw --iam-standalone-endpoint … posix /data, not on the iam command. See Global Options for their help text.

Setting --iam-standalone-endpoint selects the standalone IAM backend and takes precedence over every built-in one (--iam-dir, LDAP, S3, Vault, FreeIPA). The gateway prints initializing standalone IAM with "<endpoint>" at startup.

--iam-standalone-access and --iam-standalone-secret must be set together or both left empty, in which case they default to the gateway's own --access / --secret. The gateway has to sign its private requests with the IAM service's root credential, so that default is only correct when both processes were given the same one; if they were not, set these two explicitly.

Warning

A wrong IAM-client credential is not fatal at startup. The gateway logs a warning and starts, and then every S3 request that needs the IAM service fails with a 500 InternalError. This is deliberate — a rejected credential is reported without retrying — but it means the warning line is the only signal you get.

--iam-standalone-client-cert, --iam-standalone-client-cert-key and --iam-standalone-server-ca are file paths, all required for a TCP endpoint and unused for a unix socket. The gateway verifies the IAM service's certificate with standard Go hostname verification and no override, which is why the service's certificate needs a SAN matching the endpoint you dial — an IP SAN for an IP address endpoint.

Warning

Do not put the private endpoint on port 443. The gateway's SigV4 signing strips a default port from the signed host while the HTTP client still sends Host: iam.example.com:443 on the wire, and the IAM service canonicalizes from the raw header — the signatures will not match. Any other port, such as 7443, is fine.

--iam-standalone-default-uid, --iam-standalone-default-gid and --iam-standalone-default-project-id supply the single POSIX identity the gateway stamps onto every account it resolves through the standalone backend — its own root account included, since bucket ownership is fixed to root here and a bucket has to be chownable to its owner. An AWS IAM user has no uid, gid or project id, so the service has nowhere to keep them per account; the posix and scoutfs backends use this one fixed identity for file ownership and quota accounting, which means per-user file ownership is not available with standalone IAM.

Choose a uid/gid the gateway process can actually chown to. An unprivileged gateway without CAP_CHOWN cannot hand a file to a different uid, and cannot chgrp to a group it is not a member of; a value equal to the gateway's own euid/egid is a legitimate no-op and the chown is skipped. With --chuid/--chgid set to anything else, every CreateBucket and every object write fails, logging:

chown <name> to <uid>:<gid>: operation not permitted: --chuid/--chgid need a privileged gateway, but this one runs as euid <euid>/egid <egid>

Object writes wrap that as set temp file ownership: …; the client sees InternalError at HTTP 500 either way. The gateway prints a warning at startup when --chuid/--chgid are set and it is not running as root. Leaving these options at the default 0 under an unprivileged gateway is exactly that case.

A failed CreateBucket does not poison the bucket name. Any failure after the bucket directory is made — the chown EPERM above included — removes that directory, deletes its sidecar metadata attributes (which may live outside the bucket, under a sidecar directory) and removes its versioning directory before returning. No half-created directory is left behind and no manual rmdir is needed: the name is immediately retryable once --iam-standalone-default-uid/-gid are corrected. Previously the leftover directory made every later request for that name — a retry of CreateBucket included — fail with get bucket acl: no such key until it was removed by hand. Cleanup failures are reported only under --debug, and never replace the original error.

See Standalone IAM for what else changes on the data plane.

Generating test mTLS material

The repository ships genmtlscerts.sh to produce a working mTLS set for development and for the integration test suites:

./genmtlscerts.sh <output-dir> [server-ip]

server-ip defaults to 127.0.0.1. The script writes six files:

File	Role
ca.pem / ca.key	Self-signed CA, CN versitygw-test-ca, trusted by both sides
iam-server.pem / iam-server.key	Private-listener server certificate, CN versitygw-iam-private, extendedKeyUsage = serverAuth, subjectAltName = IP:<server-ip>
gw-client.pem / gw-client.key	Gateway client certificate, CN versitygw-s3-gateway, extendedKeyUsage = clientAuth

genmtlscerts.sh is test material, not production key management: all three certificates are issued with one-day validity and RSA-2048 keys, the CA key is written next to the certificates in the same directory, and nothing rotates. For production, issue the private-endpoint certificates from your own CA or from cert-manager, and keep the CA key out of the deployment. If you dial the private endpoint by hostname rather than by IP, issue the server certificate with a matching DNS SAN instead.

Single host with a unix socket

When the IAM service and the S3 gateway share a host, a unix socket is the simplest and the most secure option: there is no TLS material to manage on either side, and the filesystem is the trust boundary. /run/vgw must exist and be writable by the IAM service, and 0660 plus a shared group is what restricts who may connect. Note that the two processes cannot share a listen address — here the IAM service keeps the :7070 default and the gateway is moved to :7080 — and that both are given the same root credential, so the gateway's --iam-standalone-access / --iam-standalone-secret correctly default to it.

IAM service:

versitygw --access user --secret pass --port :7070 iam \
  --dir /var/lib/vgw-iam \
  --private-ports /run/vgw/iam-private.sock \
  --private-socket-perm 0660

S3 gateway:

versitygw --access user --secret pass --port :7080 \
  --iam-standalone-endpoint /run/vgw/iam-private.sock \
  --iam-standalone-default-uid 1000 \
  --iam-standalone-default-gid 1000 \
  posix /data

Two hosts with mTLS

Across hosts, pods or trust boundaries, mTLS is required. Publish only the public API port (:7070 here) beyond the trust boundary; the private port is for gateways. Both sides must trust the same CA, and iam-server.pem must carry a SAN matching exactly the address given to --iam-standalone-endpoint.

./genmtlscerts.sh /etc/vgw/certs 10.0.0.11    # test material only

On the IAM host (10.0.0.11):

versitygw --access user --secret pass \
  --port :7070 --health /healthz \
  iam \
  --dir /var/lib/vgw-iam \
  --private-ports 10.0.0.11:7443 \
  --private-cert /etc/vgw/certs/iam-server.pem \
  --private-cert-key /etc/vgw/certs/iam-server.key \
  --private-client-ca /etc/vgw/certs/ca.pem

On each S3 gateway host:

versitygw --access user --secret pass --port :7070 \
  --iam-standalone-endpoint 10.0.0.11:7443 \
  --iam-standalone-client-cert /etc/vgw/certs/gw-client.pem \
  --iam-standalone-client-cert-key /etc/vgw/certs/gw-client.key \
  --iam-standalone-server-ca /etc/vgw/certs/ca.pem \
  --iam-standalone-default-uid 1000 \
  --iam-standalone-default-gid 1000 \
  posix /data

---

Private protocol versioning

Every private request and every private response carries a protocol version in the X-Vgw-Private-Protocol header. This is how a gateway and an IAM service at different builds detect each other instead of silently misbehaving.

Item	Value
Header name	X-Vgw-Private-Protocol
Protocol version this build speaks	1
Minimum client protocol this build accepts	1

The gateway sets the header on every request, inside the SigV4 signature. The IAM service sets it on every response, from middleware that runs before routing and before authentication — so it is present on success responses, on application errors, on refusals, and on 404s for unknown routes alike. Checking ahead of authentication is deliberate: a version refusal should be reported as a version refusal, not as a misleading 403 about the credential.

A version header that is absent or unreadable is rejected outright, never assumed to be some default, because a peer whose protocol cannot be read is not one whose answers should be interpreted as IAM decisions. The value must be 1 to 4 ASCII digits parsing to a number ≥ 1, so leading or trailing whitespace, a + sign, 1.0 and v1 are all rejected. A rejected request gets HTTP 400 with code ProtocolMismatch:

{"code":"ProtocolMismatch","error":"malformed X-Vgw-Private-Protocol header \"0\""}

POST /private/version is exempt from the service's own client-version check, so a gateway the service will not otherwise serve can still learn why it was refused. It is still root-signed like every other private route, which is what makes it a useful probe. Its body is:

{"protocol":1,"minClient":1,"serverVersion":"v1.2.3"}

The gateway startup probe

The S3 gateway calls POST /private/version before it finishes starting. Because the probe is a signed request to a root-authenticated endpoint, one successful round trip proves three things at once: the transport (unix socket connect, or the mTLS handshake), the gateway's own IAM credential, and protocol compatibility. On success the gateway prints:

standalone IAM service "10.0.0.11:7443": version v1.2.3, private protocol 1

The probe retries for 30 seconds at 2 second intervals, with a 10 second timeout on each attempt. Only two conditions are retried, because only two can resolve on their own: a transport failure (socket missing, connection refused, TLS handshake failure, timeout) and a protocol mismatch that could be a rollout in progress. Anything the service answered definitively is reported immediately without retrying.

Condition	Retried	Outcome
Service not listening, socket missing, connection refused	yes	Warning after 30s; the gateway starts
TLS handshake failure	yes	Warning after 30s; the gateway starts
Service speaks an older protocol	yes	Fatal after 30s
Response carries no version header, or an unreadable one	yes	Fatal after 30s
Service refuses this gateway's version (ProtocolMismatch)	yes	Fatal after 30s
Service requires a newer client than this gateway speaks	yes	Fatal after 30s
Gateway credential rejected (403)	no	Warning, immediately; the gateway starts
Any other non-200	no	Warning, immediately; the gateway starts

An unreachable service is only a warning so that a gateway pod and an IAM Deployment can start in parallel without crash-looping — at the cost of a gateway started against a genuinely dead service blocking for the full 30 seconds before it continues.

Important

Upgrade the standalone IAM service before the S3 gateways. The version rule is "not older", not "equal": a gateway accepts a newer IAM service, and refuses an older one. Upgrading the service first therefore never breaks a running gateway, while upgrading gateways first can leave them refusing to start against a service they consider too old.

Even at a matching protocol version, the gateway fails closed on a malformed answer: a decision matrix whose row or column count does not match the resources and actions it asked about is an error, and a decision string it does not recognize is treated as a deny.

---

TLS

--cert and --key enable TLS on the public IAM API, which negotiates TLS 1.2 or better. The private endpoint's certificate is configured separately with --private-cert / --private-cert-key, and the WebUI's separately again with --webui-cert / --webui-key — which fall back to --cert / --key when the WebUI is enabled and neither is given. The messages for a half-configured or unreadable pair are in the IAM service validation table. See TLS for the gateway-wide picture.

Warning

The iam command does not wire SIGHUP certificate reload. A rotated certificate takes effect only on restart. Sending SIGHUP is not a harmless no-op either: the first signal is absorbed, and a second one blocks the signal-handling goroutine, after which SIGINT and SIGTERM no longer stop the process. Restart the service to pick up new certificates; do not use SIGHUP.

---

Startup validation and errors

Every misconfiguration in the IAM service table below is fatal: the message is printed to stderr as a single line and the process exits with status 1. There is no stack trace and no partial start. On the S3 gateway, some conditions are only warnings — the table says which. Troubleshooting indexes the same messages by symptom.

IAM service

Misconfiguration	Message
No --access	root access key is required for IAM API authentication
No --secret	root secret key is required for IAM API authentication
No listen address	no ports specified
--max-connections below 1	max-connections must be positive
--max-requests below 1	max-requests must be positive
Neither --dir nor --vault-endpoint-url	no IAM storer config specified
Both --dir and --vault-endpoint-url	multiple IAM storer configs specified: dir, vault
--dir names a directory that does not exist	init internal IAM storer: write default iam: open /path/iam.json: no such file or directory
--key without --cert	TLS key specified without cert file
--cert without --key	TLS cert specified without key file
--cert / --key unreadable or mismatched	tls: load certs: unable to set certificate: …
--socket-perm is not octal	invalid SocketPerm value "0999": must be an octal integer (e.g. '0660'): …
Some but not all of the three private cert options	--private-cert, --private-cert-key, and --private-client-ca must all be set together, or all left empty for a unix-socket-only private listener
A TCP --private-ports address with no mTLS	private listener ":7443" requires either a unix socket path or mTLS (server cert + client CA); refusing to serve on plain TCP
--private-cert / --private-cert-key unreadable	private listener: load certs: …
--private-client-ca unreadable	private listener: read CA certificate "/path/ca.pem": …
--private-client-ca contains no certificates	private listener: no valid certificates found in "/path/ca.pem"
--private-socket-perm is not octal	invalid PrivateSocketPerm value "rw": must be an octal integer (e.g. '0660'): …
A --private-ports socket path is taken by a non-socket file	path "/run/vgw/iam-private.sock" already exists and is not a socket (mode …)
--vault-role-id without --vault-role-secret	role id and role secret must both be specified
Neither Vault root token nor role id	vault authentication requires either roleid/rolesecret or root token
--vault-client-cert without --vault-client-cert-key	client certificate and client certificate key should both be specified
Vault auth and secret namespaces differ, no root token	approle tokens are namespace scoped. auth namespace "a" and secret storage namespace "b" differ. use the same namespace or authenticate with a root token
Vault AppRole login rejected	init vault IAM storer: approle authentication failure: …

S3 gateway

Misconfiguration	Message	Fatal
--iam-standalone-access without --iam-standalone-secret (or the reverse)	iam standalone: access and secret must both be set, or both left empty to sign with the root account	yes
A TCP --iam-standalone-endpoint without all three certificate options	iam standalone: client-cert, client-cert-key, and server-ca are all required for a TCP endpoint ("10.0.0.11:7443")	yes
Client certificate or key unreadable	iam standalone: load client certificate: …	yes
--iam-standalone-server-ca unreadable or empty of certificates	iam standalone: read CA certificate "/path/ca.pem": … / iam standalone: no valid certificates found in "/path/ca.pem"	yes
IAM service speaks an older protocol	iam standalone: IAM service at "…" speaks private protocol N, this gateway requires M or newer: upgrade the IAM service before the gateway (still incompatible after 30s, so this is a version skew rather than a rollout in progress)	yes
IAM service requires a newer gateway	iam standalone: IAM service at "…" serves private protocol N and newer, this gateway speaks M: upgrade the gateway (still incompatible after 30s, …)	yes
A response arrives with no protocol header	iam standalone: no X-Vgw-Private-Protocol header on the 200 response from … at "…": not a versioned IAM service (still incompatible after 30s, …)	yes
IAM service unreachable, or the gateway's credential rejected	WARNING: iam standalone: could not verify the IAM service at "…": …; the private protocol version is still checked on every request	no

At runtime, an unresolvable access key surfaces to the S3 client as InvalidAccessKeyId and a rejected session token as InvalidToken. Every other private-channel failure — a rejected gateway credential, a per-request protocol mismatch, a transport failure — surfaces as HTTP 500 InternalError, deliberately, so that a gateway misconfiguration is never reported to an end user as "your access key does not exist". When S3 clients start seeing 500s across the board, check the gateway's own log for the warning line above. See Troubleshooting and Debugging.

---

systemd

The packaged systemd template unit, versitygw@.service, accepts iam as a VGW_BACKEND value, so the standalone IAM service runs as a versitygw@<name> instance exactly like a gateway. The RPM and DEB packages install a ready-made IAM environment file at /etc/versitygw.d/example-iam.conf, alongside the gateway's example.conf. Copy it to /etc/versitygw.d/<name>.conf and start that instance:

cp /etc/versitygw.d/example-iam.conf /etc/versitygw.d/iam.conf
# set ROOT_ACCESS_KEY_ID, ROOT_SECRET_ACCESS_KEY and one storage backend
mkdir -p /var/lib/versitygw/iam
systemctl start versitygw@iam
systemctl enable versitygw@iam

The unit reads /etc/versitygw.d/%i.conf as its EnvironmentFile and passes no command-line flags, so every setting must be given as an environment variable. The required minimum is VGW_BACKEND=iam, ROOT_ACCESS_KEY_ID, ROOT_SECRET_ACCESS_KEY and exactly one storage backend — VGW_IAM_DIR, or the VGW_IAM_VAULT_* set. The global-option-before-subcommand ordering rule stated above does not apply here, because there is no command line to order. The shipped example also includes the three VGW_IAM_*OIDC* settings, commented out and defaulting to the secure posture. Options that example-iam.conf does not list — VGW_WEBUI_PORT, VGW_LOG_LEVEL, VGW_HEALTH, VGW_PPROF — still take effect if you add them to the file.

Two operational details:

• VGW_BACKEND_ARG is unused by the iam backend. The unit still passes it as an (empty) positional argument, which the iam subcommand ignores.
• The shipped example-iam.conf sets VGW_IAM_DIR=/var/lib/versitygw/iam, and neither the package nor the service creates that directory. Create it before the first start, or the service exits with init internal IAM storer: write default iam: open /var/lib/versitygw/iam/iam.json: no such file or directory.

---

Kubernetes and Docker

For the container image, set VGW_BACKEND=iam. The entrypoint then assembles versitygw $VGW_ARGS iam $VGW_BACKEND_ARG $VGW_BACKEND_ARGS, so global options belong in VGW_ARGS and iam-subcommand options in VGW_BACKEND_ARG / VGW_BACKEND_ARGS — the same ordering rule as on the command line. Arguments passed to the container directly are handed to the binary and bypass the environment path entirely.

The Helm chart deploys the standalone IAM service as a separate Deployment with iamServer.enabled: true and wires the gateway to it with iam.type: standalone. See Docker for the container and chart reference.

---

See Also

• Standalone IAM
• IAM API Reference
• Global Options
• WebGUI
• Docker
• TLS
• HealthCheck
• Troubleshooting