etcdctl no password with CLI does not work

[vivekpatani]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

I'm trying to use mTLS auth for etcd

Step 1

Here's how I start a server

./bin/etcd \
  --name etcd-node1 \
  --listen-client-urls "https://127.0.0.1:2379/" \
  --advertise-client-urls "https://127.0.0.1:2379/" \
  --listen-peer-urls "http://127.0.0.1:2380/" \
  --initial-advertise-peer-urls "http://127.0.0.1:2380/" \
  --initial-cluster "etcd-node1=http://127.0.0.1:2380/" \
  --cert-file "${SERVER_CERT}" \
  --key-file "${SERVER_KEY}" \
  --client-cert-auth \
  --trusted-ca-file "${CA_CERT}" --log-level debug

Step 2

I created a passwordless user root

./bin/etcdctl user add root --no-password

What did you expect to happen?

To be able to perform passwordless operation.

How can we reproduce it (as minimally and precisely as possible)?

N/A

Anything else we need to know?

I can't perform any etcdctl ops because when
I don't pass a password: etcdserver: user name is empty
When I do pass a password: auth: authentication failed, password was given for no password user
How does one correctly perform any etcdctl operations without a password?

Slack conversation: https://kubernetes.slack.com/archives/C3HD8ARJ5/p1748914427364059

Etcd version (please run commands below)

$ ./etcd --version
etcd Version: 3.5.21
Git SHA: 960a0d374a
Go Version: go1.23.9
Go OS/Arch: darwin/arm64

$ ./etcdctl version
etcdctl version: 3.5.21
API version: 3.5

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

[ahrtr]

Note the passwordless users are for the TLS Common Name based authentication use case only.

Refer to https://etcd.io/docs/v3.6/op-guide/authentication/rbac/#using-tls-common-name

[vivekpatani]

@ahrtr sorry. Let me explain the scenario.

Step 0: Start etcd server

./bin/etcd \
    --name etcd-1 \
    --data-dir /tmp/etcd-1 \
    --listen-client-urls https://127.0.0.1:2379 \
    --advertise-client-urls https://127.0.0.1:2379 \
    --listen-peer-urls http://127.0.0.1:2380 \
    --initial-advertise-peer-urls http://127.0.0.1:2380 \
    --initial-cluster etcd-1=http://127.0.0.1:2380 \
    --initial-cluster-token etcd-token \
    --cert-file ${SERVER_CERT} \
    --key-file ${SERVER_KEY} \
    --trusted-ca-file ${CA_CERT} \
    --client-cert-auth=true \
    --auth-token=simple \
    --client-cert-auth \
    --log-level debug

If you see I have --auth-token: simple is enabled for users with ID/Password.

Step 1: Setup auth, and create user1, and grant READWRITE permissions to that user for prefix /user1/

# create root user
./bin/etcdctl --cacert="${CA_CERT}" --cert="${CLIENT_ROOT_CERT}" --key="${CLIENT_ROOT_KEY}" user add root --no-password
User root created

# grant role root to user root
./bin/etcdctl --cacert="${CA_CERT}" --cert="${CLIENT_ROOT_CERT}" --key="${CLIENT_ROOT_KEY}" user grant-role root root --user root
Password: <press return>
Role root is granted to user root

# enable auth
> ./bin/etcdctl --cacert=${CA_CERT} --cert ${CLIENT_ROOT_CERT} --key ${CLIENT_ROOT_KEY} auth enable --user root
Password: <press return>
Authentication Enabled

> ./bin/etcdctl --cacert=${CA_CERT} --cert ${CLIENT_ROOT_CERT} --key ${CLIENT_ROOT_KEY} user add user1
Password of user1: <type password>
Type password of user1 again for confirmation: <type password>
Password: <press return>
User user1 created

> ./bin/etcdctl --cacert=${CA_CERT} --cert ${CLIENT_ROOT_CERT} --key ${CLIENT_ROOT_KEY} role add user1
Role user1 created

> ./bin/etcdctl --cacert=${CA_CERT} --cert ${CLIENT_ROOT_CERT} --key ${CLIENT_ROOT_KEY} role grant-permission user1 READWRITE /user1/
Role user1 updated

> ./bin/etcdctl --cacert=${CA_CERT} --cert ${CLIENT_ROOT_CERT} --key ${CLIENT_ROOT_KEY} user grant-role user1 user1
Role user1 is granted to user user1

Step 2: Try using legacy auth mechanism after the cluster has transitioned to mTLS from simple token based auth.

./bin/etcdctl --cacert=${CA_CERT} --user user1 put /user1/test bar
Password: <type password>

You see this on the server side

{"level":"warn","ts":"2025-06-11T11:23:03.459915-0700","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:49425","server-name":"","error":"tls: client didn't provide a certificate"}

I have two questions

• How does one successfully migrate from being ID/Password based auth -> mTLS?
• If no certificate was provided, would it be okay to fallback on to passwords for legacy users who still use passwords and presented a valid set of credentials?

Thank you for taking the time to answer this.

[ahrtr]

How does one successfully migrate from being ID/Password based auth -> mTLS?

My understanding is that username/Password based authentication doesn't conflict with TLS Common Name based authentication. You can have two users at the same time, one uses username/password to authenticate (to get a simple or JWT token), the other users TLS Common Name based authentication. Please try this out, and let me know whether it works or not.

If no certificate was provided, would it be okay to fallback on to passwords for legacy users who still use passwords and presented a valid set of credentials?

I think they are two different things. Once --client-cert-auth is enabled ( or you provide a value for --trusted-ca-file), then the client must provide a certificate for communication. This is completely transportation level authentication.

Usually etcd auth is used for authorisation although it also needs to get a token via authentication (Note that the authentication isn't needed for TLS Common Name based authentication).

The following post is somewhat related, it might be useful to you,
https://github.com/ahrtr/etcd-issues/blob/master/docs/token_management.md

[vivekpatani]

let me know whether it works or not.

It does not work. In the setup I described, the root user successfully uses certificates with the correct CN=root, allowing certificate-based operations like creating user1. However, user1 (which only has ID/Password credentials and generates a token) cannot perform any operations.

The issue occurs at the transport layer in

etcd/client/pkg/transport/listener.go

Lines 555 to 558 in 960a0d3

	cfg.ClientAuth = tls.NoClientCert
	if info.TrustedCAFile != "" || info.ClientCertAuth {
	cfg.ClientAuth = tls.RequireAndVerifyClientCert
	}

, where the TLS configuration is set to tls.RequireAndVerifyClientCert when --client-cert-auth is enabled. This causes the TLS handshake to reject any connection that doesn't present a client certificate, preventing the request from reaching the authentication layer where password-based auth could be processed.

Once --client-cert-auth is enabled ( or you provide a value for --trusted-ca-file), then the client must provide a certificate for communication. This is completely transportation level authentication.

I understand this is the current behavior. However, this creates a significant operational challenge for the following migration scenario:

1. Initial State: etcd cluster runs with HTTPS endpoints but without --client-cert-auth, using password-based authentication (--auth-token=simple)
2. Clients Exist: Applications connect using ID/password credentials to port 2379
3. Migration Need: Cluster must be upgraded to use --client-cert-auth for enhanced security
4. Constraint: Zero-downtime requirement - clients cannot all switch to certificates simultaneously during a rolling update

Current Problem: When the cluster is rolled with --client-cert-auth enabled, existing password-based clients immediately lose connectivity because the transport layer rejects their connections before authentication can occur.

Use Case: The ability to support both authentication methods during a transition period would enable:

• Gradual client migration from password to certificate-based auth
• Zero-downtime cluster upgrades
• Backward compatibility during the transition phase

This seems like a common operational pattern for production environments where immediate wholesale client changes aren't feasible. Would the etcd community be open to considering a feature that allows optional client certificates (e.g., tls.RequestClientCert instead of tls.RequireAndVerifyClientCert) with fallback to password authentication when certificates aren't provided?

The authentication flow in

etcd/server/etcdserver/v3_server.go

Lines 994 to 1004 in 960a0d3

	func (s *EtcdServer) AuthInfoFromCtx(ctx context.Context) (*auth.AuthInfo, error) {
	authInfo, err := s.AuthStore().AuthInfoFromCtx(ctx)
	if authInfo != nil || err != nil {
	return authInfo, err
	}
	if !s.Cfg.ClientCertAuthEnabled {
	return nil, nil
	}
	authInfo = s.AuthStore().AuthInfoFromTLS(ctx)
	return authInfo, nil
	}

already has the logic to handle multiple authentication methods, but it never gets reached due to the transport layer restriction.

Thank you taking the time to understand the issue. @ahrtr

[vivekpatani]

Here's how I envision the flow will sort of look like

As of today's etcd the flow roughly looks like

sequenceDiagram
    participant Client
    participant TLS as TLS Layer
    participant Auth as Auth Layer
    participant Server as etcd Server

    Note over Client,Server: Scenario 1: Client with Certificate
    Client->>TLS: Connect with client certificate
    TLS->>TLS: Verify certificate (RequireAndVerifyClientCert)
    TLS->>Auth: Certificate validated, connection established
    Auth->>Auth: Extract cert info via AuthInfoFromTLS()
    Auth->>Server: Request authorized via certificate
    Server->>Client: Success response

    Note over Client,Server: Scenario 2: Client without Certificate
    Client->>TLS: Connect without client certificate
    TLS->>TLS: Check for certificate (RequireAndVerifyClientCert)
    TLS-->>Client: ❌ Connection REJECTED (no certificate)
    Note over Client: Password auth never attempted!

Loading

After the hybrid support of password auth when mTLS auth flags are enabled.

sequenceDiagram
    participant Client
    participant TLS as TLS Layer
    participant Auth as Auth Layer
    participant Server as etcd Server

    Note over Client,Server: Scenario 1: Client with Certificate (unchanged)
    Client->>TLS: Connect with client certificate
    TLS->>TLS: Verify certificate (RequestClientCert)
    TLS->>Auth: Certificate validated, connection established
    Auth->>Auth: Extract cert info via AuthInfoFromTLS()
    Auth->>Server: Request authorized via certificate
    Server->>Client: Success response

    Note over Client,Server: Scenario 2: Client without Certificate (NEW BEHAVIOR)
    Client->>TLS: Connect without client certificate
    TLS->>TLS: Check for certificate (RequestClientCert)
    TLS->>Auth: ✅ Connection allowed (no certificate required)
    Auth->>Auth: Try AuthInfoFromTLS() - returns nil
    Auth->>Auth: Fallback enabled, try AuthInfoFromCtx() for password
    Auth->>Auth: Extract password/token from request headers
    Auth->>Server: Request authorized via password
    Server->>Client: Success response

    Note over Client,Server: Scenario 3: Client without Certificate + Fallback Disabled
    Client->>TLS: Connect without client certificate
    TLS->>TLS: Check for certificate (RequireAndVerifyClientCert)
    TLS-->>Client: ❌ Connection REJECTED (original behavior)

Loading

[ahrtr]

@vivekpatani I was thinking the issue is related to passwordless user based on your original comment, but it seems that you do not mention it anymore. So the following comment is based on your latest comments.

As mentioned in my previous comment #20159 (reply in thread), TLS/certificate is completely transportation layer authentication. Once you enable it, then the client must provide a valid certificate when talking to etcdserver. We don't have a plan to downgrade the level of security (i.e. allow users to specify a different policy such as RequestClientCert or VerifyClientCertIfGiven). I have never received any such complain or request before.

Note that etcd auth is authorization, while TLS/certificate is authentication. They are different concepts, although there is overlap/relationship in etcd's implementation.

Also AFAIK, all cloud provider should also enable --client-cert-auth, cc @tjungblu @fuweid @serathius . I don't think it's a best practice to disable it.

For your case, one workaround that I can think of is to get the certificate configured for you clients first, then upgrade etcd cluster later. But you need to verify it .

[vivekpatani]

etcd auth is for authorization, while TLS/certificate is for authentication (transportation layer),

So in the case of token (ID/Password), what does it mean for etcd to authenticate a user? (Without --client-cert-auth flag)

[ahrtr]

As mentioned previously multiple times, etcd auth is designed for authorization instead of authentication although it needs authentication first to get a token.

Note usually authentication is the precondition of authorization. You need to prove who you are first. If you read the OAuth 2.0, it follows the same pattern. You need to get resource owner's authentication first to get a token, then conduct the authorization process later.

[vivekpatani]

As mentioned previously multiple times, etcd auth is designed for authorization instead of authentication although it needs authentication first to get a token.

Excuse my use of incorrect terminology above. I meant etcd as a server, not etcd as a system. I do understand etcd performs the authorization of the user.

Note usually authentication is the precondition of authorization. You need to prove who you are first. If you read the OAuth 2.0, it follows the same pattern. You need to get resource owner's authentication first to get a token, then conduct the authorization process later.

That makes sense.

In summary, now that I have the understanding in a bit more depth, would the community be interested in having a feature parity with ZooKeeper?

ZooKeeper Client-Server Authentication Context:

If my understanding serves me right, Apache ZooKeeper supports hybrid client-server authentication through its pluggable authentication framework, allowing multiple authentication schemes to coexist for client connections:

• SASL authentication - Strong cryptographic authentication (similar to mTLS certificates)

  • Clients authenticate using Kerberos or other SASL mechanisms
  • Provides mutual authentication between client and server

• Digest authentication - Username/password based (similar to etcd's simple token auth)

  • Clients send username:password credentials
  • Server validates against configured user database

• Hybrid operation - The key feature for migration scenarios:

  • ZooKeeper server can simultaneously accept both SASL and digest authenticated clients
  • No transport-layer rejection - authentication happens at the application layer
  • Clients choose their authentication method independently
  • Enables zero-downtime migration from digest to SASL authentication

This design allows production environments to gradually migrate clients from password-based to certificate-based authentication without service interruption. New clients can use strong SASL authentication while legacy clients continue using digest authentication during the transition period.

The fundamental difference from etcd's current behavior is that ZooKeeper doesn't enforce authentication method uniformity at the transport layer - it accepts the connection first, then authenticates using whatever method the client provides.

It starts with the lowest denomination but upgrades authentication to a more secure version if available.

Would the etcd community be interested in supporting this type of hybrid client-server authentication model to enable similar zero-downtime migration scenarios?

[ahrtr]

pluggable authentication framework

We don't have such a plan. Also it isn't a priority for 3.7 either. Pls refer to https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/roadmap.md#v370

But it's open to discussion for future releases (i.e. 4.0).

[vivekpatani]

Thanks @ahrtr.

[vivekpatani]

I understand it is not on the roadmap, will/are new patches like the one above, being accepted? I understand if they are not, but just wanted to check in before I shelve the idea.

Thank you for the discussion, appreciate it.

[ahrtr]

I understand it is not on the roadmap, will/are new patches like the one above, being accepted?

I am afraid not, at least not in 3.7 time frame. We need to ensure we won't release 3.7.0 4 years later again.