can not connect the cluster with the cert and muli-addr #87

longquanwjx · 2021-09-22T12:27:42Z

etcd server version 3.4.13,
i write a demo to connect the cluster with the cert, but it get can not cannot connect
i have three nodes

node1

./etcd \
  --data-dir=./data/  \
  --name=etcd0 \
  --cert-file=./etcd.pem \
  --key-file=./etcd-key.pem \
  --trusted-ca-file=./ca.pem \
  --peer-cert-file=./etcd.pem \
  --peer-key-file=./etcd-key.pem \
  --peer-trusted-ca-file=./ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --listen-peer-urls=https://192.168.8.133:2380 \
  --initial-advertise-peer-urls=https://192.168.8.133:2380 \
  --listen-client-urls=https://192.168.8.133:2379,http://127.0.0.1:2379 \
  --advertise-client-urls=https://192.168.8.133:2379 \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster="etcd0=https://192.168.8.133:2380,etcd1=https://192.168.8.134:2380,etcd2=https://192.168.8.135:2380" \
  --initial-cluster-state=new \
  --auto-compaction-mode=periodic \
  --auto-compaction-retention=1 \
  --max-request-bytes=33554432 \
  --quota-backend-bytes=6442450944 \
  --heartbeat-interval=250 \
  --election-timeout=2000

node2

./etcd \
  --data-dir=./data/  \
  --name=etcd1 \
  --cert-file=./etcd.pem \
  --key-file=./etcd-key.pem \
  --trusted-ca-file=./ca.pem \
  --peer-cert-file=./etcd.pem \
  --peer-key-file=./etcd-key.pem \
  --peer-trusted-ca-file=./ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --listen-peer-urls=https://192.168.8.134:2380 \
  --initial-advertise-peer-urls=https://192.168.8.134:2380 \
  --listen-client-urls=https://192.168.8.134:2379,http://127.0.0.1:2379 \
  --advertise-client-urls=https://192.168.8.134:2379 \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster="etcd0=https://192.168.8.133:2380,etcd1=https://192.168.8.134:2380,etcd2=https://192.168.8.135:2380" \
  --initial-cluster-state=new \
  --auto-compaction-mode=periodic \
  --auto-compaction-retention=1 \
  --max-request-bytes=33554432 \
  --quota-backend-bytes=6442450944 \
  --heartbeat-interval=250 \
  --election-timeout=2000

node3

./etcd \
  --data-dir=./data/  \
  --name=etcd2 \
  --cert-file=./etcd.pem \
  --key-file=./etcd-key.pem \
  --trusted-ca-file=./ca.pem \
  --peer-cert-file=./etcd.pem \
  --peer-key-file=./etcd-key.pem \
  --peer-trusted-ca-file=./ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --listen-peer-urls=https://192.168.8.135:2380 \
  --initial-advertise-peer-urls=https://192.168.8.135:2380 \
  --listen-client-urls=https://192.168.8.135:2379,http://127.0.0.1:2379 \
  --advertise-client-urls=https://192.168.8.135:2379 \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster="etcd0=https://192.168.8.133:2380,etcd1=https://192.168.8.134:2380,etcd2=https://192.168.8.135:2380" \
  --initial-cluster-state=new \
  --auto-compaction-mode=periodic \
  --auto-compaction-retention=1 \
  --max-request-bytes=33554432 \
  --quota-backend-bytes=6442450944 \
  --heartbeat-interval=250 \
  --election-timeout=2000

i test with etcdctl, it test ok

knt@knt-20:~/work/3.4.13$ etcdctl --command-timeout=3s  --cacert=./ca.pem --cert=./etcd.pem --key=./etcd-key.pem --endpoints="https://192.168.8.135:2379,https://192.168.8.134:2379,https://192.168.8.133:2379" endpoint status -w=table
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.8.135:2379 |  781976c9acf7bd7 |  3.4.13 |   25 kB |      true |      false |         8 |          9 |                  9 |        |
| https://192.168.8.134:2379 | 82f03684ff969f47 |  3.4.13 |   25 kB |     false |      false |         8 |          9 |                  9 |        |
| https://192.168.8.133:2379 | c7a71eb7058e525c |  3.4.13 |   25 kB |     false |      false |         8 |          9 |                  9 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

i write a demo
but cannot connect with muli-addr

#include <etcd/Client.hpp>
#include <etcd/Response.hpp>
#include <string>
#include <iostream>

int main()
{

        etcd::Client* m = etcd::Client::WithSSL("https://192.168.8.133:2379,https://192.168.8.134:2379", "./ca.pem", "./etcd.pem", "./etcd-key.pem");
        etcd::Response resp = m->set("foo", "test", 15).get();
        printf("error_code %d, error_message %s, action %s\n", resp.error_code(),
           resp.error_message().c_str(), resp.action().c_str());
}

print

this->channel : 0x5629620d38a0error_code 14, error_message Connect Failed, action

when i change to single addr, or, remove cert in cluster ,it is ok

#include <etcd/Client.hpp>
#include <etcd/Response.hpp>
#include <string>
#include <iostream>

int main()
{

        etcd::Client* m = etcd::Client::WithSSL("https://192.168.8.133:2379", "./ca.pem", "./etcd.pem", "./etcd-key.pem");
        etcd::Response resp = m->set("foo", "test", 15).get();
        printf("error_code %d, error_message %s, action %s\n", resp.error_code(),
           resp.error_message().c_str(), resp.action().c_str());
}

print

this->channel : 0x558c44b878a0error_code 0, error_message , action set

The text was updated successfully, but these errors were encountered:

sighingnow · 2021-09-22T16:15:22Z

(I have formatted the description a bit to make it more readable).

I cannot reproduce the error, I guess you are deploying etcd on different nodes (different ip and hostname) using the same SSL keys?

sighingnow · 2021-09-22T16:17:27Z

You could get a pesudo-distributed deployment via

setup the SSL keys

./security-config/setup-ca.sh

start the first etcd instance

etcd \
    --data-dir=./default.etcd0/ \
    --name=etcd0 \
    --cert-file security-config/certs/etcd0.example.com.crt \
    --key-file security-config/private/etcd0.example.com.key \
    --client-cert-auth \
    --trusted-ca-file security-config/certs/ca.crt \
    --peer-cert-file security-config/certs/etcd0.example.com.crt \
    --peer-key-file security-config/private/etcd0.example.com.key \
    --peer-client-cert-auth \
    --peer-trusted-ca-file security-config/certs/ca.crt \
    --listen-peer-urls=https://127.0.0.1:2380 \
    --initial-advertise-peer-urls=https://127.0.0.1:2380 \
    --listen-client-urls=https://127.0.0.1:2379 \
    --advertise-client-urls=https://127.0.0.1:2379 \
    --initial-cluster-token=etcd-cluster \
    --initial-cluster="etcd0=https://127.0.0.1:2380,etcd1=https://127.0.0.1:2480" \
    --initial-cluster-state=new \
    --debug

start the second etcd instance

etcd \
    --data-dir=./default.etcd1/ \
    --name=etcd1 \
    --cert-file security-config/certs/etcd0.example.com.crt \
    --key-file security-config/private/etcd0.example.com.key \
    --client-cert-auth \
    --trusted-ca-file security-config/certs/ca.crt \
    --peer-cert-file security-config/certs/etcd0.example.com.crt \
    --peer-key-file security-config/private/etcd0.example.com.key \
    --peer-client-cert-auth \
    --peer-trusted-ca-file security-config/certs/ca.crt \
    --listen-peer-urls=https://127.0.0.1:2480 \
    --initial-advertise-peer-urls=https://127.0.0.1:2480 \
    --listen-client-urls=https://127.0.0.1:2479 \
    --advertise-client-urls=https://127.0.0.1:2479 \
    --initial-cluster-token=etcd-cluster \
    --initial-cluster="etcd0=https://127.0.0.1:2380,etcd1=https://127.0.0.1:2480" \
    --initial-cluster-state=new \
    --debug

longquanwjx · 2021-09-23T03:12:30Z

yes, i deploy etcd on different nodes (different ip and host), because our production environment is different nodes
i generate the cert using this link
https://github.com/opsnull/follow-me-install-kubernetes-cluster
the ca-config.json file

cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF

the ca-csr.json file

cat > ca-csr.json <<EOF
{
  "CN": "kubernetes-ca",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "opsnull"
    }
  ],
  "ca": {
    "expiry": "876000h"
 }
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

the etcd-csr.json file

cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "172.27.138.251",
    "172.27.137.229",
    "172.27.138.239"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "opsnull"
    }
  ]
}
EOF

cfssl gencert -ca=/opt/k8s/work/ca.pem \
    -ca-key=/opt/k8s/work/ca-key.pem \
    -config=/opt/k8s/work/ca-config.json \
    -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

ls etcd*pem

sighingnow · 2021-09-23T03:18:41Z

When generating the keys have you replaces the "hosts" field in etcd-csr.json with your own hosts?

longquanwjx · 2021-09-23T03:44:06Z

yes, i replaced,
host in my case is
"hosts": [
"127.0.0.1",
"192.168.8.133",
"192.168.8.134",
"192.168.8.135"
]

i test cert with the etcdctl , it can get result success,
and it test cert with single addr ,it can get result success
i think the cert is correct

knt@knt-20:~/work/3.4.13$ etcdctl --command-timeout=3s  --cacert=./ca.pem --cert=./etcd.pem --key=./etcd-key.pem --endpoints="https://192.168.8.135:2379,https://192.168.8.134:2379,https://192.168.8.133:2379" endpoint status -w=table
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.8.135:2379 |  781976c9acf7bd7 |  3.4.13 |   25 kB |      true |      false |         8 |          9 |                  9 |        |
| https://192.168.8.134:2379 | 82f03684ff969f47 |  3.4.13 |   25 kB |     false |      false |         8 |          9 |                  9 |        |
| https://192.168.8.133:2379 | c7a71eb7058e525c |  3.4.13 |   25 kB |     false |      false |         8 |          9 |                  9 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

sighingnow · 2021-09-23T04:55:01Z

I will try the multi-host setting latter.

longquanwjx · 2021-09-23T09:03:25Z

ok, thank you

…with SSL. Resolves etcd-cpp-apiv3#87. Signed-off-by: Tao He <sighingnow@gmail.com>

sighingnow · 2021-09-23T17:04:46Z

Will be fixed by #89, please add a common hostname, e.g., "etcd" here to the "hosts" when self-signing the certificate, and pass it to WithSSL:

etcd::Client* m = etcd::Client::WithSSL("https://192.168.8.133:2379",
                                        "./ca.pem", "./etcd.pem", "./etcd-key.pem", "etcd");

…with SSL. (#89) Resolves #87. Signed-off-by: Tao He <sighingnow@gmail.com>

sighingnow added the bug label Sep 22, 2021

sighingnow added question invalid labels Sep 22, 2021

sighingnow removed the invalid label Sep 23, 2021

sighingnow added a commit to sighingnow/etcd-cpp-apiv3 that referenced this issue Sep 23, 2021

Add an optional target_name_override to support multiple-endpoints …

63b8436

…with SSL. Resolves etcd-cpp-apiv3#87. Signed-off-by: Tao He <sighingnow@gmail.com>

sighingnow mentioned this issue Sep 23, 2021

Add an optional target_name_override to support multiple-endpoints with SSL. #89

Merged

sighingnow closed this as completed in #89 Sep 23, 2021

sighingnow added a commit that referenced this issue Sep 23, 2021

Add an optional target_name_override to support multiple-endpoints …

a949dec

…with SSL. (#89) Resolves #87. Signed-off-by: Tao He <sighingnow@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

can not connect the cluster with the cert and muli-addr #87

can not connect the cluster with the cert and muli-addr #87

longquanwjx commented Sep 22, 2021 •

edited by sighingnow

Loading

sighingnow commented Sep 22, 2021

sighingnow commented Sep 22, 2021

longquanwjx commented Sep 23, 2021 •

edited

Loading

sighingnow commented Sep 23, 2021

longquanwjx commented Sep 23, 2021

sighingnow commented Sep 23, 2021

longquanwjx commented Sep 23, 2021

sighingnow commented Sep 23, 2021

can not connect the cluster with the cert and muli-addr #87

can not connect the cluster with the cert and muli-addr #87

Comments

longquanwjx commented Sep 22, 2021 • edited by sighingnow Loading

sighingnow commented Sep 22, 2021

sighingnow commented Sep 22, 2021

longquanwjx commented Sep 23, 2021 • edited Loading

sighingnow commented Sep 23, 2021

longquanwjx commented Sep 23, 2021

sighingnow commented Sep 23, 2021

longquanwjx commented Sep 23, 2021

sighingnow commented Sep 23, 2021

longquanwjx commented Sep 22, 2021 •

edited by sighingnow

Loading

longquanwjx commented Sep 23, 2021 •

edited

Loading