bug: install latest apisix via helm install error

[ArenaSu]

Current Behavior

failed to install apisix via helm tool

Expected Behavior

install apisix via helm tool success

Error Logs

kubectl logs -f apisix-etcd-0 -n apisix

etcd 07:55:43.78
etcd 07:55:43.78 Welcome to the Bitnami etcd container
etcd 07:55:43.79 Subscribe to project updates by watching https://github.com/bitnami/containers
etcd 07:55:43.79 Submit issues and feature requests at https://github.com/bitnami/containers/issues
etcd 07:55:43.79
etcd 07:55:43.79 INFO ==> ** Starting etcd setup **
etcd 07:55:43.81 INFO ==> Validating settings in ETCD_* env vars..
etcd 07:55:43.82 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 07:55:43.82 INFO ==> Initializing etcd
etcd 07:55:43.83 INFO ==> Generating etcd config file using env variables
etcd 07:55:43.85 INFO ==> There is no data from previous deployments
etcd 07:55:43.85 INFO ==> Bootstrapping a new cluster
etcd 07:56:43.93 ERROR ==> Headless service domain does not have an IP per initial member in the cluster

Steps to Reproduce

1. create three local type pv
2. helm repo add apisix https://charts.apiseven.com
3. helm repo update
4. helm install apisix apisix/apisix --create-namespace --namespace apisix

initial service info below:

# kubectl get svc -n apisix -o wide
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE   SELECTOR
apisix-admin           ClusterIP   12.239.47.15     <none>        9180/TCP            91s   app.kubernetes.io/instance=apisix,app.kubernetes.io/name=apisix
apisix-etcd            ClusterIP   12.239.36.181    <none>        2379/TCP,2380/TCP   91s   app.kubernetes.io/instance=apisix,app.kubernetes.io/name=etcd
apisix-etcd-headless   ClusterIP   None             <none>        2379/TCP,2380/TCP   91s   app.kubernetes.io/instance=apisix,app.kubernetes.io/name=etcd
apisix-gateway         NodePort    12.239.120.215   <none>        80:30822/TCP        91s   app.kubernetes.io/instance=apisix,app.kubernetes.io/name=apisix

pod info below:

# kubectl get pod -n apisix -o wide
NAME                      READY   STATUS     RESTARTS        AGE   IP               NODE    NOMINATED NODE   READINESS GATES
apisix-6bb8f77b8d-4x4nx   0/1     Init:0/1   0               70m   12.240.176.184   node2   <none>           <none>
apisix-etcd-0             0/1     Running    7 (7m15s ago)   18m   12.240.176.191   node2   <none>           <none>
apisix-etcd-1             0/1     Running    7 (7m20s ago)   18m   12.240.176.192   node2   <none>           <none>
apisix-etcd-2             0/1     Running    7 (7m15s ago)   18m   12.240.176.193   node2   <none>           <none>

pv info below:

# kubectl get pv -n apisix
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                       STORAGECLASS   REASON   AGE
vol-1   8Gi        RWO            Retain           Bound    apisix/data-apisix-etcd-2                           107m
vol-2   8Gi        RWO            Retain           Bound    apisix/data-apisix-etcd-0                           107m
vol-3   8Gi        RWO            Retain           Bound    apisix/data-apisix-etcd-1                           107m

pvc info below:

# kubectl get pvc -n apisix
NAME                 STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-apisix-etcd-0   Bound    vol-2    8Gi        RWO                           124m
data-apisix-etcd-1   Bound    vol-3    8Gi        RWO                           124m
data-apisix-etcd-2   Bound    vol-1    8Gi        RWO                           124m

pv yaml below:

# cat pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: vol-1
  namespace: apisix
  labels:
    type: local
spec:
  capacity:
    storage: 8Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/data/vol1"
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: vol-2
  namespace: apisix
  labels:
    type: local
spec:
  capacity:
    storage: 8Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/data/vol2"
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: vol-3
  namespace: apisix
  labels:
    type: local
spec:
  capacity:
    storage: 8Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/data/vol3"

Environment

• APISIX version (run apisix version):2.15.0
• Operating system (run uname -a): Centos 7.9

Linux node1 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 8.3.4 (https://charts.bitnami.com/bitnami)
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[tzssangglass]

cc @tao12345666333 @AlinsRan

[tao12345666333]

can you show me your kubectl -n apisix get ep result?

[ArenaSu]

can you show me your kubectl -n apisix get ep result?

@tao12345666333

# kubectl -n apisix get ep
NAME                   ENDPOINTS                                                                 AGE
apisix-admin                                                                                     23h
apisix-etcd                                                                                      23h
apisix-etcd-headless   12.240.176.196:2380,12.240.176.197:2380,12.240.176.198:2380 + 3 more...   23h
apisix-gateway                                                                                   23h
[root@node1 kubearena]# kubectl get po -n apisix -o wide
NAME                      READY   STATUS             RESTARTS          AGE   IP               NODE    NOMINATED NODE   READINESS GATES
apisix-6bb8f77b8d-d5snt   0/1     Init:0/1           0                 23h   12.240.176.195   node2   <none>           <none>
apisix-etcd-0             0/1     Running            260 (7m36s ago)   23h   12.240.176.196   node2   <none>           <none>
apisix-etcd-1             0/1     Running            261 (2m46s ago)   23h   12.240.176.197   node2   <none>           <none>
apisix-etcd-2             0/1     CrashLoopBackOff   260 (3m5s ago)    23h   12.240.176.198   node2   <none>           <none>

[tao12345666333]

please provide apisix-etcd-2 's log

[ArenaSu]

please provide apisix-etcd-2 's log

@tao12345666333

# kubectl logs apisix-etcd-2 -n apisix
etcd 10:02:23.33
etcd 10:02:23.34 Welcome to the Bitnami etcd container
etcd 10:02:23.34 Subscribe to project updates by watching https://github.com/bitnami/containers
etcd 10:02:23.34 Submit issues and feature requests at https://github.com/bitnami/containers/issues
etcd 10:02:23.34
etcd 10:02:23.35 INFO  ==> ** Starting etcd setup **
etcd 10:02:23.37 INFO  ==> Validating settings in ETCD_* env vars..
etcd 10:02:23.37 WARN  ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 10:02:23.38 INFO  ==> Initializing etcd
etcd 10:02:23.38 INFO  ==> Generating etcd config file using env variables
etcd 10:02:23.40 INFO  ==> There is no data from previous deployments
etcd 10:02:23.40 INFO  ==> Bootstrapping a new cluster
etcd 10:02:23.42 INFO  ==> Obtaining cluster member ID
etcd 10:02:23.42 INFO  ==> Starting etcd in background
etcd 10:04:26.90 INFO  ==> ** etcd setup finished! **

etcd 10:04:26.92 INFO  ==> ** Starting etcd **
{"level":"info","ts":"2022-09-27T10:04:26.958Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ADVERTISE_CLIENT_URLS","variable-value":"http://apisix-etcd-2.apisix-etcd-headless.apisix.svc.cluster.local:2379,http://apisix-etcd.apisix.svc.cluster.local:2379"}
{"level":"info","ts":"2022-09-27T10:04:26.958Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTH_TOKEN","variable-value":"jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m"}
{"level":"info","ts":"2022-09-27T10:04:26.958Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_TLS","variable-value":"false"}
{"level":"info","ts":"2022-09-27T10:04:26.958Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CLIENT_CERT_AUTH","variable-value":"false"}
{"level":"info","ts":"2022-09-27T10:04:26.958Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_DATA_DIR","variable-value":"/bitnami/etcd/data"}
{"level":"info","ts":"2022-09-27T10:04:26.958Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_ADVERTISE_PEER_URLS","variable-value":"http://apisix-etcd-2.apisix-etcd-headless.apisix.svc.cluster.local:2380"}
{"level":"info","ts":"2022-09-27T10:04:26.958Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER","variable-value":"apisix-etcd-0=http://apisix-etcd-0.apisix-etcd-headless.apisix.svc.cluster.local:2380,apisix-etcd-1=http://apisix-etcd-1.apisix-etcd-headless.apisix.svc.cluster.local:2380,apisix-etcd-2=http://apisix-etcd-2.apisix-etcd-headless.apisix.svc.cluster.local:2380"}
{"level":"info","ts":"2022-09-27T10:04:26.958Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_STATE","variable-value":"new"}
{"level":"info","ts":"2022-09-27T10:04:26.958Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_TOKEN","variable-value":"etcd-cluster-k8s"}
{"level":"info","ts":"2022-09-27T10:04:26.958Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_CLIENT_URLS","variable-value":"http://0.0.0.0:2379"}
{"level":"info","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_PEER_URLS","variable-value":"http://0.0.0.0:2380"}
{"level":"info","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LOG_LEVEL","variable-value":"info"}
{"level":"info","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_NAME","variable-value":"apisix-etcd-2"}
{"level":"info","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_PEER_AUTO_TLS","variable-value":"false"}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TRUSTED_CA_FILE="}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISABLE_STORE_MEMBER_ID=no"}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_FILE=/opt/bitnami/etcd/conf/etcd.yaml"}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_ON_K8S=yes"}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SNAPSHOTS_DIR=/snapshots"}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BIN_DIR=/opt/bitnami/etcd/bin"}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_VOLUME_DIR=/bitnami/etcd"}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_ROOT_PASSWORD="}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CLUSTER_DOMAIN=apisix-etcd-headless.apisix.svc.cluster.local"}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISASTER_RECOVERY=no"}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_KEY_FILE="}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_DIR=/opt/bitnami/etcd/conf"}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_GROUP=etcd"}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_START_FROM_SNAPSHOT=no"}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOT_FILENAME="}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot"}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISABLE_PRESTOP=no"}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TMP_DIR=/opt/bitnami/etcd/tmp"}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BASE_DIR=/opt/bitnami/etcd"}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CERT_FILE="}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs"}
{"level":"warn","ts":"2022-09-27T10:04:26.959Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_USER=etcd"}
{"level":"info","ts":"2022-09-27T10:04:26.960Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd"]}
{"level":"info","ts":"2022-09-27T10:04:26.961Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["http://0.0.0.0:2380"]}
{"level":"info","ts":"2022-09-27T10:04:26.962Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:2379"]}
{"level":"info","ts":"2022-09-27T10:04:26.962Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.4","git-sha":"08407ff76","go-version":"go1.16.15","go-os":"linux","go-arch":"amd64","max-cpu-set":16,"max-cpu-available":16,"member-initialized":false,"name":"apisix-etcd-2","data-dir":"/bitnami/etcd/data","wal-dir":"","wal-dir-dedicated":"","member-dir":"/bitnami/etcd/data/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://apisix-etcd-2.apisix-etcd-headless.apisix.svc.cluster.local:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://apisix-etcd-2.apisix-etcd-headless.apisix.svc.cluster.local:2379","http://apisix-etcd.apisix.svc.cluster.local:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"apisix-etcd-0=http://apisix-etcd-0.apisix-etcd-headless.apisix.svc.cluster.local:2380,apisix-etcd-1=http://apisix-etcd-1.apisix-etcd-headless.apisix.svc.cluster.local:2380,apisix-etcd-2=http://apisix-etcd-2.apisix-etcd-headless.apisix.svc.cluster.local:2380","initial-cluster-state":"new","initial-cluster-token":"etcd-cluster-k8s","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2022-09-27T10:04:26.962Z","caller":"embed/etcd.go:368","msg":"closing etcd server","name":"apisix-etcd-2","data-dir":"/bitnami/etcd/data","advertise-peer-urls":["http://apisix-etcd-2.apisix-etcd-headless.apisix.svc.cluster.local:2380"],"advertise-client-urls":["http://apisix-etcd-2.apisix-etcd-headless.apisix.svc.cluster.local:2379","http://apisix-etcd.apisix.svc.cluster.local:2379"]}
{"level":"info","ts":"2022-09-27T10:04:26.962Z","caller":"embed/etcd.go:370","msg":"closed etcd server","name":"apisix-etcd-2","data-dir":"/bitnami/etcd/data","advertise-peer-urls":["http://apisix-etcd-2.apisix-etcd-headless.apisix.svc.cluster.local:2380"],"advertise-client-urls":["http://apisix-etcd-2.apisix-etcd-headless.apisix.svc.cluster.local:2379","http://apisix-etcd.apisix.svc.cluster.local:2379"]}
{"level":"warn","ts":"2022-09-27T10:04:26.962Z","caller":"etcdmain/etcd.go:146","msg":"failed to start etcd","error":"cannot access data directory: mkdir /bitnami/etcd/data: permission denied"}
{"level":"fatal","ts":"2022-09-27T10:04:26.962Z","caller":"etcdmain/etcd.go:204","msg":"discovery failed","error":"cannot access data directory: mkdir /bitnami/etcd/data: permission denied","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/main.go:40\nmain.main\n\t/go/src/go.etcd.io/etcd/release/etcd/server/main.go:32\nruntime.main\n\t/go/gos/go1.16.15/src/runtime/proc.go:225"}

[ArenaSu]

So the reason is hostPath dir mod ?I'll try to change dir mod by chmod cmd.

[zhoukewei9700]

@ArenaSu how did you finally fix this error?