[Bug][K8s] Helm upgrade inlong failed

[genshengyang]

What happened

helm upgrade inlong failed when volumes.persistence in values.yaml is set to true and volumes.persistence.storageClassName is set to "tke-cfs-storage".

kubectl create namespace inlong
helm upgrade inlong --install -n inlong ./

Error: UPGRADE FAILED: failed to create resource: PersistentVolumeClaim in version "v1" cannot be handled as a PersistentVolumeClaim: v1.PersistentVolumeClaim.Spec: v1.PersistentVolumeClaimSpec.Resources: v1.ResourceRequirements.Requests: unmarshalerDecoder: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP][-+]?[0-9])$', error found in #10 byte of ...|s-storage"}}}}
|..., bigger context ...|orage":"10Gi","storageClassName":"tke-cfs-storage"}}}}
|...

btw, when I changed the value of volumes.persistence.storageClassName to "tke\-cfs\-storage", the Error above disappeared,
but the new error below had occurred .

Error: UPGRADE FAILED: YAML parse error on inlong/templates/tubemq-broker-statefulset.yaml: error converting YAML to JSON: yaml: line 134: found unknown escape character

What you expected to happen

no errors

How to reproduce

volumes.persistence in values.yaml is set to true and volumes.persistence.storageClassName is set to "tke-cfs-storage"

kubectl create namespace inlong
helm upgrade inlong --install -n inlong ./

Environment

kubectl version

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GoVersion:"go1.18.1", Compiler:"gc", Platform:"windows/amd64"}
Kustomize Version: v4.5.4

Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.8-aliyun.1", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

InLong version

1.2.0

InLong Component

Other for not specified component

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[lucaspeng12138]

hello @genshengyang , could you please change docker/kubernetes/templates/mysql-pvc.yaml, to make storageClassName located before accessModes, then try again?

[genshengyang]

@lucaspeng12138 the Error above disappear, but the new error has occurred.

the mysql pod is always Pending

[lucaspeng12138]

@genshengyang could you show describe information about inlong-mysql-0?

[lucaspeng12138]

@genshengyang and describe PersistentVolumeClaim inlong-mysql-data too, thx

[lucaspeng12138]

@genshengyang I think this means user "u-cgshvjautw" have no right to access namespace "default". Could you check the rights?

[genshengyang]

@lucaspeng12138 thx, it mentions lack of resources, I will try later

[lucaspeng12138]

@genshengyang OK. I will submit a pr to fix mysql-pvc.yaml. Please notice me if there is any progress. Thanks a lot.

[genshengyang]

@lucaspeng12138
the mysql pod is still pending with enough resources.

=================================================================
kubectl describe pod inlong-mysql-0 -n inlong

Name: inlong-mysql-0
Namespace: inlong
Priority: 0
Node: cn-hongkong.192.x.x.x/192.x.x.x
Start Time: Fri, 01 Jul 2022 17:52:13 +0800
Labels: app=inlong
cluster=inlong
component=mysql
controller-revision-hash=inlong-mysql-5b44f69fbd
release=inlong
statefulset.kubernetes.io/pod-name=inlong-mysql-0
Annotations: kubernetes.io/psp: ack.privileged
Status: Pending
IP:
IPs:
Controlled By: StatefulSet/inlong-mysql
Containers:
inlong-mysql:
Container ID:
Image: mysql:8.0.28
Image ID:
Port: 3306/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 500m
memory: 512Mi
Environment:
MYSQL_ROOT_PASSWORD: <set to the key 'mysql-password' in secret 'inlong-mysql'> Optional: false
Mounts:
/docker-entrypoint-initdb.d from sql (rw)
/var/lib/mysql from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cb4zh (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: inlong-mysql-data
ReadOnly: false
sql:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: inlong-mysql-data
ReadOnly: false
default-token-cb4zh:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-cb4zh
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Warning FailedScheduling 6m2s default-scheduler running "VolumeBinding" filter plugin for pod "inlong-mysql-0": pod has unbound immediate PersistentVolumeClaims
Warning FailedScheduling 6m2s default-scheduler running "VolumeBinding" filter plugin for pod "inlong-mysql-0": pod has unbound immediate PersistentVolumeClaims
Normal Scheduled 6m default-scheduler Successfully assigned inlong/inlong-mysql-0 to cn-hongkong.192.x.x.x
Warning FailedMount 103s (x2 over 3m58s) kubelet Unable to attach or mount volumes: unmounted volumes=[sql], unattached volumes=[data sql default-token-cb4zh]: timed out waiting for the condition

======================================================================
kubectl describe PersistentVolumeClaim inlong-mysql-data -n inlong

Name: inlong-mysql-data
Namespace: inlong
StorageClass: model-nas-logs-sc
Status: Bound
Volume: nas-64a61608-0920-4ac7-b2ec-xxxxxxxxxxxx
Labels: app=inlong
app.kubernetes.io/managed-by=Helm
chart=inlong-2.0.2
cluster=inlong
component=mysql
heritage=Helm
release=inlong
Annotations: meta.helm.sh/release-name: inlong
meta.helm.sh/release-namespace: inlong
pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: nasplugin.csi.alibabacloud.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 10Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: inlong-mysql-0
inlong-mysql-0
Events:
Type Reason Age From Message

---

Normal ExternalProvisioning 7m53s persistentvolume-controller waiting for a volume to be created, either by external provisioner "nasplugin.csi.alibabacloud.com" or manually created by system administrator
Normal Provisioning 7m53s (x2 over 7m53s) nasplugin.csi.alibabacloud.com_sre-worker-k8s-test04_cf70b729-d474-4eb6-977a-75585c646573 External provisioner is provisioning volume for claim "inlong/inlong-mysql-data"
Normal ProvisioningSucceeded 7m53s (x2 over 7m53s) nasplugin.csi.alibabacloud.com_sre-worker-k8s-test04_cf70b729-d474-4eb6-977a-75585c646573 Successfully provisioned volume nas-64a61608-0920-4ac7-b2ec-xxxxxxxxxxxx

[lucaspeng12138]

@genshengyang I think mysql-pvc is running normally. Is there a PersistentVolume for PersistentVolumeClaim? If exist, may be there are some thing wrong with config. Please use kubectl get pv show access right.

[genshengyang]

@lucaspeng12138
When I deploy inlong with PVC for the first time, the inlong-mysql pod works well.
But when I uninstall the inlong, all PVC are reserved except PVC: inlong-mysql-data .
When I deploy inlong with PVC again, mysql pod is pending and it mentions that " running "VolumeBinding" filter plugin for pod "inlong-mysql-0": pod has unbound immediate PersistentVolumeClaims ".
I have to execute "kubectl delete pvc -n inlong --all" to avoid the error above，but data in PVC: inlong-mysql-data will lose.