Describe the bug
We experienced downtime in one of our production clusters when the tridentorchestrator failed with the error message:
Failed to install Trident; err: command terminated with exit code 1; Error: could not get version: Trident initialization failed; error attempting to clean up volume pvc-a075216f-4f29-47bc-99c6-e52782c1e8c6 from backend fsx01-k8sXX-san: error checking for existing volume: API status: failed, Reason: Volume name: The first character must be a letter or underscore., Code: 13001 (500 Internal Server Error)
No volumes could be mounted by new pods so that our cluster could not scale. It could seem like all operations performed by the tridentorchestrator halted when this error occured and went into a CrashLoopBackoff.
We found the reference to this volume in a tridenttransactions object and when deleted with kubectl and the removal of the finalizer things recovered.
Environment
- Trident version: 22.07
- Trident installation flags used: Trident operator (tridentEnableNodePrep: true)
- Container runtime: Docker 20.10.17-CE
- Kubernetes version: 1.22.17
- Kubernetes orchestrator: kOps 1.25.3
- Kubernetes enabled feature gates: ExpandInUsePersistentVolumes, ExpandCSIVolumes
- OS: Ubuntu 20.04.5 LTS
- NetApp backend types: Amazon FSx for NetApp ONTAP
To Reproduce
Not able to reproduce as we cannot understand what happened.
Expected behavior
Filing this bug report to better understand what happened and what could have caused this. A point could be made for better error handling as this error affected the whole cluster and a better error message would also be helpful.
Thanks!
Describe the bug
We experienced downtime in one of our production clusters when the tridentorchestrator failed with the error message:
No volumes could be mounted by new pods so that our cluster could not scale. It could seem like all operations performed by the tridentorchestrator halted when this error occured and went into a CrashLoopBackoff.
We found the reference to this volume in a tridenttransactions object and when deleted with kubectl and the removal of the finalizer things recovered.
Environment
To Reproduce
Not able to reproduce as we cannot understand what happened.
Expected behavior
Filing this bug report to better understand what happened and what could have caused this. A point could be made for better error handling as this error affected the whole cluster and a better error message would also be helpful.
Thanks!