maya-apiserver hitting panic / runtime error when POSTing volume #3064
Assignees
Labels
Community
Community Reported Issue
Comments
|
It might be worth noting that I'd been running OpenEBS before this without having enabled iscsid on the host (it's enabled when this error happens) - could be something to do with artifacts created under that condition. |
|
Checking on this. |
|
Here are the YAML resources for the StorageClasses / StoragePoolClaims: apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: openebs-bulk
annotations:
openebs.io/cas-type: cstor
cas.openebs.io/config: |
- name: StoragePoolClaim
value: "cstor-bulk-pool"
- name: ReplicaCount
value: "1"
provisioner: openebs.io/provisioner-iscsi
---
apiVersion: openebs.io/v1alpha1
kind: StoragePoolClaim
metadata:
name: cstor-bulk-pool
namespace: openebs
annotations:
cas.openebs.io/config: |
- name: PoolResourceRequests
value: |-
memory: 100Mi
- name: PoolResourceLimits
value: |-
memory: 4Gi
spec:
name: cstor-bulk-pool
type: disk
poolSpec:
poolType: striped
blockDevices:
blockDeviceList:
- blockdevice-9d50e42f15fe83269356445eea81b1e2
---
apiVersion: openebs.io/v1alpha1
kind: StoragePoolClaim
metadata:
name: cstor-work-pool
namespace: openebs
annotations:
cas.openebs.io/config: |
- name: PoolResourceRequests
value: |-
memory: 100Mi
- name: PoolResourceLimits
value: |-
memory: 4Gi
spec:
name: cstor-work-pool
type: disk
poolSpec:
poolType: striped
blockDevices:
blockDeviceList:
- blockdevice-50bfc93f4a03ff39b6219fed5965fe7b
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: openebs-work
annotations:
openebs.io/cas-type: cstor
cas.openebs.io/config: |
- name: StoragePoolClaim
value: "cstor-work-pool"
- name: ReplicaCount
value: "1"
provisioner: openebs.io/provisioner-iscsiAnd here's the blocking PVC in question: apiVersion: v1
items:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
volume.beta.kubernetes.io/storage-provisioner: openebs.io/provisioner-iscsi
creationTimestamp: "2020-06-15T22:13:22Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
app: prometheus
prometheus: prometheus-operator-prometheus
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:volume.beta.kubernetes.io/storage-provisioner: {}
f:labels:
.: {}
f:app: {}
f:prometheus: {}
f:spec:
f:accessModes: {}
f:resources:
f:requests:
.: {}
f:storage: {}
f:storageClassName: {}
f:volumeMode: {}
f:status:
f:phase: {}
manager: kube-controller-manager
operation: Update
time: "2020-06-15T22:13:25Z"
name: prometheus-prometheus-operator-prometheus-db-prometheus-prometheus-operator-prometheus-0
namespace: prometheus
resourceVersion: "787930"
selfLink: /api/v1/namespaces/prometheus/persistentvolumeclaims/prometheus-prometheus-operator-prometheus-db-prometheus-prometheus-operator-prometheus-0
uid: 46cf6742-d3ee-481c-ae06-1a1cf3890666
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: openebs-bulk
volumeMode: Filesystem
status:
phase: Pending
kind: List
metadata:
resourceVersion: ""
selfLink: "" |
|
There are two issues:
|
|
I tried deleting and re-creating the storage pools / classes, and now I'm seeing errors like this from the apiserver: |
|
For the Prometheus chart, can you try to set the metadata.name to allow for < 63 chars. Related issue: helm/charts#13170 |
|
Regarding the deletion and re-creation, are the block devices in unclaimed state. |
|
The block devices are in the "Released" state; how would I transition them to Unclaimed? Delete them and restart the ndm Pod? |
|
they should transition from Released to Unclaimed automatically. There is a job launched to clear any traces of older cstor pool prior to marking them as unclaimed. the ndm-operator logs should help to check why they are still in the "Released" state. |
|
In my case, I have assigned value of My pod |
|
I used a |
|
This looks a lot like #1688: I'll see if Kubic isn't missing UPDATE: yeop |
|
Okay, I installed e2fsprogs and rebooted, and for a brief, beautiful moment, I think I actually did see the pod get scheduled. Then at 09:30Z, almost on the dot, the kube API server died. Here are the kubelet logs from when that happened: I can't tell if the time was a coincidence, or if there's something in here the suggests what might have gone wrong. After rebooting, the API server started falling into a crash loop again. Here are the logs for one recent run, containing a few panic invocations that might explain why it's getting overwhelmed ( |
stuartpb
changed the title
7 tasks
|
Well, I've decided I'm going to try bringing the cluster back up again from scratch and see if that somehow fixes it. If not, I'll do more digging and see if I can trace where the problem's coming from. For now, I'll go ahead and consider the original issue addressed - thanks! |
stuartpb
changed the title
|
The following workaround is working for me.. My changes are below. Added In AlertManager PVC spec. In above snippet, added below entries In Prometheus PVC spec. In above snippet, added below entries After executing with above modification, able to provision both pods. Output PVC: SVC: The above entries will help to restrict number of characters for Pods and PVCs. |
resolved, thanks. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
I just installed OpenEBS, and I've been tracking down a bug in my cluster where pods aren't getting scheduled because their volumes aren't getting provisioned. The trail goes cold at the logs for openebs-apiserver:
Your Environment
kubectl get nodes:kubectl get pods --all-namespaces:kubectl get services -A:kubectl get sc:kubectl get pv -A:No resources foundkubectl get pvc -A:/etc/os-release): openSUSE MicroOS (Kubic)uname -a):Linux studtop 5.7.1-1-default #1 SMP Wed Jun 10 11:53:46 UTC 2020 (6a549f6) x86_64 x86_64 x86_64 GNU/LinuxThe text was updated successfully, but these errors were encountered: