Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unable to start pega-stream-0 #106

Closed
gsharma239 opened this issue Mar 6, 2020 · 4 comments
Closed

unable to start pega-stream-0 #106

gsharma239 opened this issue Mar 6, 2020 · 4 comments
Labels
bug

Comments

@gsharma239
Copy link

@gsharma239 gsharma239 commented Mar 6, 2020

Hi Team,

  I am trying to install pega on our AKS Kubernetes Cluster but I am getting multiple errors before I can run any of the web or batch version. While debugging, what I've observed is that the 'pega-stream-0' fails first and then rest of the images go in a 'CrashLoopBackOff' , here is how the output from the shell looks like 
ubuntu@Pega:~/aks-demo$ kubectl get pods -n mypega
NAME                          READY   STATUS              RESTARTS   AGE
mypega-cassandra-0            1/1     Running             0          12m
mypega-cassandra-1            0/1     ContainerCreating   0          6m21s
pega-batch-79b78bfd7f-b78dx   0/1     CrashLoopBackOff    5          12m
pega-search-0                 1/1     Running             0          12m
pega-stream-0                 0/1     CrashLoopBackOff    6          12m
pega-web-69b559499-ql8j7      0/1     CrashLoopBackOff    6          12m

Here is the description of the pod :

Events:
  Type     Reason                  Age                    From                                        Message
  ----     ------                  ----                   ----                                        -------
  Warning  FailedScheduling        9m9s (x3 over 9m14s)   default-scheduler                           pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
  Normal   Scheduled               9m5s                   default-scheduler                           Successfully assigned mypega/pega-stream-0 to aks-agentpool-15155817-vmss000001
  Normal   SuccessfulAttachVolume  5m17s                  attachdetach-controller                     AttachVolume.Attach succeeded for volume "pvc-8a59c95f-5fdd-11ea-9878-7a7ea9d448f2"
  Warning  FailedMount             4m45s (x2 over 7m1s)   kubelet, aks-agentpool-15155817-vmss000001  Unable to mount volumes for pod "pega-stream-0_mypega(8a5cb7b0-5fdd-11ea-9878-7a7ea9d448f2)": timeout expired waiting for volumes to attach or mount for pod "mypega"/"pega-stream-0". list of unmounted volumes=[pega-stream]. list of unattached volumes=[pega-stream pega-volume-config pega-volume-credentials default-token-2zkmb]
  Warning  FailedMount             3m51s (x8 over 4m55s)  kubelet, aks-agentpool-15155817-vmss000001  MountVolume.WaitForAttach failed for volume "pvc-8a59c95f-5fdd-11ea-9878-7a7ea9d448f2" : Cannot find Lun for disk kubernetes-dynamic-pvc-8a59c95f-5fdd-11ea-9878-7a7ea9d448f2

I've verified the volume does exist on the cluster:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                              STORAGECLASS   REASON   AGE
pvc-6e6c3656-5fde-11ea-9878-7a7ea9d448f2   10Gi       RWO            Delete           Bound    mypega/data-mypega-cassandra-1     default                 11m
pvc-8a59c95f-5fdd-11ea-9878-7a7ea9d448f2   5Gi        RWO            Delete           Bound    mypega/pega-stream-pega-stream-0   default                 17m
 

Please if someone could help identify the issue.

Regards,
Gaurav

@dcasavant
Copy link
Member

@dcasavant dcasavant commented Mar 6, 2020

It looks like you may be encountering this issue: kubernetes/kubernetes#69262. Are you able to verify the Kubernetes version you are using on AKS? They recommend updating to 1.15.0.

@DeepakYadav17
Copy link

@DeepakYadav17 DeepakYadav17 commented Mar 12, 2020

Yes, we are using AKS. We've now upgraded to 1.15.7, I have added the logs for the pods that are failing, its showing "invalid action '' passed" for the failing pods:

root@Pega:/home/ubuntu# kubectl get pods -n mypega
NAME                          READY   STATUS             RESTARTS   AGE
mypega-cassandra-0            1/1     Running            0          16h
mypega-cassandra-1            1/1     Running            0          16h
mypega-cassandra-2            1/1     Running            0          15h
pega-batch-7548864757-5kc4q   0/1     CrashLoopBackOff   15         16h
pega-db-install-qbf95         0/1     Completed          0          16h
pega-search-0                 1/1     Running            0          16h
pega-stream-0                 0/1     CrashLoopBackOff   15         16h
pega-web-784476df-hq94g       0/1     CrashLoopBackOff   15         16h
root@Pega:/home/ubuntu# kubectl logs pega-batch-7548864757-5kc4q -n mypega
Checking valid action
Invalid action '' passed.
Valid actions are :  install, upgrade, install-deploy, upgrade-deploy, pre-upgrade, post-upgrade
root@Pega:/home/ubuntu# kubectl logs pega-stream-0 -n mypega
Checking valid action
Invalid action '' passed.
Valid actions are :  install, upgrade, install-deploy, upgrade-deploy, pre-upgrade, post-upgrade
root@Pega:/home/ubuntu# kubectl logs pega-web-784476df-hq94g -n mypega
Checking valid action
Invalid action '' passed.
Valid actions are :  install, upgrade, install-deploy, upgrade-deploy, pre-upgrade, post-upgrade
root@Pega:/home/ubuntu# kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"c83d931fb9bece427bc63a02349755e0f8696d3e", GitTreeState:"clean", BuildDate:"2020-01-31T20:09:49Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
root@Pega:/home/ubuntu#

PVC seems to be fine from below:


root@Pega:/home/ubuntu# kubectl get pvc -n mypega
NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-mypega-cassandra-0     Bound    pvc-1d2ddad2-e01e-4eb1-a378-2da50fec1527   10Gi       RWO            default        16h
data-mypega-cassandra-1     Bound    pvc-6475a758-8c88-448c-9a68-ee903fc2ec0b   10Gi       RWO            default        16h
data-mypega-cassandra-2     Bound    pvc-569d82c6-7367-46a5-b7b5-5834ee17dcbd   10Gi       RWO            default        16h
esstorage-pega-search-0     Bound    pvc-d10274d4-58ae-4a42-89bb-aedd0ba73007   5Gi        RWO            default        16h
pega-stream-pega-stream-0   Bound    pvc-6b8b00ca-5b2b-41f3-9536-01be380001c0   5Gi        RWO            default        16h
root@Pega:/home/ubuntu#

Is this a known issue or are we missing on something?

@dcasavant
Copy link
Member

@dcasavant dcasavant commented Mar 26, 2020

This appears related to #96. I have been unable to reproduce this. Can you please open a request with Pega Global Client Support so we can debug further with you? You can reference this issue in your support ticket.

@dcasavant dcasavant added the bug label Mar 26, 2020
@dcasavant
Copy link
Member

@dcasavant dcasavant commented Apr 1, 2020

Closing due to inactivity. Followup is suggested through Pega Global Client Support.

@dcasavant dcasavant closed this Apr 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.