scylla reinstallation error

[sowsan]

I was able to successfully install scylla in AKS but after scaling the cluster none of the components/configuration installation was successful. Because of that, I have deleted the deployment using helm and tried to install it again. But i'm stuck here with the below error. I had to manually remove the crd as it was deleting as part of the helm delete scylla. Thanks for your help.

helm install --name scylla ./charts/scylla --wait
Error: validation failed: unable to recognize "": no matches for kind "Trait" in version "core.hydra.io/v1alpha1"

kubectl get trait
error: the server doesn't have a resource type "trait"

[technosophos]

Are you installing the CRDs manually?

$ kubectl create -f charts/scylla/crds/*.yaml

Yesterday was a very large refactor that changed both internals (to match the spec) and incidentals (such as updating the chart to work with Helm 3 Beta 1).

[sowsan]

Thanks for your response. Nope I was just following the instructions in the readme, which is using, helm install --name scylla ./charts/scylla --wait

[sowsan]

Do I need to install manually?

[technosophos]

Are you using Helm 3 Beta 1? helm version

If not, you will need to install the CRDs manually. I should update the README for that.

[sowsan]

Ok , nope.

helm version
Client: &version.Version{SemVer:"v2.13.0", GitCommit:"79d07943b03aea2b76c12644b4b54733bc5958d6", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}

Ok I will try that and update here

[technosophos]

I tried updating the directions here: #100

[sowsan]

After installing the crds manually hitting another issue with helm install. At this time, the issue is scylla pod is stuck at crashloopbackoff state.

scylla-576b549bb4-l6m4n                1/2     CrashLoopBackOff   6          9m3s

Here is the describe dump.

kubectl describe pod scylla-576b549bb4-l6m4n
Name:               scylla-576b549bb4-l6m4n
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               aks-agentpool-41590791-1/10.240.0.5
Start Time:         Tue, 27 Aug 2019 23:18:03 -0400
Labels:             app.kubernetes.io/instance=scylla
                    app.kubernetes.io/name=scylla
                    pod-template-hash=576b549bb4
Annotations:        <none>
Status:             Running
IP:                 10.244.2.32
Controlled By:      ReplicaSet/scylla-576b549bb4
Init Containers:
  devspaces-proxy-init:
    Container ID:   docker://aa7759869e8e8318b62f3e220f34de19d05c15437275265f5d244b591ba7bd91
    Image:          azds/azds-proxy-init:20190806.2
    Image ID:       docker-pullable://azds/azds-proxy-init@sha256:8e8386378c471e11819e4d48812a16b41567483a3c27978b9199c302d887de61
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 27 Aug 2019 23:18:04 -0400
      Finished:     Tue, 27 Aug 2019 23:18:05 -0400
    Ready:          True
    Restart Count:  0
    Environment:
      BASE_NAMESPACES:
    Mounts:             <none>
Containers:
  scylla:
    Container ID:   docker://3240fef9eee20d84a341a5b5397613e492891dfc45f0196751b6396eb6dfd7c1
    Image:          hydraoss/scylla:latest
    Image ID:       docker-pullable://hydraoss/scylla@sha256:26ebe4b4d2a2afe54734fe92a4f87a5f4298c8f3ae1613a487b2da72156254d2
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 27 Aug 2019 23:23:55 -0400
      Finished:     Tue, 27 Aug 2019 23:23:56 -0400
    Ready:          False
    Restart Count:  6
    Liveness:       http-get http://:8080/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:8080/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      KUBERNETES_NAMESPACE:  default (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from scylla-token-87bvv (ro)
  devspaces-proxy:
    Container ID:   docker://76230b1b7f6ac4f8a047442a7af2b62bb15bcc3d099bef9ab7a30c32450fba92
    Image:          azds/azds-proxy:20190806.2
    Image ID:       docker-pullable://azds/azds-proxy@sha256:a2b751f3fee1271077dc98f64c1e37784140162b3c23eee7d3f1d6deef59cee0
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 27 Aug 2019 23:18:12 -0400
    Ready:          True
    Restart Count:  0
    Environment:
      NAMESPACE:                 default
      TRACE_ENDPOINT:            https://azds-2fcea7cf-9c8b-4ab6-8e60-25042d01800b.wus2.azds.io/api/v3.2/traces
      TRACE_VERSION_KEY_HEADER:  x-azds-version-key
      TRACE_VERSION_KEY:         0.1
      LOG_LEVEL:                 0
      AZDS_ENVIRONMENT:          Production
    Mounts:
      /etc/kubernetes from aks-sp (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  scylla-token-87bvv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  scylla-token-87bvv
    Optional:    false
  aks-sp:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes
    HostPathType:
QoS Class:         BestEffort
Node-Selectors:    <none>
Tolerations:       node.kubernetes.io/not-ready:NoExecute for 300s
                   node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                 From                               Message
  ----     ------     ----                ----                               -------
  Normal   Scheduled  10m                 default-scheduler                  Successfully assigned default/scylla-576b549bb4-l6m4n to aks-agentpool-41590791-1
  Normal   Pulled     10m                 kubelet, aks-agentpool-41590791-1  Container image "azds/azds-proxy-init:20190806.2" already present on machine
  Normal   Created    10m                 kubelet, aks-agentpool-41590791-1  Created container
  Normal   Started    10m                 kubelet, aks-agentpool-41590791-1  Started container
  Normal   Started    10m                 kubelet, aks-agentpool-41590791-1  Started container
  Normal   Pulled     10m                 kubelet, aks-agentpool-41590791-1  Container image "azds/azds-proxy:20190806.2" already present on machine
  Normal   Created    10m                 kubelet, aks-agentpool-41590791-1  Created container
  Normal   Created    10m (x3 over 10m)   kubelet, aks-agentpool-41590791-1  Created container
  Normal   Started    10m (x3 over 10m)   kubelet, aks-agentpool-41590791-1  Started container
  Normal   Pulling    10m (x4 over 10m)   kubelet, aks-agentpool-41590791-1  pulling image "hydraoss/scylla:latest"
  Normal   Pulled     10m (x4 over 10m)   kubelet, aks-agentpool-41590791-1  Successfully pulled image "hydraoss/scylla:latest"
  Warning  BackOff    40s (x50 over 10m)  kubelet, aks-agentpool-41590791-1  **Back-off restarting failed container**

Updated by technosophos: Fixed formatting

[wonderflow]

I guess you didn't have any componentschematics.

Just try kubectl get componentschematics , if you can't find any resource, then it's just what I said. Then it was same with #102 and fixed by #103

[sowsan]

Thanks, @wonderflow , Do I need to wait for the fix to be merged to the master or is there a work-around?

[technosophos]

Okay, I cannot reproduce #102 as described, and I've tried it both in cluster and locally. So in the meantime, let me see if this is the same issue or a different one.

Can you please run these commands and dump the output here?

$ kubectl get crds
$ kubectl get componentschematics
$ kubectl get traits
$ kubectl get pods # and find the name of the scylla pod
$ kubectl logs NAME_OF_SCYLLA_POD

[sowsan]

kubectl get crds
NAME CREATED AT
componentinstances.core.hydra.io 2019-08-27T17:57:23Z
components.core.hydra.io 2019-08-27T17:57:23Z
operationalconfigurations.core.hydra.io 2019-08-27T17:57:23Z
scopes.core.hydra.io 2019-08-27T17:57:23Z
traits.core.hydra.io 2019-08-27T17:57:23Z

kubectl get componentschematics
error: the server doesn't have a resource type "componentschematics"

kubectl get traits
NAME AGE
autoscaler 16h
empty 16h
ingress 16h
manual-scaler 16h

kubectl logs scylla-576b549bb4-l6m4n
Error from server (BadRequest): a container name must be specified for pod scylla-576b549bb4-l6m4n, choose one of: [scylla devspaces-proxy] or one of the init containers: [devspaces-proxy-init]

[sowsan]

kubectl get pods
NAME READY STATUS RESTARTS AGE
scylla-576b549bb4-l6m4n 1/2 CrashLoopBackOff 192 16h

[technosophos]

Okay, you have an older version of something. I introduced several breaking changes on Monday in an attempt to match the current version of the Hydra spec. One change was moving from Component to ComponentSchematic. So you'll need to install the updated CRDs.

Can you do kubectl create -f charts/scylla/crds/componentschematics.yaml? You might need to git pull --rebase origin master first to get the latest Helm chart.

[technosophos]

Oops... did not mean to close this until I got verification taht it was solved

[sowsan]

Oops... did not mean to close this until I got verification taht it was solved

it worked. thanks for updating the doc :)