Internal error occurred: failed calling webhook "deployment.admission.stash.appscode.com": the server is currently unable to handle the request

[lorenzomorandini]

Hi,
sometimes stash doesn't allow us to create new deployments in our cluster (even if the deployment doesn't have anything to do with stash and backups).
The error is the following:

Error from server (InternalError): error when creating "manifest.yaml": Internal error occurred: failed calling webhook "deployment.admission.stash.appscode.com": the server is currently unable to handle the request

The commands from #510 allow us to fix the problem but sometimes it reappears. I couldn't find a reason, it seems to appear randomly.

[hossainemruz]

@lorenzomorandini Stash uses mutating webhook that allow us to insert sidecar while creating a workload if there is Restic configured for this workload. Every workload(Deployment,StatefulSet,DaemonSet etc.) goes through this mutation webhook. So, if stash operator pod is unhealthy then you would become unable to create a workload. The workaround you have specified remove those webhooks.

[hossainemruz]

The commands from #510 allow us to fix the problem but sometimes it reappears. I couldn't find a reason, it seems to appear randomly.

It shouldn't re-appear randomly. If you create those webhooks again and Stash operator pod is unhealthy only then the problem should re-appear.

Can you please share following information?

1. What version of Stash are you using?
2. What is your Kubernetes version?
3. What cloud provider are you using?
4. Log from stash operator pod.

[lorenzomorandini]

The stash operator was in error since he couldn't find the webhooks anymore, I uninstalled and installed stash, seems to be working now. I'm still not sure why it stopped working but if it happens again I'll report back.

1. Stash 0.8.3
2. Kubernetes 1.13.3
3. Self hosted

[tamalsaha]

Dup #709

[davidschrooten]

I have similar problem but the namespace where stash was installed is stuck on deletion. This renders my cluster unresponsive. This can have enormous consequences so perhaps we could build in a fail safe for this.

[tamalsaha]

@davidq2q , a solution will be to apply this change 36fdb9a . This will be part of next release. #726