Deployment on AWS fails waiting for persistent volume

[aktech]

Steps to reproduce:

• Render the aws project:
• Run terraform init then terraform apply in the infrastructure

First I got this:

Error: namespaces "dev" not found

  on .terraform/modules/kubernetes-conda-store-mount/modules/kubernetes/nfs-mount/main.tf line 38, in resource "kubernetes_persistent_volume_claim" "main":
  38: resource "kubernetes_persistent_volume_claim" "main" {

Then I did terraform apply again, then I got the following every-time:

Error: timeout while waiting for state to become 'Bound' (last state: 'Pending', timeout: 5m0s)

  on .terraform/modules/kubernetes-conda-store-server/modules/kubernetes/services/conda-store/main.tf line 1, in resource "kubernetes_persistent_volume_claim" "main":
   1: resource "kubernetes_persistent_volume_claim" "main" {

[aktech]

On terraform destroy:

Error: error deleting subnet (subnet-0513a3d069c573aa4): timeout while waiting for state to become 'destroyed' (last state: 'pending', timeout: 20m0s)

[costrouc]

I ran into this same issue and here is the solution/workaround that I found. I would like for qhub to automatically handle this use case but at the same time I "like" that this is difficult since resizing a pvc will delete the old one thus deleting all users data. Eventually there needs to be a better way to do this... gcp does not support disk resizing but other storage providers do e.g. rook/ceph.

Here is the workaround:

kubectl delete -n dev deployments conda-store-conda-store nfs-server-nfs
kubectl delete -n dev pvc conda-store-dev-share nfs-mount-dev-share
kubectl delete -n dev pv conda-store-dev-share nfs-mount-dev-share

Then reapply terraform deployment and it should create the needed volumes...what I sorta like from this is that we are forcing the users to delete the shared filesystems making losing data harder (kinda a "this bug is a feature :)" ).

[aktech]

Interesting, but the problem I faced with AWS was on a new deployment, which didn't had any cluster already present, so this problem shouldn't happen there, right?

[aktech]

It seems they are still there after running the above commands. Seems like following hacks works:

• https://stackoverflow.com/a/56182934/2496982
• Can't remove terminating persistent volumes (PV) kubernetes/kubernetes#77258 (comment)

[tylerpotts]

Document solution, and how to delete the persistent volume. Will leave the solution as manual so that user data is not automatically deleted.

[filippo82]

Hi all, is there any update on this issue?

[tylerpotts]

@filippo82 There is. With the newer terraform/kubernetes update, persistent volume side can be increased without being deleted/destroyed. I've verified this as of this past week

[filippo82]

Hi @tylerpotts I believe that the issue which @aktech was experiencing in June (and which I was experiencing yesterday too) was related (somehow, I think) to the order of creation of resources by Terraform. This should be now fixed by this #129. First building the Kubernets cluster with terraform apply -auto-approve -target=module.kubernetes -target=module.kubernetes-initialization -target=module.kubernetes-ingress and then everything else with a general terraform apply.

So this is issue is now fixed for me and, I believe, it is fully taken care of by qhub deploy and probably can be closed.

The issue I am having is to terraform destroy the QHub deployment. It has been a nightmare so far :/

I've opened this issue to discuss that: #144

Best,
-Filippo

[tylerpotts]

@filippo82 Thanks for the clarification. I'll close out this issue

[prasunanand]

Solution: The issue is with conda enironment syntax. So the conda environment is not ready. Hence the PV fails. If the environment is taking too long to build, this error may still exist.