Fix Helm Error 403 on chart repository

[laisbsc]

A user reported getting errors when trying to destroy an old qhub deployment using terraform destroy -auto-approve, more details on the issue on #218.

The probable cause is Helm (see epic #224) and a possible solution can be found at Stack Overflow.

[tylerpotts]

This is resolved. Based on the comments in this PR, my guess is the core problem was the use of helm version 2.x. I was unable to reproduce the problem before merging the above PR.

The PR for the terraform modules was merged in any case to explicitly pull from the latest URL for helm charts.

[laisbsc]

This is resolved. Based on the comments in this PR, my guess is the core problem was the use of helm version 2.x. I was unable to reproduce the problem before merging the above PR.

By the looks of it, he just noticed there was no Helm installed on his dependencies. Check out #218 again.

[rsignell-usgs]

@laisbsc and @tylerpotts , I upgraded helm in my qhub environment:

# Name                    Version                   Build  Channel
kubernetes-helm           3.2.4                h412194d_0    conda-forge

but I'm still getting the error:

Error: failed to fetch https://kubernetes-charts.storage.googleapis.com/index.yaml : 403 Forbidden 

when I try:

cd infrastructure
terraform destroy

Do I need to do terraform render again?

[laisbsc]

Do I need to do terraform render again?

I reckon that could sort out the problem, @rsignell-usgs. Do you mind re-rendering and coming back to us?

Chat soon!

[rsignell-usgs]

@laisbsc , okay, here's what I've done so far:

After upgrading helm, I did:

  qhub render -c qhub-config.yaml -o test -f
  cp qhub-config.yaml test
  cd test
  cd infrastructure 
  terraform destroy

This didn't work, told me I needed to run terraform init, so I did that and then ran terraform destroy again.

This ran for a awhile and then produced this error:

Error: Error deleting IAM Role (earthmap-dev-eks-node-group-role): DeleteConflict: Cannot delete entity, must detach all policies first. 

[rsignell-usgs]

@laisbsc or @tylerpotts , should I just go ahead and start deleting stuff (like this IAM role) using the AWS console, or do I want to try to wait and fix via Terraform or AWS CLI?

[tylerpotts]

@rsignell-usgs I chatted with @costrouc this morning, and he mentioned that he also ran into some problems going from helm 2.x to 3.x. His suggestion is to delete the .helm directory in your home directory and try again. I'd also suggest uninstalling helm 2.x off your computer if you no longer need it

[rsignell-usgs]

@tylerpotts , I deleted the .helm directory and tried again, but still getting:

Error: Error deleting IAM Role (earthmap-dev-eks-node-group-role): DeleteConflict: Cannot delete entity, must detach all policies first.
        status code: 409, request id: d1374ebd-dbc5-48ec-be96-85474c5757e4

Should we keep trying or should I punt and use the console?

[tylerpotts]

Ah, I should have read your error more closely. This doesn't look like a helm error to me. I think this is AWS-ism that you will have to go into console to resolve manually.

In my experience, Terraform doesn't do well if the destroy is interrupted halfway and often requires manual deletion via console.

[rsignell-usgs]

I'm happy to report that:

• I was able to destroy the rest of the infrastructure manually on our old qhub
• I was able to simply retrigger the github action that failed and it looked like it pulled the updated qhub-terraform-modules and it succeeded. So we are back in business over at https://jupyter.qhub.esipfed.org ! (and I added you guys as authorized users if you want to login and check it out)

[laisbsc]

I'm happy to report that

The happiness! <3 We're happy you got that sorted too. Wondering if this deserves a 'Troubleshoot' section on the docs.

So we are back in business over at https://jupyter.qhub.esipfed.org ! (and I added you guys as authorized users if you want to login and check it out)

I went ahead and checked the URL. And damn this is (happy) tear-inducing! The visualizations are so neat!
And I'm also delighted that we (as in Tyler) was able to help with the hiccup.

Take care and chat soon!

[rsignell-usgs]

@jkellndorfer and I are trying to delete our old qhub deployment, and using the above approach bombed out on the terraform destroy command with:

Error: Unsupported argument

  on kubernetes.tf line 2, in provider "kubernetes":
   2:   load_config_file       = false

what now, oh Qhub gurus?

[tylerpotts]

@rsignell-usgs I'd suggest going into that kubernetes.tf file and deleting that load_config_file argument

[rsignell-usgs]

@tylerpotts you da man! commented that out, ran terraform destroy again, got:

Destroy complete! Resources: 67 destroyed.

[tylerpotts]

@rsignell-usgs Great news!

I think the problem is/was that the module for kubernetes has been updated and no longer supports that argument. This is part of the reason we're hoping to more tightly pin dependencies in this next release