cli: uninstall command #725

ndhanushkodi · 2021-09-15T00:03:48Z

Based off of a demo branch @sadjamz worked on here: https://github.com/hashicorp/consul-k8s-cli/tree/uninstall

Changes proposed in this PR:

consul-k8s uninstall uninstalls the Consul Helm installation with options to wipe all data
If you don't provide the release name/namespace, it will find the release whose metadata.name matches "consul", and then prompt to uninstall that release and any pvcs/secrets based on that.
You can provide the release name/namespace in a case where you have already helm uninstall'ed the consul helm installation, but still have PVCs, Secrets, etc to clean up.
Note that this PR is difficult to unit test-- acceptance tests are coming as the next task after install/uninstall are complete. The Helm steps are difficult to mock out for reasons mentioned in the install PR:
- cli: support install command #713 (comment)
- consul-k8s/cli/cmd/install/install.go
  
  Line 338 in e77da79
  
  // Note that this function is tricky to test because mocking out the action.Configuration struct requires a

How I've tested this PR:
Manual testing of the cases mentioned below, a few unit tests.

How I expect reviewers to test this PR:

Code review
Test it out:
cd cli
go build -o ./bin/consul-k8s
./bin/consul-k8s uninstall
If you can test out the command with the following flags and let me know if you have behaviour suggestions.
a) no flags
b) -name and -namespace set
c) -auto-approve
d) -wipe-all-data

Checklist:

Tests added
CHANGELOG entry added

HashiCorp engineers only, community PRs should not add a changelog entry.
Entries should use present tense (e.g. Add support for...)

cli/README.md

kschoche · 2021-09-15T14:53:21Z

cli/README.md

+  -skip-wipe-data
+      Skip deleting all PVCs, Secrets, and Service Accounts associated with
+      Consul Helm installation without prompting for approval to delete. The
+      default is false.
+
+  -wipe-data
+      Delete all PVCs, Secrets, and Service Accounts associated with Consul
+      Helm installation without prompting for approval to delete. Only use
+      this when persisted data from previous installations is no longer
+      necessary. The default is false.


If skip-wipe-data is defaulted false (meaning wipe the data) and wipe-data is defaulted false (meaning dont wipe the data) what is the actual default behaviour? I'm assuming that we meant that skip-wipe-data is defaulted to true?

cli/cmd/uninstall/uninstall.go

kschoche · 2021-09-15T14:57:59Z

cli/cmd/uninstall/uninstall.go

+	c.once.Do(c.init)
+
+	defer func() {
+		if err := c.Close(); err != nil {


Just out of curiosity what does c.Close() here do? would c.UI.Output() still be usable if c.Close() failed?

👍 that's a good question. I'm curious about this as well. It looks like c.Close attempts to close the UI if the UI implements io.Closer interface, so I'm curious if that means that we won't be able to output to it anymore.

I switched this to using the logger instead. In our case, the UI doesn't implement the closer interface, so Close() would be a no-op. But we could potentially choose to implement it in the future if we say "upgrade" our UI.

cli/cmd/uninstall/uninstall.go

kschoche · 2021-09-15T15:12:57Z

cli/cmd/uninstall/uninstall.go

+		}
+	}
+	if len(secretNames) > 0 {
+		c.UI.Output("Consul secrets deleted.", terminal.WithSuccessStyle())


This is a super nit comment, but if there are 3 secrets and 2 got deleted and the third exits the for loop because you cant delete it for some reason then neither of these Outputs get processed properly. I suppose it is okay since it "should work" in happy path.

Maybe the correct thing to do is output the Error via c.UI.Output() instead of return the error, and then continue to process the other secrets?

I guess this logic applies to the other similar functions, if you choose.

Yeah +1 to Kyle's question. I left a similar question in the deletePVCs function too.

I think its ok because in an error case, it will exit the function at the point of the error without printing either of these outputs here, and just print the first error it got to. Then when a user re-runs uninstall after fixing their issue, it should succeed and delete the secrets properly.

kschoche

This is really great, I left some mostly minor comments but assuming you get time to look at them I think it's good to go!
I did not get a chance to test run the CLI unfortunately.. will get to that soon :)

cli/cmd/install/install.go

cli/README.md

ishustava

This is impressive work!!! I'm so excited about this ❤️ This is a lot of work, and I really appreciate how easy this was to read.

I've only reviewed the command (and not tests) and had some comments that I wanted to clarify before proceeding with the rest of the review. I've also left some minor edits.

I've also ran some manual tests and discussed the issues I ran into with Nitya already but wanted to document them here for posterity:

I think we need to change the helm list functionality to list any pending/uninstalled releases so that if your installation has failed, the uninstall will still find the release.
We need make sure that the kubernetes client is initialized with the correct namespace. With the 0.33.0 release currently, uninstall fails because tls-init-cleanup job doesn't have Namespace set in its template. We've fixed it now in the latest helm release, but we should still make sure that the client is initialized correctly.
It would be nice to default release name and namespace to consul for a case when I forgot to wipe-data in a previous uninstall. That way rerunning uninstall would still just work.

There is also another issue that we haven't discussed that I think would be nice to fix but doesn't have to be part of this PR. In my testing I kept running into errors saying that tls-init-cleanup job or service account etc already exist. It'd be nice if the uninstall command cleaned those up before calling helm delete.

cli/cmd/uninstall/uninstall.go

cli/cmd/common/utils.go

cli/cmd/install/install.go

ishustava · 2021-09-17T18:38:31Z

cli/cmd/uninstall/uninstall.go

+	c.once.Do(c.init)
+
+	defer func() {
+		if err := c.Close(); err != nil {


👍 that's a good question. I'm curious about this as well. It looks like c.Close attempts to close the UI if the UI implements io.Closer interface, so I'm curious if that means that we won't be able to output to it anymore.

cli/cmd/uninstall/uninstall.go

ishustava · 2021-09-17T20:03:20Z

cli/cmd/uninstall/uninstall.go

+		}
+	}
+	if len(secretNames) > 0 {
+		c.UI.Output("Consul secrets deleted.", terminal.WithSuccessStyle())


Yeah +1 to Kyle's question. I left a similar question in the deletePVCs function too.

ishustava · 2021-09-17T20:04:18Z

cli/cmd/uninstall/uninstall.go

+}
+
+// deleteServiceAccounts deletes service accounts that have foundReleaseName in their name.
+func (c *Command) deleteServiceAccounts(foundReleaseName, foundReleaseNamespace string) error {


I'm curious why do we delete only service accounts and not roles, rolebindings, clusterroles, and clusterrolebindings?

removing this function in favor of: https://app.asana.com/0/1200500259727746/1201031492148291/f

Actually in testing I am noticing the consul-tls-init role, rolebinding, and service account are sticking around if they are not manually deleted when installing with -preset=secure. I went ahead and added that logic to delete roles/rolebindings/service accounts. I plan to add logic in the future for jobs, clusterroles, and clusterrolebindings if we see cases those aren't being deleted!

cli/cmd/uninstall/uninstall.go

- delete roles, rolebindings, service accounts by label - use hack for setting namespace for helm kube client - fallback to release name/namespace consul

ishustava

Looks good ❤️ Thank you so much for considering the feedback so thoughtfully 🙏

I've added some minor comments, but they are not blocking.

cli/cmd/uninstall/uninstall.go

cli/cmd/uninstall/uninstall_test.go

cli/cmd/uninstall/uninstall.go

ishustava · 2021-09-22T22:10:40Z

cli/cmd/uninstall/uninstall_test.go

+	rolebindings, err := c.kubernetes.RbacV1().RoleBindings("default").List(context.TODO(), metav1.ListOptions{})
+	require.NoError(t, err)
+	require.Len(t, rolebindings.Items, 0)
+}


I think we can also add tests for the findExistingInstallation function.

This one is kind of tricky for similar reasons to

consul-k8s/cli/cmd/install/install.go

Line 338 in e77da79

// Note that this function is tricky to test because mocking out the action.Configuration struct requires a

.

I think I can open a separate PR this week to refactor the helm logic out of uninstall.findExistingInstallation and install.checkForPreviousInstallations, and unit test the part that's unrelated to helm logic. The reason to pull it out into a separate PR is since I'll be focusing on testing within the acceptance tests PR we can add this additional testing there after a quick refactor.

The port-forward sometimes fails randomly

ndhanushkodi requested review from a team, ishustava, sadjamz and kschoche and removed request for a team September 15, 2021 04:18

ndhanushkodi force-pushed the cli-uninstall branch from 92a202b to ac3a4fb Compare September 15, 2021 06:23

kschoche reviewed Sep 15, 2021

View reviewed changes

cli/README.md Outdated Show resolved Hide resolved

kschoche reviewed Sep 15, 2021

View reviewed changes

cli/README.md Outdated Show resolved Hide resolved

kschoche reviewed Sep 15, 2021

View reviewed changes

cli/README.md Outdated Show resolved Hide resolved

kschoche reviewed Sep 15, 2021

View reviewed changes

cli/cmd/uninstall/uninstall.go Outdated Show resolved Hide resolved

kschoche reviewed Sep 15, 2021

View reviewed changes

cli/cmd/uninstall/uninstall.go Show resolved Hide resolved

kschoche reviewed Sep 15, 2021

View reviewed changes

cli/cmd/uninstall/uninstall.go Outdated Show resolved Hide resolved

kschoche reviewed Sep 15, 2021

View reviewed changes

cli/cmd/uninstall/uninstall.go Outdated Show resolved Hide resolved

kschoche reviewed Sep 15, 2021

View reviewed changes

cli/cmd/uninstall/uninstall.go Outdated Show resolved Hide resolved

kschoche reviewed Sep 15, 2021

View reviewed changes

kschoche approved these changes Sep 15, 2021

View reviewed changes

ndhanushkodi force-pushed the cli-install branch 2 times, most recently from 4857b15 to 749c942 Compare September 16, 2021 01:43

ndhanushkodi force-pushed the cli-uninstall branch from ac3a4fb to 4f73fd4 Compare September 16, 2021 02:10

ndhanushkodi force-pushed the cli-install branch from 8b22214 to 5f3b109 Compare September 16, 2021 02:30

ndhanushkodi force-pushed the cli-uninstall branch from 4f73fd4 to dd282f4 Compare September 16, 2021 04:48

Base automatically changed from cli-install to main September 16, 2021 05:04

ndhanushkodi force-pushed the cli-uninstall branch from dd282f4 to d00c301 Compare September 16, 2021 05:06

ndhanushkodi commented Sep 16, 2021

View reviewed changes

cli/cmd/install/install.go Show resolved Hide resolved

ndhanushkodi commented Sep 16, 2021

View reviewed changes

cli/README.md Outdated Show resolved Hide resolved

ishustava reviewed Sep 17, 2021

View reviewed changes

ndhanushkodi added area/acls Related to ACLs and removed area/acls Related to ACLs labels Sep 21, 2021

ndhanushkodi added the area/acls Related to ACLs label Sep 21, 2021

ndhanushkodi requested a review from ishustava September 22, 2021 06:36

ndhanushkodi force-pushed the cli-uninstall branch 3 times, most recently from a5a0c22 to eb31fec Compare September 22, 2021 07:10

ndhanushkodi added 11 commits September 22, 2021 12:12

cli: initial uninstall

ca13e5f

Finish most of uninstall and start tests

25f608a

Add flags to skip prompting and more unit tests

4264c5d

cleanup and add readme

2876b44

cleanups and review comments

95d8fd3

change -wipe-data and -auto-approve flags based on eng sync

f7b2718

update docs for -wipe-data flag

0a9f1ea

Address review comments and bugs.

5f4e2a5

- delete roles, rolebindings, service accounts by label - use hack for setting namespace for helm kube client - fallback to release name/namespace consul

remove old hack for overriding namespace

1a38b8f

refactor: add helper function to initialize action config

8f41651

add timeout flag docs

280cd71

ndhanushkodi force-pushed the cli-uninstall branch from 560b896 to 280cd71 Compare September 22, 2021 19:13

ishustava approved these changes Sep 22, 2021

View reviewed changes

ndhanushkodi added 2 commits September 23, 2021 10:26

small cleanups

d0e0d15

remove hack with env var, modify found logic

829318e

ndhanushkodi merged commit 9a974c1 into main Sep 23, 2021

ndhanushkodi deleted the cli-uninstall branch September 23, 2021 17:55

lawliet89 pushed a commit to lawliet89/consul-k8s that referenced this pull request Oct 6, 2021

Retry port-forward on failure (hashicorp#725)

3ae0369

The port-forward sometimes fails randomly

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cli: uninstall command #725

cli: uninstall command #725

ndhanushkodi commented Sep 15, 2021 •

edited by ishustava

Loading

kschoche Sep 15, 2021

kschoche Sep 15, 2021 •

edited

Loading

ishustava Sep 17, 2021

ndhanushkodi Sep 22, 2021

kschoche Sep 15, 2021

kschoche Sep 15, 2021

ishustava Sep 17, 2021

ndhanushkodi Sep 22, 2021

kschoche left a comment

ishustava left a comment

ishustava Sep 17, 2021

ishustava Sep 17, 2021

ishustava Sep 17, 2021

ndhanushkodi Sep 21, 2021

ndhanushkodi Sep 22, 2021 •

edited

Loading

ishustava left a comment

ishustava Sep 22, 2021

ndhanushkodi Sep 22, 2021

cli: uninstall command #725

cli: uninstall command #725

Conversation

ndhanushkodi commented Sep 15, 2021 • edited by ishustava Loading

Choose a reason for hiding this comment

kschoche Sep 15, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kschoche left a comment

Choose a reason for hiding this comment

ishustava left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ndhanushkodi Sep 22, 2021 • edited Loading

Choose a reason for hiding this comment

ishustava left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ndhanushkodi commented Sep 15, 2021 •

edited by ishustava

Loading

kschoche Sep 15, 2021 •

edited

Loading

ndhanushkodi Sep 22, 2021 •

edited

Loading