docs: Instructions on `make verify-kube-connect` step when using k3d #4687

dewan-ahmed · 2020-10-28T04:14:22Z

When setting up my local development environment following the developer guide, I got stuck at testing the connection from build container to my Kubernetes cluster which is a k3s cluster using k3d. After posting the question on argocd slack, I was pointed to this discussion where someone else faced the same challenge. Being the second person to get stuck on this issue, I would not want a third person to get stuck and hence proposing this doc update. This PR will provide step-by-step guidance to someone using k3d to execute a successful make verify-kube-connect.

Checklist:

This is a doc update.
The title of the PR states what changed but it doesn't have an associated issue number (used for the release note).
Does this PR require documentation updates?
I've updated documentation as required by this PR.
Optional. My organization is added to USERS.md.
I've signed the CLA and my build is green (troubleshooting builds).

sbose78 · 2020-10-28T04:22:38Z

Shorten title of the PR, and have a longer description mentioning what led you to go down this path :-)

sbose78 · 2020-10-28T04:23:31Z

docs/developer-guide/contributing.md

@@ -156,6 +156,29 @@ make: *** [Makefile:386: verify-kube-connect] Error 1

 you should edit your `~/.kube/config` and modify the `server` option to point to your correct K8s API (as described above).

+### If using k3d to spin-up your Kubernetes cluster


You could have the title as "Using k3d".

sbose78 · 2020-10-28T04:24:26Z

docs/developer-guide/contributing.md

@@ -156,6 +156,29 @@ make: *** [Makefile:386: verify-kube-connect] Error 1

 you should edit your `~/.kube/config` and modify the `server` option to point to your correct K8s API (as described above).

+### If using k3d to spin-up your Kubernetes cluster
+
+Because it's running in a docker container, you're dealing with docker's internal networking rules when using k3d. This is the cost of a fully self-contained, disposable k8s cluster. The following steps should help with a successful `make verify-kube-connect` execution.


Introduce k3d, provide some details about how it different than a usual kubernetes cluster - and the subsequent issues.

:)

thanks for the note @sbose78 :)
Added an intro for k3d and some comparison with traditional k8s cluster on local machine.

codecov-io · 2020-10-28T04:24:27Z

Codecov Report

Merging #4687 into master will decrease coverage by 0.36%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4687      +/-   ##
==========================================
- Coverage   41.37%   41.00%   -0.37%     
==========================================
  Files         126      127       +1     
  Lines       17154    17367     +213     
==========================================
+ Hits         7097     7122      +25     
- Misses       9042     9223     +181     
- Partials     1015     1022       +7

Impacted Files	Coverage Δ
util/helm/cmd.go	`25.69% <0.00%> (-13.44%)`	⬇️
util/helm/client.go	`45.03% <0.00%> (-11.01%)`	⬇️
util/argo/argo.go	`59.32% <0.00%> (-5.37%)`	⬇️
controller/appcontroller.go	`50.56% <0.00%> (-1.92%)`	⬇️
pkg/apis/application/v1alpha1/types.go	`57.92% <0.00%> (-0.37%)`	⬇️
cmd/argocd-util/commands/projects.go	`34.48% <0.00%> (-0.30%)`	⬇️
util/settings/settings.go	`40.65% <0.00%> (ø)`
cmd/argocd/commands/repo.go	`0.00% <0.00%> (ø)`
cmd/argocd-util/commands/project_allowlist.go	`47.29% <0.00%> (ø)`
util/db/repository.go	`63.73% <0.00%> (+0.26%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3bf9deb...30e9bef. Read the comment docs.

dewan-ahmed · 2020-10-28T05:45:14Z

Shorten title of the PR, and have a longer description mentioning what led you to go down this path :-)

Thanks for the note @sbose78.

Shortened the PR and tried to provide some context in the description.

amitkrout · 2020-10-28T10:10:49Z

docs/developer-guide/contributing.md

@@ -156,6 +156,29 @@ make: *** [Makefile:386: verify-kube-connect] Error 1

 you should edit your `~/.kube/config` and modify the `server` option to point to your correct K8s API (as described above).

+### Using k3d
+
+k3d is a lightweight wrapper to run [k3s](https://github.com/rancher/k3s), a minimal Kubernetes distribution, in docker. Because it's running in a docker container, you're dealing with docker's internal networking rules when using k3d. A typical Kubernetes cluster running on your local machine is part of the same network that you're on so you can access it using **kubectl**. However, a Kubernetes cluster running within a docker container (in this case, the one launched by make) cannot access 0.0.0.0 from inside the container itself, when 0.0.0.0 is a network resource outside the container itself (and/or the container's network). This is the cost of a fully self-contained, disposable Kubernetes cluster. The following steps should help with a successful `make verify-kube-connect` execution.


In the first point of [1] it has been mentioned that a minimal Kubernetes cluster k3s is enough for dev environment and from [2] i can see k3s is only supported on linux distribution. So on mac and windows we need a third party tool like k3d, VirtualBox etc which can provide a containerize/virtual linux env to install k3s.

Instead of specifying a tool to install k3s, IMO just create a hyper link k3d in the first point of [1] as one of containerize env provider and ofcourse for connectivity you need to provide ip address configuration information in k3d containerize env.

[1] https://github.com/argoproj/argo-cd/blob/master/docs/developer-guide/contributing.md#before-you-start
[2] https://rancher.com/docs/k3s/latest/en/installation/installation-requirements/#operating-systems

amitkrout · 2020-10-28T10:14:55Z

docs/developer-guide/contributing.md

+1. Find the actual external IP of your host. You can do so by executing `ifconfig` on Mac/Linux and `ipconfig` on Windows. For most users, the following command works to find the IP address.
+
+```
+IP=`ifconfig en0 | grep inet | grep -v inet6 | awk '{print $2}'`


It will error out on linux and windows because the same interface device won't be available

on linux you can run ip addr. I don't know about windows.

Also keep in mind this IP will be dynamically assigned by the router so if your router restarts for any reason your IP will change and you will have to recreate the cluster. You can assign your PC a static IP to prevent that from happening

Thanks for the review Amit. I added command for Linux.

amitkrout · 2020-10-28T10:15:11Z

docs/developer-guide/contributing.md

+
+3. Execute a `kubectl version` to make sure you can still connect to the Kubernetes API server via this new IP. If not, go back to step 1.
+
+4. Run `make verify-kube-connect` and check if it works.


This validation has been covered as part of - https://github.com/argoproj/argo-cd/blob/master/docs/developer-guide/contributing.md#test-connection-from-build-container-to-your-k8s-cluster

Thanks for the review Amit. I merged these two points as a single step. I tried removing both steps and then the procedure seemed incomplete to me. Therefore, I traded a single instance of repeat for clarity.

amitkrout · 2020-10-28T10:16:21Z

docs/developer-guide/contributing.md

+
+4. Run `make verify-kube-connect` and check if it works.
+
+5. Finally, so that you don't have to keep updating your kube-config whenever you spin up a new k3d cluster, add `--api-port $IP:6550` to your **k3d cluster create** command, where $IP is the value from step 1. An example command is provided here.


IMO theis should be part of the k3d documentation.

I checked existing k3d doc and they don't seem to have this explanation.

dewan-ahmed · 2020-10-28T18:53:01Z

Server timeout Error: TestAutoSyncSelfHealDisabled: app_autosync_test.go:28: timeout waiting for: sync status to be Synced, is Unknown

/retest

dewan-ahmed · 2020-10-28T19:33:44Z

/test all

dewan-ahmed · 2020-10-29T15:49:08Z

Hi @sbose78 pls let me know if I can make any other improvement on this PR.

alexmt

LGTM. Thank you!

…zation-generators * 'master' of github.com:argoproj/argo-cd: fix: RevisionFormField component crashes in 'refs' API returns no tags (argoproj#4735) docs: add Opensurvey to USERS.md (argoproj#4727) docs: correct parameters usage in CLI (argoproj#4725) fix: Repo-server has silent unmarshalling errors leading to empty applications (argoproj#4423) (argoproj#4708) fix: inject artificial delay between sync waves to better support health assessments (argoproj#4715) fix: exclude files listed under exclusions (argoproj#4686) feat: support resource actions on CRDs that use status subresources (argoproj#4690) feat: Add autocomplete for repo Revisions (argoproj#4645) (argoproj#4713) fix: webhook don't refresh apps pointing to HEAD (argoproj#4717) feat: Add support for ExecProvider cluster auth (argoproj#4600) (argoproj#4710) fix: adding helm values file in New App (argoproj#4635) docs: Instructions on `make verify-kube-connect` step when using k3d (argoproj#4687) feat: Annotation based app paths detection in webhooks (argoproj#4699) fix: adding commonAnnotations for Kustomize (argoproj#4613)

dewan-ahmed added 2 commits October 28, 2020 00:08

Special instructions on using k3d

3f74e04

Minor formatting

faabe3c

dewan-ahmed changed the title ~~Special instructions for make verify-kube-connect step when using k3d to spin-up Kubernetes cluster~~ docs: Special instructions for make verify-kube-connect step when using k3d to spin-up Kubernetes cluster Oct 28, 2020

sbose78 reviewed Oct 28, 2020

View reviewed changes

Providing an intro to k3d

30e9bef

dewan-ahmed changed the title ~~docs: Special instructions for make verify-kube-connect step when using k3d to spin-up Kubernetes cluster~~ docs: Instructions on make verify-kube-connect step when using k3d Oct 28, 2020

amitkrout reviewed Oct 28, 2020

View reviewed changes

Adding review comments and fixing a typo under faq

b973975

jessesuen assigned sbose78 Oct 28, 2020

alexmt approved these changes Oct 29, 2020

View reviewed changes

alexmt merged commit 9cdfe40 into argoproj:master Oct 30, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: Instructions on `make verify-kube-connect` step when using k3d #4687

docs: Instructions on `make verify-kube-connect` step when using k3d #4687

dewan-ahmed commented Oct 28, 2020 •

edited

sbose78 commented Oct 28, 2020

sbose78 Oct 28, 2020

sbose78 Oct 28, 2020

dewan-ahmed Oct 28, 2020

codecov-io commented Oct 28, 2020 •

edited

dewan-ahmed commented Oct 28, 2020

amitkrout Oct 28, 2020 •

edited

amitkrout Oct 28, 2020

tetchel Oct 28, 2020

tetchel Oct 28, 2020 •

edited

dewan-ahmed Oct 28, 2020

amitkrout Oct 28, 2020

dewan-ahmed Oct 28, 2020

amitkrout Oct 28, 2020

dewan-ahmed Oct 28, 2020

dewan-ahmed commented Oct 28, 2020

dewan-ahmed commented Oct 28, 2020

dewan-ahmed commented Oct 29, 2020

alexmt left a comment

		@@ -156,6 +156,29 @@ make: *** [Makefile:386: verify-kube-connect] Error 1

		you should edit your `~/.kube/config` and modify the `server` option to point to your correct K8s API (as described above).

		### If using k3d to spin-up your Kubernetes cluster


		3. Execute a `kubectl version` to make sure you can still connect to the Kubernetes API server via this new IP. If not, go back to step 1.

		4. Run `make verify-kube-connect` and check if it works.


		4. Run `make verify-kube-connect` and check if it works.

		5. Finally, so that you don't have to keep updating your kube-config whenever you spin up a new k3d cluster, add `--api-port $IP:6550` to your k3d cluster create command, where $IP is the value from step 1. An example command is provided here.

docs: Instructions on make verify-kube-connect step when using k3d #4687

docs: Instructions on make verify-kube-connect step when using k3d #4687

Conversation

dewan-ahmed commented Oct 28, 2020 • edited

sbose78 commented Oct 28, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Oct 28, 2020 • edited

Codecov Report

dewan-ahmed commented Oct 28, 2020

amitkrout Oct 28, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tetchel Oct 28, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dewan-ahmed commented Oct 28, 2020

dewan-ahmed commented Oct 28, 2020

dewan-ahmed commented Oct 29, 2020

alexmt left a comment

Choose a reason for hiding this comment

docs: Instructions on `make verify-kube-connect` step when using k3d #4687

docs: Instructions on `make verify-kube-connect` step when using k3d #4687

dewan-ahmed commented Oct 28, 2020 •

edited

codecov-io commented Oct 28, 2020 •

edited

amitkrout Oct 28, 2020 •

edited

tetchel Oct 28, 2020 •

edited