Can not start in AWS IAM environment

[danishnawab]

Describe the bug
In my organization, we use AWS IAM to authenticate access to our Kubernetes cluster. Mizu seems to have trouble booting up in such an environment.
It just silently fails when running mizu tap PODNAME, but when running mizu view the following error is shown:

Failed to found mizu service services "mizu-api-server" is forbidden: User "<redacted>" cannot get resource "services" in API group "" in the namespace "mizu"

I suspect mizu is incompatible with the authentication mechanism and perhaps that is also the reason why mizu tap PODNAME does nothing.

To Reproduce
Steps to reproduce the behavior:

1. Have your .kube/config use aws-iam-authenticator to authenticate to the Kubernetes cluster
2. Run mizu tap PODNAME
3. Mizu terminates after the log message Waiting for Mizu Agent to start...

Expected behavior
Mizu should boot successfully

Logs

[2022-02-07T22:42:44.902+0100] DEBUG ▶ Checking for newer version... ▶ [19526 versionCheck.go:47 CheckNewerVersion]
[2022-02-07T22:42:44.902+0100] DEBUG ▶ Init config finished
 Final config: {
	"Tap": {
		"UploadIntervalSec": 10,
		"PodRegexStr": ".*",
		"GuiPort": 8899,
		"ProxyHost": "127.0.0.1",
		"Namespaces": null,
		"Analysis": false,
		"AllNamespaces": false,
		"PlainTextFilterRegexes": null,
		"IgnoredUserAgents": null,
		"DisableRedaction": false,
		"HumanMaxEntriesDBSize": "200MB",
		"DryRun": false,
		"Workspace": "",
		"EnforcePolicyFile": "",
		"ContractFile": "",
		"AskUploadConfirmation": true,
		"ApiServerResources": {
			"CpuLimit": "750m",
			"MemoryLimit": "1Gi",
			"CpuRequests": "50m",
			"MemoryRequests": "50Mi"
		},
		"TapperResources": {
			"CpuLimit": "750m",
			"MemoryLimit": "1Gi",
			"CpuRequests": "50m",
			"MemoryRequests": "50Mi"
		},
		"ServiceMesh": false
	},
	"Version": {
		"DebugInfo": false
	},
	"View": {
		"GuiPort": 8899,
		"Url": ""
	},
	"Logs": {
		"FileStr": ""
	},
	"Auth": {
		"EnvName": "up9.app",
		"Token": ""
	},
	"Config": {
		"Regenerate": false
	},
	"AgentImage": "docker.io/up9inc/mizu:0.25.0",
	"KratosImage": "gcr.io/up9-docker-hub/mizu-kratos/stable:0.0.0",
	"KetoImage": "gcr.io/up9-docker-hub/mizu-keto/stable:0.0.0",
	"ImagePullPolicyStr": "Always",
	"MizuResourcesNamespace": "mizu",
	"Telemetry": true,
	"DumpLogs": true,
	"KubeConfigPathStr": "",
	"ConfigFilePath": "/Users/<redacted>/.mizu/config.yaml",
	"HeadlessMode": false,
	"LogLevelStr": "INFO",
	"ServiceMap": false,
	"OAS": false,
	"Elastic": {
		"User": "",
		"Password": "",
		"Url": ""
	}
}
 ▶ [19526 config.go:57 InitConfig]
[2022-02-07T22:42:44.902+0100] INFO  ▶ Mizu will store up to 200MB of traffic, old traffic will be cleared once the limit is reached. ▶ [19526 tap.go:80 func9]
[2022-02-07T22:42:44.902+0100] DEBUG ▶ Using kube config /Users/<redacted>/.kube/config ▶ [19526 provider.go:1176 loadKubernetesConfiguration]
[2022-02-07T22:42:45.839+0100] DEBUG ▶ Finished version validation, github version 0.25.0, current version 0.25.0, took 937.132394ms ▶ [19526 versionCheck.go:95 CheckNewerVersion]
[2022-02-07T22:42:46.898+0100] INFO  ▶ Tapping pods in namespaces "<redacted>" ▶ [19526 tapRunner.go:116 RunMizuTap]
[2022-02-07T22:42:47.276+0100] INFO  ▶ �[1;32m+<redacted>�[0m ▶ [19526 tapRunner.go:186 printTappedPodsPreview]
[2022-02-07T22:42:47.277+0100] INFO  ▶ Waiting for Mizu Agent to start... ▶ [19526 tapRunner.go:126 RunMizuTap]
[2022-02-07T22:42:50.447+0100] DEBUG ▶ Checking for newer version... ▶ [19538 versionCheck.go:47 CheckNewerVersion]
[2022-02-07T22:42:50.447+0100] DEBUG ▶ Init config finished
 Final config: {
	"Tap": {
		"UploadIntervalSec": 10,
		"PodRegexStr": ".*",
		"GuiPort": 8899,
		"ProxyHost": "127.0.0.1",
		"Namespaces": null,
		"Analysis": false,
		"AllNamespaces": false,
		"PlainTextFilterRegexes": null,
		"IgnoredUserAgents": null,
		"DisableRedaction": false,
		"HumanMaxEntriesDBSize": "200MB",
		"DryRun": false,
		"Workspace": "",
		"EnforcePolicyFile": "",
		"ContractFile": "",
		"AskUploadConfirmation": true,
		"ApiServerResources": {
			"CpuLimit": "750m",
			"MemoryLimit": "1Gi",
			"CpuRequests": "50m",
			"MemoryRequests": "50Mi"
		},
		"TapperResources": {
			"CpuLimit": "750m",
			"MemoryLimit": "1Gi",
			"CpuRequests": "50m",
			"MemoryRequests": "50Mi"
		},
		"ServiceMesh": false
	},
	"Version": {
		"DebugInfo": false
	},
	"View": {
		"GuiPort": 8899,
		"Url": ""
	},
	"Logs": {
		"FileStr": ""
	},
	"Auth": {
		"EnvName": "up9.app",
		"Token": ""
	},
	"Config": {
		"Regenerate": false
	},
	"AgentImage": "docker.io/up9inc/mizu:0.25.0",
	"KratosImage": "gcr.io/up9-docker-hub/mizu-kratos/stable:0.0.0",
	"KetoImage": "gcr.io/up9-docker-hub/mizu-keto/stable:0.0.0",
	"ImagePullPolicyStr": "Always",
	"MizuResourcesNamespace": "mizu",
	"Telemetry": true,
	"DumpLogs": false,
	"KubeConfigPathStr": "",
	"ConfigFilePath": "/Users/<redacted>/.mizu/config.yaml",
	"HeadlessMode": false,
	"LogLevelStr": "INFO",
	"ServiceMap": false,
	"OAS": false,
	"Elastic": {
		"User": "",
		"Password": "",
		"Url": ""
	}
}
 ▶ [19538 config.go:57 InitConfig]
[2022-02-07T22:42:50.448+0100] DEBUG ▶ Using kube config /Users/<redacted>/.kube/config ▶ [19538 provider.go:1176 loadKubernetesConfiguration]
[2022-02-07T22:42:50.946+0100] DEBUG ▶ successfully reported telemetry for cmd view ▶ [19538 telemetry.go:36 ReportRun]
[2022-02-07T22:42:51.299+0100] DEBUG ▶ Finished version validation, github version 0.25.0, current version 0.25.0, took 852.210223ms ▶ [19538 versionCheck.go:95 CheckNewerVersion]
[2022-02-07T22:42:52.147+0100] ERROR ▶ Failed to found mizu service services "mizu-api-server" is forbidden: User "<redacted>" cannot get resource "services" in API group "" in the namespace "mizu" ▶ [19538 viewRunner.go:32 runMizuView]

Desktop (please complete the following information):

• OS: macOS

Additional context
Unfortunately, I am not too familiar with the authentication setup on the AWS side. But it is clear that our users do not have the access rights needed by Mizu.
Is it possible for Mizu to work around this limitation?

[IgorGov]

HI @danishnawab
Thanks for reaching out.
Mizu tap creates an "agent" pod inside your cluster, in order to access the web ui mizu CLI tries to create k8s proxy or port forward. So the user which running the CLI need to have certain permissions in the cluster like get pod/service. Just wanted to double check that your IAM user have these permissions in a newly created namespace like "mizu"?

[danishnawab]

@IgorGov No, my user doesn't have access to the mizu namespace, not out of the box.
Access within our organization is limited to the namespaces we deploy our stuff on. And, mizu is not one of those.
Could there be a way by which mizu could run in such environments? I suppose there would be others with a similar authentication setup.
Perhaps mizu could instead start the agent in the existing namespaces instead of a new namespace?

[IgorGov]

Mizu have the ability to be deployed (and created needed resources) to an existing namespace, you can use the "mizu-resources-namespace" flag:
./mizu tap --set mizu-resources-namespace=<namespace for k8s resource to be deployed> -n <the same namespace>

Notice that you won't be able to deploy to namespace "A" and sniff traffic from namespace "B".

Let me know if that helps

[danishnawab]

So deploying mizu in one of the existing services worked to the point that the server could come up and the traffic viewer was loaded in my browser, however, I didn't see any traffic.
Looking at the logs, I could see the following statements:

[2022-02-08T23:46:31.669+0100] DEBUG ▶ error while getting kubernetes server version, err: Get "https://<redacted>:443/version?timeout=32s": getting credentials: exec: executable aws-iam-authenticator failed with exit code 1 ▶ [68653 provider.go:1149 GetKubernetesVersion]
[2022-02-08T23:46:31.670+0100] ERROR ▶ Get "https://<redacted>:443/version?timeout=32s": getting credentials: exec: executable aws-iam-authenticator failed with exit code 1 ▶ [68653 common.go:94 handleKubernetesProviderError]

At first, I assumed that my IAM user doesn't have the access rights to query the k8s version, but then I tried executing kubectl version and it successfully listed both the client and the server versions.

[IgorGov]

can you please run the command with flag "--set dump-logs=true", it should generate a zip file with all mizu logs (cli, server and tappers). The command should look like:
../mizu tap --set mizu-resources-namespace=<ns> -n <ns> --set dump-logs=true

Can you confirm that CLI detects pods to tap? the output should look like:

Can you confirm that mizu tapper pods are up and running? see the highlighted pods:

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.