Mangle Support for Amazon EKS

[Anvesh42]

Mangle team, (@rpraveen-vmware @ashrimalivmware)

I communicated to you team earlier on https://github.com/vmware/mangle/issues/82 & https://github.com/vmware/mangle/issues/105 which were related to chaos setup on OpenShift cluster.

This time we are experiencing different issue. We are trying to have chaos functionality implemented in the AWS EKS environment for the GF applications and we are observing issues with kubeconfig integration. Here are the details,

Scenario

1. Mangle product is running on its own dedicated EKS cluster, lets says, EKS-Mangle
2. Target service (where chaos injection needs to happen) is running on another EKS cluster, let's say, EKS-Target
3. The kubeconfig file of EKS-Target,

EKS-Target-kubeconfig.txt

4. The required IAM policies and EKS RBAC policies have been established.

Problem Statement

While configuring & testing the endpoint connection on EKS-Mangle using the kubeconfig file of EKS-Target, we see failed connection error.
Test Connection failed for endpoint, Please reverify the credentials {0} . Reason: Test Connection failed for endpoint, Please reverify the credentials

Our Initial RCA

1. We enabled the trace level logging on the mangle UI to grep the underlying root cause.
2. We found that the issue is related to AWS authentication from mangle. Mangle package doesn't have the AWS CLI or the AWS-IAM-Authenticator required to authenticate against AWS. Please see trace log below,

exec executable aws not found

Some Useful Resources

1. https://itnext.io/how-does-client-authentication-work-on-amazon-eks-c4f2b90d943b#609a
2. https://github.com/kubernetes-sigs/aws-iam-authenticator
3. https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

From the available documentation on mangle, I do not see any specific use case support for EKS-mangle integration. Is this a limitation at this point?

Thanks
Anvesh

[rpraveen-vmware]

Hi @Anvesh42
Thanks for reaching out to us.
As we see that, to access the EKS Kubernetes cluster,
the pre-requisite utilities are:
kubectl, aws-iam-authenticator, awscli and priam.

-> Currently in our Mangle container, we have only the kubectl installed from the above list.
Hence, we will not be able to access the resources of another EKS cluster from the present cluster.

-> We suggest you to deploy Mangle in the same cluster and access the resources locally (by adding K8s endpoint and namespace without kubeconfig).

-> However, we consider this as a feature request and try to address as part of our next Mangle release.

Let us know if any further queries.
cc: @ashrimalivmware @george

Regards,
Praveen R

[Anvesh42]

@rpraveen-vmware Thanks for your response.

To your 2nd pointer, whether the endpoint is local or remote, the kubeconfig file field is imperative and must be defined.

In this case, the kube-system namespace is running within the same EKS cluster where mangle has been deployed and mangle expects the kubeconfig file. Please help understand when you say configure K8s endpoint without the kubeconfig file.

To your 3rd one, will you be able to provide us the timelines on the availability of mangle image that supports EKS workflows?

Thanks
Anvesh

[rpraveen-vmware]

@Anvesh42
To answer the 2nd pointer:
-> Go to "Add k8s Credential" and just give the "Name" of the k8s credential. Don't upload any kubeconfig file.
In this case, use this k8sCredentialName while creating k8s Endpoint.
Mangle considers it as local cluster when you don't provide kubeconfig file.

-> We discussed to have this as part of the next release of Mangle. Will update you on the timelines.
cc: @ashrimalivmware

[Anvesh42]

@rpraveen-vmware Thanks Praveen. That worked for us.

A quick question please - "Does cassandra DB configuration within the mangle product offer integration with AWS S3 or EBS? The default configuration is the NFS. Please see the image below. We wanted to see if cassandra storage can be moved to S3 or EBS instead of a filesystem.

Also, does mangle support other DB's such as mongodb & postgres? If so, will the team be able to provide us the supporting YAML's?

[ashrimalivmware]

Hi @Anvesh42,
1: Mangle doesn't offer any support with AWS S3 or EBS for Mangle DB (cassandra). Not sure if its possible to achieve using AWS capabilities.
2: Mangle DB only supports cassandra DB.

Thanks,
-Avinash

[Anvesh42]

@rpraveen-vmware @ashrimalivmware Due to current limitation concerning mangle-EKS integration, we are in the process of chaos testing design change. We have a question concerning the new design,

Does mangle support integration with cassandra given the scenario where mangle runs on one Kubernetes cluster while cassandra on another assuming firewall remains open between the two clusters?

Our current standard configuration:

Configuration we are trying to explore:

Please note the clusterB string in the cassandra contact point value.

We tried testing this approach and were able to find a hint of successful connection but couldn't be definite about it. We had an unsuccessful attempt trying to access mangle storage data on cassandra container (running on another cluster) via cqlsh command. Mangle documentation is limited in this area. Inputs are welcome.

Please let us know your thoughts on this design.

Thanks
Anvesh

[rpraveen-vmware]

Hi @Anvesh42
This approach, to have DB in one cluster and the service in another cluster is something new that we have not tried off.
We haven't tried out this deployment approach for our k8s.

-> Seeing that you had successful connection from Mangle to cassandra, need to check if it is really supported to provide the mount volume value of the DB in other cluster.

What's the reason behind this approach..? just curious to know..
cc: @ashrimalivmware

[Anvesh42]

@rpraveen-vmware @ashrimalivmware

Do you think your team will be able to provide us the timelines at this point on EKS support for mangle product?

FYI, below are the 2 last comments from your team in this regard,

-> Currently in our Mangle container, we have only the kubectl installed from the above list.
Hence, we will not be able to access the resources of another EKS cluster from the present cluster.
-> However, we consider this as a feature request and try to address as part of our next Mangle release.

Thanks
Anvesh