docs: Using IAM roles instead of AWS credentials in management cluster

[Ankitasw]

What type of PR is this?
/kind documentation

What this PR does / why we need it:
This PR adds documentation for usage of IAM roles in management cluster instead of using AWS credentials with which the management cluster was created.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #3130

Checklist:

• squashed commits
• includes documentation
• adds unit tests
• adds or updates e2e tests

Release note:

Added doc for using IAM roles instead of AWS credentials while deploying mgmt cluster

[k8s-ci-robot]

@Ankitasw: This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[richardcase]

These instructions could also apply to using IRSA with EKS with minimal changes. Should we also cover this?

[Ankitasw]

Yes, I have a plan to cover it in this PR, that's why still WIP :)

[richardcase]

Awesome @Ankitasw 🎉

[Ankitasw]

@richardcase I would be covering option 4 in this PR as described in this slack thread. But since you listed IRSA as option 2 in the thread, if it is possible with minimal changes I can do it in this PR itself. It would be really helpful if you could direct me to what steps has to be done in case of IRSA with EKS.
Also, for enforcing use of IAM roles in EKS managed cluster instead of AWS credentials CAPA started with, what additional steps would be required apart from the steps listed in this doc?
P.S I am not aware of these concepts around EKS that's why want some help 😄
cc @sedefsavas

[sedefsavas]

How about listing all options in that thread and If missing some details, could catch up during office hours and fill those.

[Ankitasw]

To summarize all the options we have to go forward with when we pivot to management cluster so that we can document it together and we have covered only 1 scenario in this PR:

• To create a dedicated IAM user for the controller which will be long lived credentials. We can create a bootstrap user in CAPA. These credentials are the ones that need to be encoded.
• To use short lived credentials for the kind cluster only and then pivot into the new management cluster using IRSA for CAPA.(applicable for EKS)
• To use short lived credentials for the kind cluster only and then use one of the “multi-tenancy” types for any future workload clusters created from your new management cluster.
• To use the role attached to management cluster instances by zeroing the secret bootstrap cluster used after moving the mgmt cluster to AWS, this can be done by using clusterawsadm or manually setting the secret data to nil.
• To use PodIdentity webhooks.(applicable for EKS)

[Ankitasw]

cc @richardcase @sedefsavas any advice on how to move ahead in this PR and what all to cover? If the content is suffice for using IAM roles, then we could close this PR and then open issue for other options listed here. Wdyt?

[sedefsavas]

This does not seem correct, we specifically want to use control plane nodes as they have the proper IAM permissions.

I added details on how to move the management cluster, feel free to add some details of that sort here:

Since only control-plane nodes have the required IAM roles attached, CAPA deployment should have the necessary tolerations for master (control-plane) node and node selector for master. To modify the CAPA deployment before running clusterctl init, check [cluster api book](https://cluster-api.sigs.k8s.io/clusterctl/commands/init.html).
Run clusterctl init --infrastructure aws on the workload cluster by setting export AWS_B64ENCODED_CREDENTIALS="Cg==" (equivalent to empty string)

[Ankitasw]

I have mentioned about zeroing the credentials in point 4 below, just before running clusterctl init. This line makes more sense

Since only control-plane nodes have the required IAM roles attached, CAPA deployment should have the necessary tolerations for master (control-plane) node and node selector for master.

as compared to

Control plane nodes on the workload cluster will have IAM roles attached which is enough for the target management cluster to work. We want to move the bootstrap cluster to this workload cluster to turn it to a management cluster.

Does it make sense if I change this line and then keep rest of the instructions as is?

[sedefsavas]

I think we can merge this one and file an issue that lists all the other options discussed on the slack thread and here for future reference.

[Ankitasw]

I think we can merge this one and file an issue that lists all the other options discussed on the slack thread and here for future reference.

Thanks @sedefsavas i will file an issue for the same.

[richardcase]

Based on the discussion and that #3510 has been created:

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: richardcase

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [richardcase]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment