Flux fails to clone CodeCommit repo in cross-account setup

[paraspachpute]

Describe the bug

When trying to set up Flux source with HTTPS credentials in an EKS Fargate cluster on Account A, where the CodeCommit repository is in another account, Account B, the flux create source git command fails and returns the error message "unable to clone <repo_name>: authorization failed".

I have already created a role in Account B with CodeCommit permissions for the OIDC provider from Account A, and tested that I can access and clone repos from the cluster using a sample nginx-git pod with the service account needed for authentication attached to it.

I added the same role to the service accounts in the Flux gotk-components file using annotations, and then deployed the controllers, but the error persisted.

Is this an issue with the git implementation flux uses?

Steps to reproduce

Prerequisites

• Two AWS accounts: Account A for the EKS Fargate cluster and Account B for the CodeCommit repository
• OIDC provider associated with the EKS Fargate cluster in Account A

Steps

1. Create a role in Account B with CodeCommit permissions for the OIDC provider from Account A.
2. Edit the source controller service account from the gotk-components file, and add the annotation eks.amazonaws.com/role-arn:<arn of your role created in step 1 from account B>
3. Deploy these changes and add Flux source to the CodeCommit repo using the HTTPS credentials:

flux create source git flux-system --url=https://git-codecommit.ap-south-1.amazonaws.com/v1/repos/<repo_name> --branch=master --username=<aws-user-git-credentials> --password=<password> --interval=1m

Expected behavior

Flux reconciliation successful

Screenshots and recordings

No response

OS / Distro

N/A

Flux version

v0.40.2

Flux check

► checking prerequisites
✔ Kubernetes 1.22.17-eks-48e63af >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.28.1
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.28.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.23.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.32.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.30.2
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.33.0
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta2
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta2
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta1
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta1
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta2
✔ receivers.notification.toolkit.fluxcd.io/v1beta2
✔ all checks passed

Git provider

CodeCommit

Container Registry provider

No response

Additional context

Additional information

To test git clone is working for cross-account in EKS, I tried the following:

Follow the above steps until step 2.

1. Create a service account in Account A with the annotation eks.amazonaws.com/role-arn:<arn of your role created in step 1 from account B>
2. Create a pod with the above serviceAccount in Account A and use image: emarcs/nginx-git (so that git is preinstalled).
3. Exec into the pod and do a git clone to the CodeCommit repo in Account B using HTTPS (fill in the git credentials when prompted).

The repository gets cloned successfully.

Code of Conduct

• I agree to follow this project's Code of Conduct

[a-tharva]

Is this issue related to git not setting the credentials for other account for flux

[vivekt333]

+1

[paraspachpute]

@a-tharva No, i dont think that should be the problem, we anyway set the credentials by passing them via flags while creating the source.

[stefanprodan]

Using IRSA for Git operations is not supported. Instead of connecting Flux to the repo in the other account, you can push the manifests from that repo to ECR, then use a OCIRepository which supports AWS IRSA. An workflow example can be found here https://fluxcd.io/flux/cheatsheets/oci-artifacts/