MSK: operation error Kafka: CreateCluster, https response error StatusCode: 403

[kappa8219]

Describe the bug

arn:aws:iam::aws:policy/AmazonMSKFullAccess attached with Pod Identity results in:

{
  "level": "error",
  "ts": "2025-05-09T05:31:01.021Z",
  "msg": "Reconciler error",
  "controller": "cluster",
  "controllerGroup": "kafka.services.k8s.aws",
  "controllerKind": "Cluster",
  "Cluster": {
    "name": "cluster-name",
    "namespace": "ack-system"
  },
  "namespace": "ack-system",
  "name": "x",
  "reconcileID": "7680a7be-2523-4689-9268-0c04a18db412",
  "error": "operation error Kafka: CreateCluster, https response error StatusCode: 403, RequestID: 3bba50f8-f56f-4d73-a50f-23eef5249e01, api error AccessDeniedException: User: xxx is not authorized to perform: kafka:CreateCluster on resource: *",
  "stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"
}

Steps to reproduce

Expected outcome
Create cluster

Environment

• Kubernetes version 1.31
• Using EKS - yes
• AWS service - MSK

[github-actions]

Hello @kappa8219 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[kappa8219]

Same as for closed Issue #2074

[gecube]

Hi! Please kindly check with full admin perms.

[kappa8219]

Hi! Please kindly check with full admin perms.

It is strange but still the same with policy arn:aws:iam::aws:policy/AdministratorAccess

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

{
  "level": "error",
  "ts": "2025-05-13T07:42:48.221Z",
  "msg": "Reconciler error",
  "controller": "cluster",
  "controllerGroup": "kafka.services.k8s.aws",
  "controllerKind": "Cluster",
  "Cluster": {
    "name": "xxxx",
    "namespace": "ack-system"
  },
  "namespace": "ack-system",
  "name": "xxxxx",
  "reconcileID": "337fb40f-400e-4d38-92c7-e7e35b418320",
  "error": "operation error Kafka: CreateCluster, https response error StatusCode: 403, RequestID: 3a37877e-b0d6-428f-8d9e-63b5e1d42f17, api error AccessDeniedException: User: arn:aws:sts::xxxx:assumed-role/ack-controllers-kafka/eks-terra-clus-ack-contro-9cd673e7-892d-4e7d-8dac-ee1119aa67b0 is not authorized to perform: kafka:CreateCluster on resource: *",
  "stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"
}

[gecube]

@kappa8219 did you restart controller after changing the perms? Very strange as it worked for me two weeks ago.

[kappa8219]

@gecube Sure, restarted. I have pod identify as role appliance mechanism. But all other controllers assume same way, all fine.

[gecube]

ah... I used IRSA... I still did not switch to pod identity.

[vinicius91]

Any updates on this investigation? I'm facing similar issues on my stack too

[gecube]

@vinicius91 I obsoleted MSK and switched to Strimzi (as we have EKS). I recommend to consider this option to everybody.

But if we are talking about MSK and it is mandatory requirement to use it - unfortunately, I did not progress with investigation.

[vinicius91]

Strimzi was our first candidate to provision Kafka since we are also on EKS, but we decided to give MSK a try assuming that it would be a smoother experience since we already use other ack controllers, but so far it has been the opposite.

Would you say that Strimzi is on the same complexity level as the Provisioned Standard when it comes to storage management?

[gecube]

@vinicius91 I think that strimzi gives better experience.. at leas from the maintenance and cost perspective. But it could sound like anti-advertisement or advertisement against managed amazon services and msk in particular - no way :-)

[vinicius91]

Yes, but it would nice if they put more effort into the product. This unaddressed issue here since 2024 is a bad advertisement in itself :(

[michaelhtm]

Hello @kappa8219 @vinicius91
I was unable to replicate this issue. I'm also using PodIdentity with AmazonMSKFullAccess permission, and was able to create the msk Cluster successfully..
I'm using v1.2.1.

Not sure what could be the issue here..