AccessDeniedException for kafka connector running on kafka connect cluster in kubernetes

[antontreushchenko]

Hi i have a problem that i cant solve. I have a kafka connect cluster, and a connector that uses a glue schema registry. When I run my connector locally everything works fine, but when I do it in a kubernetes cluster I get this error:

Caused by: software.amazon.awssdk.services.glue.model.AccessDeniedException: User: arn:aws:sts::***:assumed-role/aws-***/*** is not authorized to perform: glue:GetSchemaVersion because no identity-based policy allows the glue:GetSchemaVersion action (Service: Glue, Status Code: 400, Request ID: ***, Extended Request ID: null)

Here are the connector settings:

{
  "name": "***",
  "config": {
    "connector.class": "io.confluent.connect.s3.S3SinkConnector",
    "storage.class": "io.confluent.connect.s3.storage.S3Storage",
    "format.class": "io.confluent.connect.s3.format.avro.AvroFormat",
    "s3.bucket.name": "***",
    "s3.region": "***",
    "topics": "***",
    "tasks.max": 1,
    "behavior.on.null.values": "ignore",
    "avro.codec": "deflate",
    "flush.size": 10000,
    "timezone": "UTC",
    "locale": "en_US",
    "partition.duration.ms": 10000,
    "partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
    "rotate.schedule.interval.ms": 10000,
    "path.format": "'dt'=yyyy-MM-dd",
    "timestamp.extractor": "Wallclock",
    "errors.log.include.messages": "true",
    "errors.deadletterqueue.context.headers.enable": "true",
    "errors.log.enable": "true",
    "errors.tolerance": "all",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter",
    "value.converter.endpoint": "https://glue.***.amazonaws.com",
    "value.converter.region": "***",
    "value.converter.registry.name": "***",
    "value.converter.schemaName": "***",
    "value.converter.avroRecordType": "GENERIC_RECORD",
    "value.converter.schemas.enable": "true",
    "value.converter.schemaAutoRegistrationEnabled": "true"
  }
}

My irsa. Not used in connector. Used to grant access:

data "aws_iam_policy_document" "kafka-connect" {
  statement {
    effect = "Allow"
    actions = [
      "s3:ListAllMyBuckets"
    ]
    resources = ["arn:aws:s3:::***"]
  }

  statement {
    effect = "Allow"
    actions = [
      "s3:ListBucket",
      "s3:GetBucketLocation",
      "s3:PutObject",
      "s3:GetObject",
      "s3:AbortMultipartUpload",
      "s3:ListMultipartUploadParts",
      "s3:ListBucketMultipartUploads"
    ]
    resources = [
      "arn:aws:s3:::***",
    ]
  }

  statement {
    effect = "Allow"
    actions = [
      "glue:GetSchemaVersion",
    ]
    resources = [
      "arn:aws:glue:${var.aws_region}:${local.account_id}:registry/***",
      "arn:aws:glue:${var.aws_region}:${local.account_id}:registry/***/*",
    ]
  }

  statement {
    effect = "Allow"
    actions = [
      "sts:AssumeRole",
    ]
    resources = [
      "*"
    ]
  }

  statement {
    effect = "Allow"
    actions = [
      "kafka:Describe*",
      "kafka:Get*",
      "kafka:List*",
      "kafka:Update*"
    ]
    resources = [
      "arn:aws:kafka:${var.aws_region}:${local.account_id}:cluster/***/*"
    ]
  }
}

module "irsa-kafka-connect-streaming" {
  source = "***/irsa"

  create_namespace                  = true
  namespace                         = "***"
  service_name                      = "kafka-connect"
  iam_policy                        = data.aws_iam_policy_document.kafka-connect.json
  role_permissions_boundary_arn     = "" # Should have no boundary
}

In the error the role is not irsa but instance profile

[mohitpali]

GetSchemaVersion does not support resource based policies yet. Can you check this and modify permissions to use "*" resource for schema versions ?

https://docs.aws.amazon.com/glue/latest/dg/schema-registry-gs.html

{ "Sid" : "GetSchemaVersion", "Effect" : "Allow", "Action" : [ "glue:GetSchemaVersion" ], "Resource" : ["*"] }

[antontreushchenko]

Hi! I tried using
{ "Sid" : "GetSchemaVersion", "Effect" : "Allow", "Action" : [ "glue:GetSchemaVersion" ], "Resource" : ["*"] }
but that didn't work either. The error was the same

[mohitpali]

I would suggest reaching out to AWS Support. That way the details like AWSAccountId can be shared. It will also help the service teams debug.

[kapilkodwani]

Hi, @antontreushchenko and @mohitpali we are still facing the same issue. Is this solved or can we have some guide on how to fix this permission issue?

[adz218]

Hello, also facing this issue. I have granted

        {
            "Action": "glue:*",
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        },

but am still seeing AccessDeniedException

[iNoahNothing]

Related: #68