Create AWS KMS new key issue

[dimitryvoronov]

What happened?

Hello crossplane community,
An error occurs when a new AWS KMS key is created.

How can we reproduce it?

Crossplane installed as k8s operator as replicaset with crossplane pod, aws provider and rbac manager.
I am trying to create a new KMS key, empty one as described below:

file: newkms.yaml

apiVersion: kms.aws.crossplane.io/v1alpha1
kind: Key
metadata:
  name: eks-kms-key-crossplane
  namespace: infra
spec:
  forProvider:
    enabled: true
    customerMasterKeySpec: "SYMMETRIC_DEFAULT"
    description: ksm key for eksvault crossplane
    keyUsage: ENCRYPT_DECRYPT
    region: eu-central-1

Once I run kubectl apply -f newkms.yaml
I get Invalid keyID error.

Error description:
kubectl-output-kms-key.txt

What environment did it happen in?

AWS environment:
K8s v1.18.9-eks-d1db3c
Crossplane version: 1.2.1

Am I missing something or this looks like a bug, on slack channel there was a kind of similar issue but also with defining an external-name in annotations.
Any advice or help really appreciated. Thank you.

[muvaf]

Could you reformat the YAML and code snippets with markdown? Also it's really hard to read/reproduce with all the indentation gone. See Syntax highlighing section in https://guides.github.com/features/mastering-markdown/ for details.

[dimitryvoronov]

Hello @muvaf apologies for that, I've marked a template I've used for object creation and uploaded an output of kubectl for convience.

[librannk]

I am also getting an error while creating not exactly similar, "failed to describe Key: EmptyStaticCreds: static credentials are empty". [ NOTE: Using IRSA for credentials ]

        apiVersion: kms.aws.crossplane.io/v1alpha1
        kind: Key
        metadata:
          name: kms-key-test
          namespace: crossplane-local
        spec:
          writeConnectionSecretToRef:
            name: kms-key-test
            namespace: crossplane-local
          providerConfigRef:
            name: crossplane-provider-aws-local
          forProvider:
            region: us-east-1
            origin: "EXTERNAL"
            enabled: true
            customerMasterKeySpec: "SYMMETRIC_DEFAULT"
            keyUsage: ENCRYPT_DECRYPT
            tags:
              - tagKey: KMS Key
                tagValue: "" 

[librannk]

just to add created the sample key from the documentation, even that
didn't work.

https://github.com/crossplane/provider-aws/blob/master/examples/kms/key.yaml

tired with aws cli, and it worked with aws cli fine , so can anyone suggest on this ?

aws kms create-key --policy '      {                                                                       
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
              "AWS": ["arn:aws:iam::<accountid>:root", "arn:aws:iam::<accountid>:role/crossplane-local"]
            },
            "Action": "kms:*",
            "Resource": "*"
          }
        ]
      }' --region us-east-1

[haarchri]

what is the exception in crossplane when you create the example ?
on my side it is working:

➜  kms git:(master) ✗ kubectl get key.kms.aws.crossplane.io/dev-key
NAME      READY   SYNCED   EXTERNAL-NAME
dev-key   True    True     ea87e120-8417-4ade-8499-9e093d5664d7

[librannk]

thanks for commenting, are you using IRSA for credentials? or directly? .... I am using IRSA ..

[jwitrick]

I am similar getting an error when trying to create an AWS KMS key:
Message: observe failed: failed to describe Key: NotFoundException: Invalid keyId efskey

Below is my example:

apiVersion: kms.aws.crossplane.io/v1alpha1
kind: Key
metadata:
  name: efskey
spec:
  deletionPolicy: Delete
  writeConnectionSecretToRef:
    name: efs-key
    namespace: default
  forProvider:
    description: Encryption key used to encrypt EFS resources
    region: ap-southeast-2
    policy: |-
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "Allow access to EFS for all principals in the account that are authorized to use EFS",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:CreateGrant",
                "kms:DescribeKey"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "kms:ViaService": "elasticfilesystem.ap-southeast-2.amazonaws.com",
                    "kms:CallerAccount": "YYYYYYYYYY"
                }
            }
          },
          {
            "Sid": "Allow direct access to key metadata to the account",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::YYYYYYYYYY:root"
            },
            "Action": [
                "kms:Describe*",
                "kms:Get*",
                "kms:List*",
                "kms:RevokeGrant"
            ],
            "Resource": "*"
          }
        ]
      }
    
    tags:
    - tagKey: Name
      tagValue: EFS

Crossplane version: 1.2.2
EKS version: 1.20

[haarchri]

today we also have one environment with this issue directly in eks...

looks like directly after applied the ressource in k8s - crossplane adds annotation for example:
crossplane.io/external-name: efs-key
but this is wrong because the external-name is
obj.KeyMetadata.KeyId

[Hammond95]

I have found the same issue still occurs.
I am running on the following versions:

EKS: 1.18
crossplane/crossplane:v1.2.1
crossplane/provider-aws-controller:v0.18.1

---

manifest-1.yaml - Invalid keyId

apiVersion: kms.aws.crossplane.io/v1alpha1
kind: Key
metadata:
  name: "martin"
  labels:
    purpose: paas-orgs
    provider: aws
spec:
  deletionPolicy: "Delete"
  providerConfigRef:
    name: crossplane-provider-aws
  forProvider:
    bypassPolicyLockoutSafetyCheck: false
    customerMasterKeySpec: "SYMMETRIC_DEFAULT"
    enabled: true
    keyUsage: "ENCRYPT_DECRYPT"
    origin: "AWS_KMS"
    policy: "POLICY JSON"
    region: "eu-west-1"
    tags: 
      - tagKey: author
        tagValue: crossplane-provider-aws
  writeConnectionSecretToRef:
    name: "martin-test-xplane-kms-key-secret"
    namespace: "crossplane-system"

Results:

• The k8s object is created

NAME             READY       SYNCED        EXTERNAL-NAME
martin             False          martin

• Getting the error: failed to describe Key: NotFoundException: Invalid keyId martin

---

manifest-2.yaml - keyId not found

Using a valid keyId but the key doesn't exist on the account, but my objective is to create a new key.

apiVersion: kms.aws.crossplane.io/v1alpha1
kind: Key
metadata:
  name: "martin"
  annotations:
    crossplane.io/external-name: "55555555-089b-4193-bf05-03f6a3e743c4"
  labels:
    purpose: paas-orgs
    provider: aws
spec:
  deletionPolicy: "Delete"
  providerConfigRef:
    name: crossplane-provider-aws
  forProvider:
    bypassPolicyLockoutSafetyCheck: false
    customerMasterKeySpec: "SYMMETRIC_DEFAULT"
    enabled: true
    keyUsage: "ENCRYPT_DECRYPT"
    origin: "AWS_KMS"
    policy: "... POLICY JSON ..."
    region: "eu-west-1"
    tags: 
      - tagKey: author
        tagValue: crossplane-provider-aws
  writeConnectionSecretToRef:
    name: "martin-test-xplane-kms-key-secret"
    namespace: "crossplane-system"

Results:

• The k8s object is created

NAME             READY       SYNCED        EXTERNAL-NAME
martin             False          martin            55555555-089b-4193-bf05-03f6a3e743c4

• Getting the error: failed to describe Key: NotFoundException: Key 'arn:aws:kms:eu-west-1:**********:key/55555555-089b-4193-bf05-03f6a3e743c4' does not exist

---

Using an existing keyId it works as expected attaching the k8s resource to the existing AWS one.

NAME                                   READY   SYNCED   EXTERNAL-NAME
7*******-0b**-4ce4-####-7e3d1*******   True    True     7*******-0b**-4ce4-####-7e3d1*******

[haarchri]

in my point of view the problem is that provider-aws directly after kubectl apply adds the k8s object also adds the annotation fir external-name
the following code-snip is never entered: https://github.com/crossplane/provider-aws/blob/master/pkg/controller/kms/key/zz_controller.go#L73 so the creation is not triggered

and the problem with failed to describe key is then from this code-snip: https://github.com/crossplane/provider-aws/blob/master/pkg/controller/kms/key/zz_controller.go#L82 because the Observer thoughts that he need to check if the key exists ...

[hasheddan]

Hey folks! So the issue with the external-name is due to the fact that the KMS key ID is non-deterministic I believe (correct me if I am wrong). What we would want to do in this case is override the default initializers, like we do here:
https://github.com/crossplane/provider-aws/blob/3f1b7187455f14c306b3037312a19a791ac9d9eb/pkg/controller/servicediscovery/httpnamespace/setup.go#L59

This is saying only use ProviderConfig defaulting and not the default NewNameAsExternalName() initializer, though we actually use OpenAPI default as well since this PR: crossplane/crossplane-runtime#252, so you could actually just pass managed.WithInitializers(). You will also need to set the external name after creation as @haarchri suggests.

The static credentials bug is a separate issue that needs to be addressed in authentication with the v1 API.

[haarchri]

we have changed to managed.WithInitializers() in #1050