Implement Managed Streaming for Kafka (MSK) - Cluster

[haarchri]

Signed-off-by: Christopher Haar chhaar30@googlemail.com

Description of your changes

kafka cluster (MSK)
Implements #415

I have:

• Read and followed Crossplane's contribution process.
• Run make reviewable test to ensure this PR is ready for review.

How has this code been tested

kubectl apply -f examples/kafka/cluster.yaml
cluster.kafka.aws.crossplane.io/kafka-test-server

kubectl get cluster.kafka.aws.crossplane.io/kafka-test-server

apiVersion: kafka.aws.crossplane.io/v1alpha1
kind: Cluster
metadata:
  annotations:
    crossplane.io/external-name: arn:aws:kafka:eu-central-1:255932642927:cluster/kafka-test-server/af7e1917-25f4-4418-861e-fa401d8beba4-1
  creationTimestamp: "2021-06-21T07:45:52Z"
  finalizers:
  - finalizer.managedresource.crossplane.io
  name: kafka-test-server
spec:
  deletionPolicy: Delete
  forProvider:
    brokerNodeGroupInfo:
      brokerAZDistribution: DEFAULT
      clientSubnets:
      - subnet-f2ccb898
      - subnet-aac361d6
      instanceType: kafka.t3.small
      securityGroups:
      - sg-7f714d0e
      storageInfo:
        ebsStorageInfo:
          volumeSize: 1
    clusterName: kafka-test-server
    encryptionInfo:
      encryptionAtRest:
        dataVolumeKMSKeyID: arn:aws:kms:eu-central-1:255932642927:key/6d90627a-663e-426c-b5c4-63578d217388
      encryptionInTransit:
        clientBroker: TLS
        inCluster: true
    enhancedMonitoring: DEFAULT
    kafkaVersion: 2.6.1
    numberOfBrokerNodes: 2
    openMonitoring:
      prometheus:
        jmxExporter:
          enabledInBroker: false
        nodeExporter:
          enabledInBroker: false
    region: eu-central-1
    tags:
      myKey: myValue
    zookeeperConnectString: z-3.kafka-test-server.xxxxxx.c1.kafka.eu-central-1.amazonaws.com:2181,z-2.kafka-test-server.xxxxxx.c1.kafka.eu-central-1.amazonaws.com:2181,z-1.kafka-test-server.xxxxxx.c1.kafka.eu-central-1.amazonaws.com:2181
    zookeeperConnectStringTLS: z-3.kafka-test-server.xxxxxx.c1.kafka.eu-central-1.amazonaws.com:2182,z-2.kafka-test-server.xxxxxx.c1.kafka.eu-central-1.amazonaws.com:2182,z-1.kafka-test-server.xxxxxx.c1.kafka.eu-central-1.amazonaws.com:2182
  providerConfigRef:
    name: default
status:
  atProvider:
    clusterARN: arn:aws:kafka:eu-central-1:255932642927:cluster/kafka-test-server/af7e1917-25f4-4418-861e-fa401d8beba4-1
    state: ACTIVE
  conditions:
  - lastTransitionTime: "2021-06-21T07:46:00Z"
    reason: ReconcileSuccess
    status: "True"
    type: Synced
  - lastTransitionTime: "2021-06-21T08:11:05Z"
    reason: Available
    status: "True"
    type: Ready

[hasheddan]

during deletion i see in debug the following message - but cluster is removed successfully in k8s and in aws ( process takes 5-10 minutes)
Message: delete failed: failed to delete Cluster: BadRequestException: You can't delete cluster in DELETING state. - any ideas here ?

@haarchri one option here would be to ignore this error in your postDelete hook. Let me know if you have any questions! :)

[negz]

This looks good to me apart from a few nits and once we address the delete issues.

[haarchri]

during deletion i see in debug the following message - but cluster is removed successfully in k8s and in aws ( process takes 5-10 minutes)
Message: delete failed: failed to delete Cluster: BadRequestException: You can't delete cluster in DELETING state. - any ideas here ?

@haarchri one option here would be to ignore this error in your postDelete hook. Let me know if you have any questions! :)

@hasheddan can you have a look ?
https://github.com/haarchri/provider-aws/blob/kafka/pkg/controller/kafka/cluster/setup.go#L71

thanks ;)

[hasheddan]

@haarchri assuming the BadRequest only happen on delete when the cluster is in DELETING state, this LGTM! I see you have a new merge commit, could you remove that and instead rebase your changes on master? Let me know if you have any questions about how to do so :)

[haarchri]

history cleanup done & final test done

aws-console:

crossplane:

kubectl apply -f cluster.yaml 
cluster.kafka.aws.crossplane.io/kafka-test-server created

NAME                                                READY   SYNCED   EXTERNAL-NAME
cluster.kafka.aws.crossplane.io/kafka-test-server   True    True     arn:aws:kafka:eu-central-1:255932642927:cluster/kafka-test-server/49ad028c-aead-4d05-8519-c725e0828854-1

kubectl delete cluster.kafka.aws.crossplane.io/kafka-test-server
cluster.kafka.aws.crossplane.io "kafka-test-server" deleted