aws-ddb-sink-binding doesn't send data to DynamoDB

[anuj-upadhyay-2001]

aws-ddb-sink-binding.yaml

apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
  name: aws-ddb-sink-binding
spec:
  source:
    ref:
      kind: KafkaTopic
      apiVersion: kafka.strimzi.io/v1beta1
      name: my-topic
  sink:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1alpha1
      name: aws-ddb-sink
    properties:
      table: kafkaTable
      region: ap-south-1
      accessKey: ********************************
      secretKey: *********************************

What i have already checked:
1] Strimzi is setup and working.
2] Kafka producer sends data in proper JSON Format
3] My partition key in the table is also present in the JSON sent by the Kafka producer
4] I also have an elasticsearch-index-sink setup with the same Kafka settings as the source and that sink connector is working perfectly.
5] The accessKey and secretKey properties are set to the AWS_ACCESS_KEY_ID and AWS_SECRET_KEY values respectively.

[oscerd]

What is the camel-k version you're using?

[oscerd]

Please add some information from the logs of the running integration.

[anuj-upadhyay-2001]

What is the camel-k version you're using?

https://camel.apache.org/camel-kamelets/0.8.x/aws-ddb-sink.html

I have been using 0.8.x kamelets for both Elasticsearch and DynamoDB

[oscerd]

Please try with the latest version 0.9.0 and Camel-K 1.10.0

[anuj-upadhyay-2001]

We tried with the latest version of Camel-K 1.10.0 and there is no difference between the yaml configurations on aws-ddb-sink between 0.8.x and 0.9.x. There is still no data being sent to the kafkaTable.

Describing the binding itself we don't see any events:

Name:         aws-ddb-sink-binding
Namespace:    kafka
Labels:       <none>
Annotations:  <none>
API Version:  camel.apache.org/v1alpha1
Kind:         KameletBinding
Metadata:
  Creation Timestamp:  2022-09-16T12:52:22Z
  Generation:          1
  Managed Fields:
    API Version:  camel.apache.org/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:sink:
          .:
          f:properties:
            .:
            f:accessKey:
            f:defaultCredentialProvider:
            f:region:
            f:secretKey:
            f:table:
          f:ref:
            .:
            f:apiVersion:
            f:kind:
            f:name:
        f:source:
          .:
          f:ref:
            .:
            f:apiVersion:
            f:kind:
            f:name:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2022-09-16T12:52:22Z
  Resource Version:  2540
  UID:               c57f0ba9-5ea6-4d27-a29c-46b6171db982
Spec:
  Sink:
    Properties:
      Access Key:                   ***********************
      Default Credential Provider:  false
      Region:                       ap-south-1
      Secret Key:                   ************************
      Table:                        kafkaTable
    Ref:
      API Version:  camel.apache.org/v1alpha1
      Kind:         Kamelet
      Name:         aws-ddb-sink
  Source:
    Ref:
      API Version:  kafka.strimzi.io/v1beta1
      Kind:         KafkaTopic
      Name:         my-topic
Events:             <none>

[oscerd]

Can you post your logs?

[anuj-upadhyay-2001]

I can get the logs for the camel-k-operator pod with the kubectl logs -p command.
but how would i go about getting logs for the binding? There is no way to get logs for type : kameletBinding

[anuj-upadhyay-2001]

logs.txt

[oscerd]

You need to kamel log the name of your integration and check there.

[anuj-upadhyay-2001]

>>kamel log aws-ddb-sink-binding
Integration 'aws-ddb-sink-binding' not yet available. Will keep checking ...

[oscerd]

As far as I see the integration is named aws-ddb-sink-binding

[anuj-upadhyay-2001]

As far as I see the integration is named aws-ddb-sink-binding

aws-ddb-sink-binding is also not available.
Also when i run kamel get integration
it shows no integrations

[oscerd]

It's hard to say in this way.

[anuj-upadhyay-2001]

Let me explain the whole situation:

We have a docker-desktop provisioned kubernetes cluster on a windows host.

We have setup a strimizi kafka cluster and produced a few messages.

We have already tried the elasticsearch-index-sink with kafka as source and its elasticsearch-index-sink-binding has worked flawlessly.

Even though the integration worked with ELK, when we run kamel get integration we don't actually see the elasticsearch-index-sink-binding.

and kamel log elasticsearch-index-sink-binding returns:

Integration 'elasticsearch-index-sink-binding' not yet available. Will keep checking ...

even though the integration is actually working and we can see our data in a kibana view.

But the aws-ddb-sink-binding doesn't work with the same source and proper configuration.

[oscerd]

I will try to find the time to test. You should also do something kamel get klb, to see if kameletBinding has been created or not.

[anuj-upadhyay-2001]

Kamel get klb reveals that no binding has been created, in both elasticsearch-index-sink and aws-ddb-sink but es is working while ddb isn't.

[oscerd]

This should be related: https://issues.apache.org/jira/browse/CAMEL-18603

[anuj-upadhyay-2001]

This should be related: https://issues.apache.org/jira/browse/CAMEL-18603

I see that the issue is resolved, i will wait for new Camel release 3.2.0 and the subsequent Camel-K release which will hopefully be using the updated Camel version and then test again.

[oscerd]

This is should be fixed in 1.10.3 and 1.11.0. Re-open in case it's not.