Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Missing data type transformer from null to type camel:azure-storage-blob:application-cloudevents on the exchange #1777

Closed
matzew opened this issue Dec 6, 2023 · 24 comments
Closed

Missing data type transformer from null to type camel:azure-storage-blob:application-cloudevents on the exchange #1777

matzew opened this issue Dec 6, 2023 · 24 comments

Comments

@matzew
Copy link
Member

matzew commented Dec 6, 2023

Trying to use the azure-storage-blog Kamelet, like:

apiVersion: camel.apache.org/v1
kind: Pipe
metadata:
  name: azure-storage-blob-source-pipe
spec:
  integration:
    dependencies:
    - "camel:cloudevents"
  source:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1
      name: azure-storage-blob-source
    dataTypes:
      out:
        format: azure-storage-blob:application-cloudevents
    properties:
      accountName: XXX
      containerName: YYY
      accessKey: some-key
  sink:
    # dataTypes:
    #   in:
    #     format: http:application-cloudevents
    ref:
      kind: Broker
      apiVersion: eventing.knative.dev/v1
      name: kamelet

But I am getting:

azure-storage-blob-source-pipe-767955c64f-jjdwh integration 2023-12-06 11:00:33,820 WARN  [org.apa.cam.com.azu.sto.blo.BlobConsumer] (Camel (camel-1) thread #1 - azure-storage-blob://XXX/YYYcontainer) Error processing exchange. Exchange[SOMETHING-something]. Caused by: [org.apache.camel.CamelExecutionException - Missing data type transformer from null to type camel:azure-storage-blob:application-cloudevents on the exchange: Exchange[]]: org.apache.camel.CamelExecutionException: Missing data type transformer from null to type camel:azure-storage-blob:application-cloudevents on the exchange: Exchange[]
azure-storage-blob-source-pipe-767955c64f-jjdwh integration 	at org.apache.camel.processor.transformer.DataTypeProcessor.process(DataTypeProcessor.java:86)
azure-storage-blob-source-pipe-767955c64f-jjdwh integration 	at jdk.internal.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
azure-storage-blob-source-pipe-767955c64f-jjdwh integration 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
azure-storage-blob-source-pipe-767955c64f-jjdwh integration 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
azure-storage-blob-source-pipe-767955c64f-jjdwh integration 	at org.apache.camel.support.ObjectHelper.invokeMethodSafe(ObjectHelper.java:382)

When I remove the following:

    dataTypes:
      out:
        format: azure-storage-blob:application-cloudevents

than I get binary into my "consumer function" (behind a trigger of the given brker)
@christophd
Copy link
Contributor

That's probably because you run Camel K with Apache Camel 3.20.x which is the default in Camel K 2.x

You can try to update to Camel 4.x as described in this post: https://camel.apache.org/blog/2023/09/camel-4-on-camel-k/

@christophd
Copy link
Contributor

Oh wait, wrong Apache Camel 4.x is the default since Camel K 2.1

@christophd
Copy link
Contributor

Sorry for the noise. must be something else. Let me see where this data type is supposed to live

@christophd
Copy link
Contributor

I don't know exactly, but the data type implementation for azure-storage-blob:application-cloudevents seems to be missing in the Camel version of your Pipe

Your Pipe uses a Camel and Kamelets version in the runtime and the data type might not be available in this version.

I recall that the specific data type has been moved from Kamelets utils library to Camel core library in the past. I try to figure out what version this data type must be available

@matzew
Copy link
Member Author

matzew commented Dec 6, 2023

Thanks, @christophd
I am running of this installation: https://github.com/matzew/eventing-cloud-integration/tree/main/system

@oscerd
Copy link
Contributor

oscerd commented Dec 6, 2023

The data type for Azure Storage Blob has been added in camel-kamelets 4.1.0 and migrated in Camel in 4.2.0.

So in the last official release of Camel K (the 2.1.0) the data type for Azure Storage Blob is not available.

@matzew
Copy link
Member Author

matzew commented Dec 6, 2023

Thanks - can I override that somehow?

@matzew
Copy link
Member Author

matzew commented Dec 6, 2023

So, main is also still using, 4.0.0
https://github.com/apache/camel-k/blob/main/script/Makefile#L89

@oscerd
Copy link
Contributor

oscerd commented Dec 6, 2023

So, main is also still using, 4.0.0 https://github.com/apache/camel-k/blob/main/script/Makefile#L89

Yes, also the nightly are still using that version.

@oscerd
Copy link
Contributor

oscerd commented Dec 6, 2023

Thanks - can I override that somehow?

One thing you could try to do, it's overriding the Kamelets. In particular you could try to change this line in the Kamelet you have in the cluster: https://github.com/apache/camel-kamelets/blob/v4.1.0/kamelets/azure-storage-blob-source.kamelet.yaml#L155

and use the 4.1.0 version instead of 4.0.0 (I suppose it's 4.0.0 listed there). That way you could at least have the bean from the library.

Let us know if it works.

@matzew
Copy link
Member Author

matzew commented Dec 6, 2023

@oscerd you mean in here:

apiVersion: camel.apache.org/v1
kind: Pipe
metadata:
  name: azure-storage-blob-source-pipe
spec:
  integration:
    dependencies:
    - "camel:cloudevents"
  source:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1
      name: azure-storage-blob-source
    dataTypes:
      out:
        format: azure-storage-blob:application-cloudevents
    properties:
      accountName: XXX
      containerName: YYY
      accessKey: some-key
  sink:
    # dataTypes:
    #   in:
    #     format: http:application-cloudevents
    ref:
      kind: Broker
      apiVersion: eventing.knative.dev/v1
      name: kamelet

@oscerd
Copy link
Contributor

oscerd commented Dec 6, 2023

No, you need to modify the azure-storage-blob-source Kamelet definition and look at the dependency section

@oscerd
Copy link
Contributor

oscerd commented Dec 6, 2023

It's a CR installed by camel k

@christophd
Copy link
Contributor

Camel K runtime needs to wait for Camel-Quarkus to update to Camel 4.1.x or 4.2.x

The latest available Camel-Quarkus version is 3.2.3 which uses Camel 4.0.3

Camel K is the last piece in a long chain of dependencies. We try to change this in order to loosen the dependency chain a bit

@matzew
Copy link
Member Author

matzew commented Dec 7, 2023

So, I tried to patch it:

kubectl patch kamelets.camel.apache.org azure-storage-blob-source --type='json' -p='[{"op": "replace", "path": "/spec/dependencies/4", "value": "mvn:org.apache.camel.kamelets:camel-kamelets-utils:4.1.0"}]'

But, in the pod for the azure-storage... pipe, I do still see:

org.apache.camel.CamelExecutionException: Missing data type transformer from null to type camel:azure-storage-blob:application-cloudevents on the exchange: Exchange[]
	at org.apache.camel.processor.transformer.DataTypeProcessor.process(DataTypeProcessor.java:86)
	at jdk.internal.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)

@oscerd
Copy link
Contributor

oscerd commented Dec 7, 2023

I have no other attempt to try, I think you should wait for a new camel-k release.

@matzew
Copy link
Member Author

matzew commented Dec 7, 2023

Let's hope that the nightly is updated before 😄 it still points to 4.0.0, for some reason

@oscerd
Copy link
Contributor

oscerd commented Dec 7, 2023

@squakez is is feasible to try to have a nightly pointing a new Camel version 4.2.0?

@matzew matzew mentioned this issue Dec 7, 2023
Closed
@christophd
Copy link
Contributor

@oscerd the problem is that we need a Camel-Quarkus release pointing to Camel 4.2.0. But I just learned that there has been such a release of Camel-Quarkus 3.6 pointing to Camel 4.2.0 last week.

So we can use that one and do a new Camel K runtime release that will then use Camel 4.2.0

@oscerd
Copy link
Contributor

oscerd commented Dec 7, 2023

Yes, it sounds good.

@christophd
Copy link
Contributor

In the meantime you could try one more thing:

Go to and apply this Kamelet in your user namespace:
https://github.com/apache/camel-kamelets/blob/4.1.x/kamelets/azure-storage-blob-source.kamelet.yaml

It points to the Kamelets utils library 4.1.0 which should have the data type included.

I think it is worth a try

@christophd
Copy link
Contributor

@matzew I see your patch may have done something similar using Kamelets utils library 4.1.0 but I am not sure if this change really gets into the Pipe runtime because the image for the Pipe may not be rebuilt because of that patch.

To be sure you may delete all integrationkits from the operator namespace. k delete integrationkit --all -n operator-namespace first and the try again

@oscerd
Copy link
Contributor

oscerd commented Dec 7, 2023

He tried that already, I guess

@matzew matzew mentioned this issue Dec 12, 2023
Closed
@christophd christophd mentioned this issue Jan 19, 2024
Merged
@christophd
Copy link
Contributor

Turns out this is related to camel-k#5047 because service discovery mechanism for data types has been broken. Fixed with camel-k#5055

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@matzew @christophd @oscerd