Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running YAML code #1579

Closed
doru1004 opened this issue Jun 26, 2020 · 32 comments
Closed

Running YAML code #1579

doru1004 opened this issue Jun 26, 2020 · 32 comments

Comments

@doru1004
Copy link
Contributor

I am trying to transform a Camel route with a simple processor into a yaml version of itself.

I have my processor in a different file MyProcessor.java.

I have tried writing something like this:

- from:
    uri: "timer:tick"
    parameters:
      period: "1000"
    steps:
      - process:
          class: MyProcessor
      - to: "log:info"

But that is a guess and when I do kamel run it doesn't actually work:

[1] Exception in thread "main" org.apache.camel.FailedToCreateRouteException: Failed to create route route1 at: >>> process[Processor@0x0] <<< in route: Route(route1)[From[timer://tick?period=1000] -> [process[Pro... because of ref must be specified on: process[Processor@0x0]
[1] 	at org.apache.camel.reifier.RouteReifier.doCreateRoute(RouteReifier.java:393)
[1] 	at org.apache.camel.reifier.RouteReifier.createRoute(RouteReifier.java:117)
[1] 	at org.apache.camel.impl.DefaultCamelContext.startRouteDefinitions(DefaultCamelContext.java:374)
[1] 	at org.apache.camel.impl.DefaultCamelContext.startRouteDefinitions(DefaultCamelContext.java:348)
[1] 	at org.apache.camel.impl.engine.AbstractCamelContext.doInit(AbstractCamelContext.java:2617)
[1] 	at org.apache.camel.support.service.BaseService.init(BaseService.java:83)
[1] 	at org.apache.camel.impl.engine.AbstractCamelContext.init(AbstractCamelContext.java:2450)
[1] 	at org.apache.camel.support.service.BaseService.start(BaseService.java:111)
[1] 	at org.apache.camel.impl.engine.AbstractCamelContext.start(AbstractCamelContext.java:2467)
[1] 	at org.apache.camel.k.main.ApplicationRuntime$MainAdapter.doStart(ApplicationRuntime.java:206)
[1] 	at org.apache.camel.support.service.BaseService.start(BaseService.java:115)
[1] 	at org.apache.camel.main.MainSupport.run(MainSupport.java:83)
[1] 	at org.apache.camel.k.main.ApplicationRuntime.run(ApplicationRuntime.java:70)
[1] 	at org.apache.camel.k.main.Application.main(Application.java:42)
[1] Caused by: java.lang.IllegalArgumentException: ref must be specified on: process[Processor@0x0]
[1] 	at org.apache.camel.util.ObjectHelper.notNull(ObjectHelper.java:171)
[1] 	at org.apache.camel.reifier.ProcessReifier.createProcessor(ProcessReifier.java:39)
[1] 	at org.apache.camel.reifier.ProcessorReifier.makeProcessor(ProcessorReifier.java:766)
[1] 	at org.apache.camel.reifier.ProcessorReifier.addRoutes(ProcessorReifier.java:511)
[1] 	at org.apache.camel.reifier.RouteReifier.doCreateRoute(RouteReifier.java:391)

Are there any examples out there which invoke a processor as one of the steps?

@lburgazzoli
Copy link
Contributor

A fix for this will be available in the next release of the camel-k-runtime, however storing arbitrary code in a CR is not the recommended approach and best is to leverage jitpack or publish your artifact to a maven repo

@doru1004
Copy link
Contributor Author

Just to make sure I understand, is there no yaml way in which I could specify a custom processor? The yaml code I wrote there my be wrong, in particular this part:

- process:
          class: MyProcessor

I understand the process step is supported and I'm wondering what can I actually pass to it if not a custom class.

@doru1004
Copy link
Contributor Author

In other words, what would be the correct syntax for passing a processor class to the process step.

@lburgazzoli
Copy link
Contributor

The current implementation only accept a reference to a Processor stored in the registry so to reference a class you should use the bean dsl, however in the next release there should be support for that syntax too (the property it’s probably not be named class)

@doru1004
Copy link
Contributor Author

Got it. That's helpful to know!

@doru1004
Copy link
Contributor Author

What is the approximate timeline for the next release?

@doru1004
Copy link
Contributor Author

Do you have an example of how this can be done using YAML? What's unclear to me is how this method:

bindBeans(camelContext.getRegistry());

can be called from yaml (where bindBeans binds MyProcessor to the Camel registry).

@davsclaus
Copy link
Contributor

You can add @BindToRegisty on the MyProcessor class

@doru1004
Copy link
Contributor Author

I did that and then added this to my route:

.to("bean:MyProcessor")

I got an error saying:

No bean could be found in the registry for: MyProcessor

@doru1004
Copy link
Contributor Author

doru1004 commented Jun 28, 2020

Note to the above comment: I am doing this in an all Camel example and I was able to get it to work by not using the @BindToRegistry annotation but by doing this:

camelContext.getRegistry().bind("MyProcessor", new MyProcessor());

How do I transition from this to using the annotation instead? Or is that possible only with camel k ?

@davsclaus
Copy link
Contributor

Yeah that would work as you use the Java API.

But ideally we should make it possible to auto register beans via that annotation (or if you use cdi annotation with quarkus enabled).

Also for what's it work, you can also register via properties

camel.beans.myProcessor=#class:com.foo.MyProcessor

@doru1004
Copy link
Contributor Author

So you mean doing something like this:

kamel run --property camel.beans.myProcessor=#class:com.foo.MyProcessor

?

@doru1004
Copy link
Contributor Author

The current implementation only accept a reference to a Processor stored in the registry so to reference a class you should use the bean dsl,

What's the YAML side code for using the reference?

@lburgazzoli
Copy link
Contributor

I've been recommending to use something like:

- from:
    uri: "direct:route"
    steps:
      - bean:
          bean-type: org.apache.camel.k.loader.yaml.support.MyUppercaseProcessor
      - to: "mock:route"

does that work for you ?

@lburgazzoli
Copy link
Contributor

lburgazzoli commented Jun 29, 2020

The current implementation only accept a reference to a Processor stored in the registry so to reference a class you should use the bean dsl,

What's the YAML side code for using the reference?

You need to bind a processor to the context with what @davsclaus described and the using the ref property of the process step, something like:

- from:
    uri: "direct:route"
    steps:
      - process:
          ref: "myProcessor"
      - to: "mock:route"

The processor need to be bound to the context, i.e. by using an additional class, like:

public class MyConfigurer {
    @BindToRegistry
    public myProcessor() {
        return e -> { ... };
    }
}

And the run it like:

kamel run integration.yaml MyConfigurer.java

btw, we have also a gitter room and a mailing list for general discussions, whereas here is to report issues.

[1] https://gitter.im/apache/camel-k
[2] https://camel.apache.org/community/mailing-list

@doru1004
Copy link
Contributor Author

Thank you for the detailed explanation. I'm working on getting it work right now.

A small issue occurs when I try to run this command:

kamel run integration.yaml MyConfigurer.java

I get:

Error: unable to determine integration name

It seems to happen when I pass 2 inputs. Passing each one of those files individually does not result in that error.

@davsclaus
Copy link
Contributor

Ah you can specify the name via --name parameter.

@davsclaus
Copy link
Contributor

I guess maybe Kamel CLI should just compute the name from the 1st file

@doru1004
Copy link
Contributor Author

The --name fixed it :) Thanks.

@doru1004
Copy link
Contributor Author

I am extremely close to getting this to work.

A quick question: MyProcessor class uses other classes, defined in separate files. How do I tell kamel about those dependencies?

I tried passing the JAVA files to the to "kamel run" but that didn't work.

@lburgazzoli
Copy link
Contributor

I’d suggest to use jitpack or similar solution for that

@doru1004
Copy link
Contributor Author

I see, for now I just put all the classes into the file containing MyConfigurer and that seems to have been fine although not an ideal solution.

@doru1004
Copy link
Contributor Author

Does MyProcessor require any annotation as @Component or something like that?

Now that I have no more compilation errors I get:

[1] 2020-06-29 19:43:16.722 INFO  [main] ApplicationRuntime - Listener org.apache.camel.k.listener.RoutesConfigurer@2d6764b2 executed in phase ConfigureRoutes
[1] Exception in thread "main" org.apache.camel.RuntimeCamelException: java.lang.ClassNotFoundException: MyProcessor
[1] 	at org.apache.camel.RuntimeCamelException.wrapRuntimeException(RuntimeCamelException.java:68)
[1] 	at org.apache.camel.support.service.BaseService.doFail(BaseService.java:411)
[1] 	at org.apache.camel.support.service.BaseService.fail(BaseService.java:337)
[1] 	at org.apache.camel.support.service.BaseService.init(BaseService.java:88)
[1] 	at org.apache.camel.main.MainSupport.run(MainSupport.java:79)
[1] 	at org.apache.camel.k.main.ApplicationRuntime.run(ApplicationRuntime.java:70)
[1] 	at org.apache.camel.k.main.Application.main(Application.java:42)

which looks like a Camel specific problem. I suspect my reference to MyProcessor isn't quite correct.

kamel run --property camel.beans.myProcessor=#class:MyProcessor ...

And my configuration class looks like this:

public class RegistryConfigurer {
    @BindToRegistry
    public MyProcessor myProcessor() {
        return new MyProcessor();
    }
}

Am I missing some annotation on the MyProcessor class? (@Configuration or @Component ? )

@doru1004
Copy link
Contributor Author

doru1004 commented Jun 29, 2020

It looks like at the time of binding the bean to the registry MyProcessor class is not found.

[1] Caused by: java.lang.ClassNotFoundException: MyProcessor
[1] 	at org.apache.camel.impl.engine.DefaultClassResolver.resolveMandatoryClass(DefaultClassResolver.java:87)
[1] 	at org.apache.camel.support.PropertyBindingSupport.resolveBean(PropertyBindingSupport.java:1031)
[1] 	at org.apache.camel.main.BaseMainSupport.bindBeansToRegistry(BaseMainSupport.java:928)
[1] 	at org.apache.camel.main.BaseMainSupport.doConfigureCamelContextFromMainConfiguration(BaseMainSupport.java:738)
[1] 	at org.apache.camel.main.BaseMainSupport.autoconfigure(BaseMainSupport.java:507)
[1] 	at org.apache.camel.main.BaseMainSupport.postProcessCamelContext(BaseMainSupport.java:570)
[1] 	at org.apache.camel.main.BaseMainSupport.initCamelContext(BaseMainSupport.java:379)
[1] 	at org.apache.camel.k.main.ApplicationRuntime$MainAdapter.doInit(ApplicationRuntime.java:197)
[1] 	at org.apache.camel.support.service.BaseService.init(BaseService.java:83)
[1] 	... 3 more

@doru1004
Copy link
Contributor Author

MyProcessor as a class is defined in the MyConfigurer.java file which I pass to kamel run.

@lburgazzoli
Copy link
Contributor

lburgazzoli commented Jun 30, 2020

@doru1004 I'm honestly lost about what you are doing.

So it seems you have both a @BindToRegistry and --property camel.beans.myProcessor=#class:MyProcessor and well, both are supposed to do the same thing.

As already stated there is an issue in referencing a class by name if the class is provided as a CR (i.e. using kamel run) so camel.beans.myProcessor=#class:MyProcessor won't wor and this will be fixed in the next version.

camel-k is not a generic build system so adding arbitrary classes to the kamel run won't always work by design and as I already said a number of time, use jitpack or publish your artefact to a maven repo.

@doru1004
Copy link
Contributor Author

What I am trying to do is have a working application using Camel K. I have managed to do a Java only application. What I would now want to do is rewrite the Camel integration in Yaml.

Some initial features that I want to exercise are:

  1. sending HTTP requests to some third-party service and getting a response
  2. have a custom processor to process that response

I would like to do this from YAML to see how that can be done using YAML instead of Camel Java code.

In this particular issue I am trying to handle point 2.

It seems that the problematic part is being able to include processor part.

This is my call right now after you telling me that the --property is not required:

kamel run --name=test routes.yaml RegistryConfigurer.java --dev

routes contains this:

- from:
    uri: "timer:clock"
    parameters:
      period: "1000"
    steps:
      - set-body:
        constant: "foo"
      - process:
        ref: "myProcessor"
      - to: log:info

Registry configurer file contains this:

... 
class MyProcessor implements Processor {
  void public process(...) {
     ... 
  }
}

public class RegistryConfigurer {
    @BindToRegistry
    public MyProcessor myProcessor() {
        return new MyProcessor();
    }
}

After eliminating the --property flag from the kamel run CLI, I get the following error. Can this error be fixed if I publish my class in a maven repo?

[1] 2020-06-30 13:39:36.150 INFO  [main] ApplicationRuntime - Listener org.apache.camel.k.listener.RoutesConfigurer@51972dc7 executed in phase ConfigureRoutes
[1] Exception in thread "main" java.lang.IllegalStateException: com.fasterxml.jackson.databind.JsonMappingException: Step should not have more tha one child (through reference chain: org.apache.camel.k.loader.yaml.parser.FromStepParser$FromStepDefinition["steps"]->java.util.ArrayList[0])
[1] 	at org.apache.camel.k.loader.yaml.spi.StepParser$Context.node(StepParser.java:83)
[1] 	at org.apache.camel.k.loader.yaml.parser.FromStepParser.toStartProcessor(FromStepParser.java:38)
[1] 	at org.apache.camel.k.loader.yaml.spi.StartStepParser.invoke(StartStepParser.java:29)
[1] 	at org.apache.camel.k.loader.yaml.YamlSourceLoader$1.configure(YamlSourceLoader.java:93)
[1] 	at org.apache.camel.builder.RouteBuilder.checkInitialized(RouteBuilder.java:473)
[1] 	at org.apache.camel.builder.RouteBuilder.configureRoutes(RouteBuilder.java:421)
[1] 	at org.apache.camel.builder.RouteBuilder.addRoutesToCamelContext(RouteBuilder.java:401)
[1] 	at org.apache.camel.impl.engine.AbstractCamelContext.addRoutes(AbstractCamelContext.java:1190)
[1] 	at org.apache.camel.main.RoutesConfigurer.configureRoutes(RoutesConfigurer.java:92)
[1] 	at org.apache.camel.main.BaseMainSupport.configureRoutes(BaseMainSupport.java:552)
[1] 	at org.apache.camel.main.BaseMainSupport.postProcessCamelContext(BaseMainSupport.java:572)
[1] 	at org.apache.camel.main.BaseMainSupport.initCamelContext(BaseMainSupport.java:379)
[1] 	at org.apache.camel.k.main.ApplicationRuntime$MainAdapter.doInit(ApplicationRuntime.java:197)
[1] 	at org.apache.camel.support.service.BaseService.init(BaseService.java:83)
[1] 	at org.apache.camel.main.MainSupport.run(MainSupport.java:79)
[1] 	at org.apache.camel.k.main.ApplicationRuntime.run(ApplicationRuntime.java:70)
[1] 	at org.apache.camel.k.main.Application.main(Application.java:42)
[1] Caused by: com.fasterxml.jackson.databind.JsonMappingException: Step should not have more tha one child (through reference chain: org.apache.camel.k.loader.yaml.parser.FromStepParser$FromStepDefinition["steps"]->java.util.ArrayList[0])
[1] 	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:397)
[1] 	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:368)
[1] 	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:302)
[1] 	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)
[1] 	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:27)
[1] 	at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
[1] 	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
[1] 	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
[1] 	at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1719)
[1] 	at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1350)
[1] 	at org.apache.camel.k.loader.yaml.spi.StepParser$Context.node(StepParser.java:81)
[1] 	... 16 more
[1] Caused by: java.lang.IllegalStateException: Step should not have more tha one child
[1] 	at org.apache.camel.k.loader.yaml.model.Step$Deserializer.deserialize(Step.java:52)
[1] 	at org.apache.camel.k.loader.yaml.model.Step$Deserializer.deserialize(Step.java:38)
[1] 	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:286)

@lburgazzoli
Copy link
Contributor

That error is probably about a wrong YAML indentation, as per the YAML guide, each step is represented by YAML map that has a single entry where the field name is the EIP name.

@lburgazzoli
Copy link
Contributor

Here an example of a working code:

- from:
    uri: "timer:clock"
    parameters:
      period: "1000"
    steps:
      - set-body:
          constant: "foo"
      - process:
          ref: "toUpper"
      - to: log:info
import org.apache.camel.Processor;
import org.apache.camel.BindToRegistry;

public class MyConfigurer {
    @BindToRegistry
    public Processor toUpper() {
        return e -> e.getMessage().setBody(
            e.getMessage().getBody(String.class).toUpperCase()
        );
    }
}

tested on OpenShift with the following command:

kamel run --dev --name example routes.yaml MyConfigurer.java

@doru1004
Copy link
Contributor Author

Thank you so much!! Thanks a lot for the patience. I managed to get it to work!! :) After taking out the --property from the command line It looks like in the end it really was just an indentation issue in my YAML. This was really helpful though! I'm happy to close this now and I hope this issue helps other users of Camel K also.

@sidharthramesh
Copy link

Hey @lburgazzoli, thank you for your example. I'm trying out the same code and commands with my camel-k installation, and I get this error:

org.apache.camel.NoSuchBeanException: No bean could be found in the registry for: toUpper of type: org.apache.camel.Processor
[2]     at org.apache.camel.reifier.AbstractReifier.mandatoryLookup(AbstractReifier.java:145)
[2]     at org.apache.camel.reifier.ProcessReifier.createProcessor(ProcessReifier.java:41)
[2]     at org.apache.camel.reifier.ProcessorReifier.makeProcessor(ProcessorReifier.java:844)
[2]     at org.apache.camel.reifier.ProcessorReifier.addRoutes(ProcessorReifier.java:585)
[2]     at org.apache.camel.reifier.RouteReifier.doCreateRoute(RouteReifier.java:236)
[2]     at org.apache.camel.reifier.RouteReifier.createRoute(RouteReifier.java:74)
[2]     at org.apache.camel.impl.DefaultModelReifierFactory.createRoute(DefaultModelReifierFactory.java:49)
[2]     at org.apache.camel.impl.DefaultCamelContext.startRouteDefinitions(DefaultCamelContext.java:887)
[2]     at org.apache.camel.impl.DefaultCamelContext.startRouteDefinitions(DefaultCamelContext.java:775)
[2]     at org.apache.camel.impl.engine.AbstractCamelContext.doInit(AbstractCamelContext.java:2937)
[2]     at org.apache.camel.quarkus.core.FastCamelContext.doInit(FastCamelContext.java:172)
[2]     at org.apache.camel.support.service.BaseService.init(BaseService.java:83)
[2]     at org.apache.camel.impl.engine.AbstractCamelContext.init(AbstractCamelContext.java:2620)
[2]     at org.apache.camel.support.service.BaseService.start(BaseService.java:111)
[2]     at org.apache.camel.impl.engine.AbstractCamelContext.start(AbstractCamelContext.java:2639)
[2]     at org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:255)
[2]     at org.apache.camel.quarkus.main.CamelMain.doStart(CamelMain.java:94)
[2]     at org.apache.camel.support.service.BaseService.start(BaseService.java:119)
[2]     at org.apache.camel.quarkus.main.CamelMain.startEngine(CamelMain.java:140)
[2]     at org.apache.camel.quarkus.main.CamelMainRuntime.start(CamelMainRuntime.java:49)
[2]     at org.apache.camel.quarkus.core.CamelBootstrapRecorder.start(CamelBootstrapRecorder.java:45)
[2]     at io.quarkus.deployment.steps.CamelBootstrapProcessor$boot173480958.deploy_0(Unknown Source)
[2]     at io.quarkus.deployment.steps.CamelBootstrapProcessor$boot173480958.deploy(Unknown Source)
[2]     at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
[2]     at io.quarkus.runtime.Application.start(Application.java:101)
[2]     at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:103)
[2]     at io.quarkus.runtime.Quarkus.run(Quarkus.java:67)
[2]     at io.quarkus.runtime.Quarkus.run(Quarkus.java:41)
[2]     at io.quarkus.runtime.Quarkus.run(Quarkus.java:120)
[2]     at io.quarkus.runner.GeneratedMain.main(Unknown Source)
[2]     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[2]     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[2]     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2]     at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[2]     at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:60)
[2]     at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:31)

Just have the 2 files as it is and running the same command:

kamel run --dev --name example integration.yaml MyConfigurer.java

@squakez
Copy link
Contributor

squakez commented Aug 29, 2022

@sidharthramesh I think this is no longer a valid way because some classloading changes introduced in the latest Camel versions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants