CAMEL-12665 - Apache Pulsar integration via Component by foamdino · Pull Request #2849 · apache/camel

foamdino · 2019-03-29T07:22:28Z

Contains new camel-pulsar component and example usage camel-example-spring-pulsar.

Looking forward to feedback on design/style code quality.

oscerd

The component must be added to the kit, the component starter seems to be missing, we may need to add a Karaf feature to support this on OSGi container and we need integration tests for Karaf and Spring Boot.
First we need to clean up everything a bit, lot of TODOs and some unused classes or empty methods etc.

oscerd · 2019-03-29T07:54:24Z

@davsclaus please have a look too :-)

onderson · 2019-03-29T08:47:16Z

+import org.slf4j.LoggerFactory;
+
+@Ignore //TODO use TestContainers to spin up local pulsar broker
+public class PulsarConsumerInTest extends CamelTestSupport {


why not having, single test class. This is a fast review. I may not see the reason promptly

oscerd · 2019-04-05T08:37:55Z

What's the status of this one? Once we have this finished, we need a new PR for master

foamdino · 2019-04-05T10:29:54Z

I'm going to spend some time doing the last changes today (against this branch), then open a new PR against master when this version is finished.

foamdino · 2019-04-09T09:49:15Z

This is now ready for another review.

oscerd

Only that minor comment. Probably you need to run again the profile sourcecheck and fix Codestyle

oscerd · 2019-04-09T10:03:25Z

btw we have a camel-testcontainers and camel-testcontainers-spring, so you can use them instead of using the explicit dependency. There are examples, like camel-nats or camel-consul

oscerd · 2019-04-09T12:22:03Z

Thanks

…revents mocking

foamdino · 2019-04-10T05:05:05Z

Pushed a load of checkstyle changes

oscerd · 2019-04-10T08:26:44Z

We need to align the component code and example code in 2.x to what we have in master now.

foamdino · 2019-04-10T10:59:23Z

We need to align the component code and example code in 2.x to what we have in master now.
Ok I can revert the checkstyle changes to make this closer to the version in master - obviously the changes to the package names due to the master changes will have to stay

oscerd · 2019-04-10T11:01:34Z

Yes, but something is not like camel 3, for example the META-INF stuff for the component is still to be explicitly added when on 2.x, while on 3.x is just an annotation that will do the trick... So it's not an easy operation to backport from master to 2.x

… as it prevents mocking" This reverts commit c48c4df.

oscerd · 2019-04-11T06:25:00Z

+    private final PulsarClient pulsarClient;
+
+    PulsarComponent(CamelContext context, AutoConfiguration autoConfiguration, PulsarClient pulsarClient) {
+        super(context);


The constructor must be changed like on master.

davsclaus · 2019-04-11T09:33:29Z

Is the persistence and non-persistence part of Apache Pulsar topic? eg it looks like you pass on this to pulsar and it knows about it.

I am asking as the endpoint uri syntax is a bit wrong and we need to tidy this up a bit. For example we may want to have an enum with those 2 values so Camel knows what you can specify which also allows tooling to know that.

And do you need that double slash in the topic name?

eg can this

persistent://public/default/camel-concurrent-producers-topic

become

persistent:public/default/camel-concurrent-producers-topic

oscerd · 2019-04-11T09:35:51Z

Is the persistence and non-persistence part of Apache Pulsar topic? eg it looks like you pass on this to pulsar and it knows about it.

I am asking as the endpoint uri syntax is a bit wrong and we need to tidy this up a bit. For example we may want to have an enum with those 2 values so Camel knows what you can specify which also allows tooling to know that.

And do you need that double slash in the topic name?

eg can this
persistent://public/default/camel-concurrent-producers-topic
become
persistent:public/default/camel-concurrent-producers-topic

We can improve it for sure. On master I had to change it to persistence in uri:syntax because it breaks the adoc rendering

davsclaus · 2019-04-11T09:39:15Z

Also we need to make the unit tests run when there is no docker environment, so you dont get failures.

rshermanTHG · 2019-04-11T09:49:15Z

Is the persistence and non-persistence part of Apache Pulsar topic? eg it looks like you pass on this to pulsar and it knows about it.
I am asking as the endpoint uri syntax is a bit wrong and we need to tidy this up a bit. For example we may want to have an enum with those 2 values so Camel knows what you can specify which also allows tooling to know that.
And do you need that double slash in the topic name?
eg can this
persistent://public/default/camel-concurrent-producers-topic
become
persistent:public/default/camel-concurrent-producers-topic
We can improve it for sure. On master I had to change it to persistence in uri:syntax because it breaks the adoc rendering

In Pulsar a topic is either persistent or non-persistent. It must include the double forward slashes after the colon when passed to pulsar i.e persistent://public/default/camel-concurrent-producers-topic

oscerd · 2019-04-15T06:40:43Z

what's the status of this one?

rshermanTHG · 2019-04-18T09:41:29Z

We need to align the component code and example code in 2.x to what we have in master now.

Have aligned the changes done in master.

oscerd · 2019-04-23T10:39:36Z

I'm working on the merge. @foamdino and @rshermanTHG can you please improve the documentation under src/main/docs on master?

oscerd · 2019-04-23T11:02:35Z

The build is failing. Can you please check on 2.x?

[ERROR] Failed to execute goal org.apache.camel:camel-package-maven-plugin:2.24.0-SNAPSHOT:prepare-spring-boot-auto-configuration (validate) on project camel-pulsar: Execution validate of goal org.apache.camel:camel-package-maven-plugin:2.24.0-SNAPSHOT:prepare-spring-boot-auto-configuration failed: String index out of range: -1 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.camel:camel-package-maven-plugin:2.24.0-SNAPSHOT:prepare-spring-boot-auto-configuration (validate) on project camel-pulsar: Execution validate of goal org.apache.camel:camel-package-maven-plugin:2.24.0-SNAPSHOT:prepare-spring-boot-auto-configuration failed: String index out of range: -1
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution validate of goal org.apache.camel:camel-package-maven-plugin:2.24.0-SNAPSHOT:prepare-spring-boot-auto-configuration failed: String index out of range: -1
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:148)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
    at java.lang.String.substring (String.java:1967)
    at org.apache.camel.maven.packaging.SpringBootAutoConfigurationMojo.executeComponent (SpringBootAutoConfigurationMojo.java:592)
    at org.apache.camel.maven.packaging.SpringBootAutoConfigurationMojo.execute (SpringBootAutoConfigurationMojo.java:196)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)

oscerd · 2019-04-23T11:18:12Z

@davsclaus can you take a look?

davsclaus · 2019-04-23T11:21:37Z

@oscerd I think I fixed something similar on master branch in that SB generator.

davsclaus · 2019-04-23T11:25:03Z

okay building 2.x now

davsclaus · 2019-04-23T11:45:31Z

Okay you need to be able to build the component without having docker running, so that should be fixed too.

davsclaus · 2019-04-23T11:51:13Z

Okay so the src/main/resources/META-INF/services/org/apache/camel/component/pulsar file has log4j settings, instead it should be

class=org.apache.camel.component.pulsar.PulsarComponent

davsclaus · 2019-04-23T11:51:30Z

@oscerd you can fix it as above ^^^

oscerd · 2019-04-23T12:00:25Z

Thank you

oscerd · 2019-04-23T12:09:20Z

I'll backport the Karaf feature and tests for Karaf and SB too

oscerd · 2019-04-23T13:12:41Z

Backported. Thanks. Please @rshermanTHG and @foamdino take a look at the documentation under src/main/docs and improve it if you have time.

squashed commit of cherrypick to camel-2.x branch

93b8b2b

oscerd reviewed Mar 29, 2019

View reviewed changes

Comment thread components/camel-pulsar/src/main/java/org.apache.camel.component.pulsar/PulsarComponent.java Outdated

oscerd requested changes Mar 29, 2019

View reviewed changes