CAMEL-12418: High CPU load on events watching #2283

nryanov · 2018-04-08T22:44:30Z

I believe that scheduledExecutorService with schedule method will be a little better then Thread.sleep(mills) (maybe i'm wrong).

Also, i didn't add default value as proposed in description of this problem because as i found in ConsulClientConfiguration the default value of blockSeconds is 10:

@UriParams
public class ConsulClientConfiguration implements Cloneable {
    . . . 
    @UriParam(label = "consumer,watch", defaultValue = "10")
    private Integer blockSeconds = 10;
    . . . 
}

Check style

Result of mvn clean install -Psourcecheck:

[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Camel :: Consul 2.22.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ camel-consul ---
[INFO] Deleting D:\Program Files\WORK\WORK_Java\camel\components\camel-consul\target
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-maven) @ camel-consul ---
[INFO] 
[INFO] --- maven-bundle-plugin:3.5.0:cleanVersions (versions) @ camel-consul ---
[INFO] 
[INFO] --- maven-checkstyle-plugin:3.0.0:checkstyle (default-cli) @ camel-consul ---
[INFO] Starting audit...
Audit done.
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) @ camel-consul ---
[INFO] 
[INFO] --- camel-package-maven-plugin:2.22.0-SNAPSHOT:prepare-components (prepare) @ camel-consul ---
[INFO] Generated D:\Program Files\WORK\WORK_Java\camel\components\camel-consul\target\generated\camel\components\META-INF\services\org\apache\camel\component.properties containing 1 Camel component: consul
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ camel-consul ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO] Copying 3 resources
[INFO] Copying 1 resource
[INFO] skip non existing resourceDirectory D:\Program Files\WORK\WORK_Java\camel\components\camel-consul\target\generated\camel\dataformats
[INFO] skip non existing resourceDirectory D:\Program Files\WORK\WORK_Java\camel\components\camel-consul\target\generated\camel\languages
[INFO] 
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ camel-consul ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 37 source files to D:\Program Files\WORK\WORK_Java\camel\components\camel-consul\target\classes
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:testResources (default-testResources) @ camel-consul ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ camel-consul ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 23 source files to D:\Program Files\WORK\WORK_Java\camel\components\camel-consul\target\test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:2.20.1:test (default-test) @ camel-consul ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-bundle-plugin:3.5.0:manifest (bundle-manifest) @ camel-consul ---
[WARNING] Manifest org.apache.camel:camel-consul:jar:2.22.0-SNAPSHOT : Unused Import-Package instructions: [org.springframework.ws.*, org.springframework.xml.*, org.springframework.*, org.apache.cxf.*, org.apache.qpid.*, org.apache.abdera.*, org.apache.commons.httpclient.*, org.apache.velocity.*, org.apache.xmlbeans.*, org.eclipse.jetty.*, com.thoughtworks.xstream.*, org.antlr.stringtemplate.*, org.ccil.cowan.tagsoup.*, org.mortbay.cometd.*, net.sf.flatpack.*, net.sf.saxon.*, freemarker.*, javax.persistence.*, org.apache.lucene.*, org.apache.solr.*] 
[INFO] 
[INFO] --- camel-package-maven-plugin:2.22.0-SNAPSHOT:validate-components (validate) @ camel-consul ---
[INFO] Validation complete
[INFO] 
[INFO] --- camel-package-maven-plugin:2.22.0-SNAPSHOT:prepare-spring-boot-starter (validate) @ camel-consul ---
[INFO] Spring-Boot-Starter: starter dir for the component is: D:\Program Files\WORK\WORK_Java\camel\platforms\spring-boot\components-starter\camel-consul-starter
[INFO] Reusing the existing pom.xml for the starter
[INFO] 
[INFO] --- camel-package-maven-plugin:2.22.0-SNAPSHOT:prepare-spring-boot-auto-configuration (validate) @ camel-consul ---
[INFO] Updated existing file: D:\Program Files\WORK\WORK_Java\camel\platforms\spring-boot\components-starter\camel-consul-starter\src\main\java\org\apache\camel\component\consul\springboot\ConsulComponentConfiguration.java
[INFO] Updated existing file: D:\Program Files\WORK\WORK_Java\camel\platforms\spring-boot\components-starter\camel-consul-starter\src\main\java\org\apache\camel\component\consul\springboot\ConsulComponentAutoConfiguration.java
[INFO] 
[INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ camel-consul ---
[INFO] Building jar: D:\Program Files\WORK\WORK_Java\camel\components\camel-consul\target\camel-consul-2.22.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-site-plugin:3.5.1:attach-descriptor (attach-descriptor) @ camel-consul ---
[INFO] 
[INFO] --- camel-package-maven-plugin:2.22.0-SNAPSHOT:update-readme (readme) @ camel-consul ---
[INFO] 
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ camel-consul ---
[INFO] Installing D:\Program Files\WORK\WORK_Java\camel\components\camel-consul\target\camel-consul-2.22.0-SNAPSHOT.jar to C:\Users\GrIfOn\.m2\repository\org\apache\camel\camel-consul\2.22.0-SNAPSHOT\camel-consul-2.22.0-SNAPSHOT.jar
[INFO] Installing D:\Program Files\WORK\WORK_Java\camel\components\camel-consul\pom.xml to C:\Users\GrIfOn\.m2\repository\org\apache\camel\camel-consul\2.22.0-SNAPSHOT\camel-consul-2.22.0-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 16.378 s
[INFO] Finished at: 2018-04-09T01:29:11+03:00
[INFO] Final Memory: 53M/553M
[INFO] ------------------------------------------------------------------------

davsclaus

We should have code that shutdown that executor service - also the executor service should be created via Camel's API for that, see ExecutorServiceManager you can get via CamelContext

nryanov · 2018-04-10T17:30:21Z

Ok, i'll fix it

davsclaus · 2018-04-16T10:42:42Z

@Gr1f0n6x let us know how it goes for you

nryanov · 2018-04-17T01:20:18Z

@davsclaus, I tried to use ConsulEndpoint for getting CamelContext as it implemented CamelContextAware.
Also, i move field scheduledExecutorService to the ConsulEventConsumer class from the inner class EventWatcher because otherwise it will created more than one thread for this background work.
I'm not sure about doStop but i overrided it and shutdown executor there.

Sorry for delay. I had some problems with building the project

dmvolod · 2018-04-17T08:38:39Z

...mel-consul/src/main/java/org/apache/camel/component/consul/endpoint/ConsulEventConsumer.java

    }

    @Override
    protected Runnable createWatcher(EventClient client) throws Exception {
        return new EventWatcher(client);
    }

+    @Override
+    protected void doStop() throws Exception {
+        scheduledExecutorService.shutdown();


@Gr1f0n6x , please use ExecutorServiceManager.shutdownGraceful there

@dmvolod, did it. Thank you for hint, didn't notice this method before

davsclaus · 2018-04-18T07:34:11Z

...mel-consul/src/main/java/org/apache/camel/component/consul/endpoint/ConsulEventConsumer.java


    public ConsulEventConsumer(ConsulEndpoint endpoint, ConsulConfiguration configuration, Processor processor) {
        super(endpoint, configuration, processor, Consul::eventClient);
+        this.executorServiceManager = endpoint.getCamelContext().getExecutorServiceManager();
+        this.scheduledExecutorService = this.executorServiceManager.newSingleThreadScheduledExecutor(this, "ConsulEventConsumer");


We create these in the doStart method, so we have start/stop logic together, and the constructors are light-weight

* Add delay between listEvents requests * Use ExecutorServiceManager got via CamelContext * Add explicit field executorServiceManager * Use ExecutorServiceManager.shutdownGraceful

Add delay between listEvents requests

5ea5cbd

davsclaus requested changes Apr 9, 2018

View reviewed changes

Use ExecutorServiceManager got via CamelContext

45f400f

dmvolod reviewed Apr 17, 2018

View reviewed changes

nryanov added 2 commits April 17, 2018 16:20

Add explicit field executorServiceManager

2c6a5bc

Use ExecutorServiceManager.shutdownGraceful

9751687

davsclaus reviewed Apr 18, 2018

View reviewed changes

davsclaus merged commit 501045c into apache:master Apr 18, 2018

davsclaus added a commit that referenced this pull request Apr 18, 2018

CAMEL-12418: High CPU load on events watching (#2283)

cd0d07d

davsclaus added a commit that referenced this pull request Apr 18, 2018

CAMEL-12418: High CPU load on events watching (#2283)

0db10ba

nryanov deleted the CAMEL-12418 branch April 18, 2018 12:21

tdiesler pushed a commit to tdiesler/camel that referenced this pull request Feb 20, 2019

CAMEL-12418: High CPU load on events watching (apache#2283)

864af85

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CAMEL-12418: High CPU load on events watching #2283

CAMEL-12418: High CPU load on events watching #2283

nryanov commented Apr 8, 2018 •

edited

Loading

davsclaus left a comment

nryanov commented Apr 10, 2018

davsclaus commented Apr 16, 2018

nryanov commented Apr 17, 2018

dmvolod Apr 17, 2018

nryanov Apr 17, 2018

davsclaus Apr 18, 2018

CAMEL-12418: High CPU load on events watching #2283

CAMEL-12418: High CPU load on events watching #2283

Conversation

nryanov commented Apr 8, 2018 • edited Loading

Check style

davsclaus left a comment

Choose a reason for hiding this comment

nryanov commented Apr 10, 2018

davsclaus commented Apr 16, 2018

nryanov commented Apr 17, 2018

dmvolod Apr 17, 2018

Choose a reason for hiding this comment

nryanov Apr 17, 2018

Choose a reason for hiding this comment

davsclaus Apr 18, 2018

Choose a reason for hiding this comment

nryanov commented Apr 8, 2018 •

edited

Loading