Updates to Camel route information causes a large number of requests which clobber the UI #1903

Closed
rmoquin opened this Issue Feb 12, 2015 · 14 comments

Projects

None yet

4 participants

@rmoquin
rmoquin commented Feb 12, 2015

I'm opening a new one for this, even though I see things that could be potentially related. I've been seeing a problem with multiple versions of hawt.io where anytime a count changes on a Camel route, the dynatree containing the route list immediately becomes empty and so does the rest of the Camel information. I decided to finally see if I could get any information on why the dynatree becomes blank after something like a count changes. I noticed two things. If I tell Chrome to show paint rectangles, when I exercise a camel route (causing the data in hawt.io to change in the display), I'll see the Camel Contexts dynatree gets hammered by DOM updates (it must be) caused by a large amount of ajax requests from hawt.io in response to the Camel metric change. If I look in the network timeline, you can see the updates going every 5 seconds and when the Camel data changes, you see a lot of requests going out at the same time.

Attached is a screenshot, notice it shows the page was last refresh about 20 seconds previously but shows that in that 20 second period, 72 total ajax requests were made. Notice that there are a large number of jolokia requests occurring at the same time. I think all the updates from the requests clobber the Camel Contexts dynatree. It make hawtio virtually unusable.
hawtio_lots_of_requests

@davsclaus
Member

What version of ActiveMQ, Camel and hawtio do you use?

There is a bug in latest AMQ release that causes it to update mbeans, and thus causes hawtio to redraw the tree. There is new releases on the way that will fix this.

@davsclaus davsclaus added the question label Feb 13, 2015
@gashcrumb
Member

Also can you expand one of those requests and show either the request headers or request body? Would helps us track down where in the code those requests are coming from. And yeah, hawtio version would be good too, there's a bit of caching for most views that would cut down on ajax requests but not everywhere...

@rmoquin
rmoquin commented Feb 16, 2015

After I sent this I realized I didn't give you much detail to go on. I'm
currently using ActiveMQ 5.11.0 and I've tried hawtio 1.4.47, as well as 46
and 45... the problem varies a bit between releases but I didn't remember
having this problem until recently.

Could definitely be AMQ. I can send more detail if needed. It happens in
the latest karaf and only really seems to affect the camel component.

Ryan
On Feb 13, 2015 6:58 AM, "Claus Ibsen" notifications@github.com wrote:

What version of ActiveMQ, Camel and hawtio do you use?

There is a bug in latest AMQ release that causes it to update mbeans, and
thus causes hawtio to redraw the tree. There is new releases on the way
that will fix this.


Reply to this email directly or view it on GitHub
#1903 (comment).

@rmoquin
rmoquin commented Feb 16, 2015

Oh and camel 2.14.1....
On Feb 13, 2015 6:58 AM, "Claus Ibsen" notifications@github.com wrote:

What version of ActiveMQ, Camel and hawtio do you use?

There is a bug in latest AMQ release that causes it to update mbeans, and
thus causes hawtio to redraw the tree. There is new releases on the way
that will fix this.


Reply to this email directly or view it on GitHub
#1903 (comment).

@davsclaus
Member

Yes its a bug in ActiveMQ 5.11.0. A 5.11.1 is on the way, you can try it before its public GA
http://activemq.2283324.n4.nabble.com/VOTE-Release-Apache-ActiveMQ-5-11-1-td4691535.html

@rmoquin
rmoquin commented Feb 17, 2015

Awesome, thanks Claus!

Ryan
On Feb 17, 2015 1:51 AM, "Claus Ibsen" notifications@github.com wrote:

Yes its a bug in ActiveMQ 5.11.0. A 5.11.1 is on the way, you can try it
before its public GA

http://activemq.2283324.n4.nabble.com/VOTE-Release-Apache-ActiveMQ-5-11-1-td4691535.html


Reply to this email directly or view it on GitHub
#1903 (comment).

@rmoquin
rmoquin commented Feb 17, 2015

Hi Claus,

Unfortunately I still hit the same problem with ActiveMQ 5.12-SNAPSHOT on
Karaf 3.0.3, Camel 2.15-SNAPSHOT and Hawtio 1.4.47. 1.4.46 seems exhibits
the same problem, but it's much less severe. I'll get you some logs
showing where the calls are coming from.

Ryan

On Tue, Feb 17, 2015 at 11:05 AM, Ryan Moquin fragility2.0@gmail.com
wrote:

Awesome, thanks Claus!

Ryan
On Feb 17, 2015 1:51 AM, "Claus Ibsen" notifications@github.com wrote:

Yes its a bug in ActiveMQ 5.11.0. A 5.11.1 is on the way, you can try it
before its public GA

http://activemq.2283324.n4.nabble.com/VOTE-Release-Apache-ActiveMQ-5-11-1-td4691535.html


Reply to this email directly or view it on GitHub
#1903 (comment).

@rmoquin
rmoquin commented Feb 26, 2015

It seems like ActiveMQ 5.11.1 is better than 5.11 sometimes, such as when updates occur, sometimes the midde panel of camel details won't blank out and only the Camel tree view blanks out. Here is a screenshot after a camel count updated. I will see the count increment and then immediately the screenshot is usually the result. Also, an inflight count change doesn't cause this problem, only when something completes a route, or if a route is reloaded. If the route is reloaded, it blanks out never comes back unless you swap tabs or refresh the page.

screen shot 2015-02-26 at 7 41 50 am

This won't be near as helpful as if I was using the uncompressed script, which I think I can use if I install the dev feature in Karaf? These stacks (even though they paste messy) look like they could still be useful. These are a couple of the pieces of code that all triggered as soon as a message exchange completed going through a route. There are 3 distinct traces and 22 total calls when the camel route completes. It appears the DynatreeNode.activate is the culprit.

This is the first of the 22 and is called once and looks to be the normal refresh every 5 seconds:


send @ app.js?f9f85ff34e487c35:4
$.extend.ajax @ app.js?f9f85ff34e487c35:3
request @ app.js?f9f85ff34e487c35:16
(anonymous function) @ app.js?f9f85ff34e487c35:16


This is triggered second and only once.


send @ app.js?f9f85ff34e487c35:4
$.extend.ajax @ app.js?f9f85ff34e487c35:3
request @ app.js?f9f85ff34e487c35:16
h @ app.js?f9f85ff34e487c35:17
c.maybeReloadTree @ app.js?f9f85ff34e487c35:54
(anonymous function) @ app.js?f9f85ff34e487c35:13
lcb @ app.js?f9f85ff34e487c35:16
r.success @ app.js?f9f85ff34e487c35:16
(anonymous function) @ app.js?f9f85ff34e487c35:16
g.success @ app.js?f9f85ff34e487c35:16
l @ app.js?f9f85ff34e487c35:1
m.fireWith @ app.js?f9f85ff34e487c35:1
d @ app.js?f9f85ff34e487c35:3
d @ app.js?f9f85ff34e487c35:4


This is triggered 3rd but the call is repeated 20 times.


send @ app.js?f9f85ff34e487c35:4
$.extend.ajax @ app.js?f9f85ff34e487c35:3
request @ app.js?f9f85ff34e487c35:16
(anonymous function) @ app.js?f9f85ff34e487c35:77
i @ app.js?f9f85ff34e487c35:14
(anonymous function) @ app.js?f9f85ff34e487c35:14
f.$eval @ app.js?f9f85ff34e487c35:14
f.$digest @ app.js?f9f85ff34e487c35:14
f.$apply @ app.js?f9f85ff34e487c35:14
C @ app.js?f9f85ff34e487c35:54
e.length.h @ app.js?f9f85ff34e487c35:54
DynaTreeNode._activate @ app.js?f9f85ff34e487c35:25
DynaTreeNode.activate @ app.js?f9f85ff34e487c35:25
h @ app.js?f9f85ff34e487c35:54
h @ app.js?f9f85ff34e487c35:62
(anonymous function) @ app.js?f9f85ff34e487c35:62

@jcable
jcable commented Mar 30, 2015

I'm having this problem too. Is there any update? I'll check out tomorrow with some AMQ-free routes and confirm it's AMQ causing the problem for me too.

@davsclaus
Member

Yep got this today with 5.12.1 on Karaf 4.0.3 with Camel 2.16 and 2.17-SNAPSHOT.

@davsclaus
Member

We had this issue before in AMQ where it will add/remove mbeans all the time.

@davsclaus
Member

Ah you likely need to use a pooled connection factory with AMQ so the connection is re-used which causes the mbeans to not be created all the time
http://camel.apache.org/activemq

@davsclaus
Member

Here is how I configured a connection pool that works

  <!-- the ConnectionFactory to connect to the JMS broker -->
  <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
    <property name="brokerURL" value="tcp://localhost:61616"/>
    <property name="userName" value="karaf"/>
    <property name="password" value="karaf"/>
  </bean>

  <!-- pool the connection factory so it runs faster -->
  <bean id="pooledConnectionFactory"
        class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start" destroy-method="stop">
    <property name="maxConnections" value="8"/>
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
  </bean>

@davsclaus davsclaus referenced this issue in FuseByExample/esb-transactions Nov 21, 2015
Open

Add connection pool to the AMQ component #17

@davsclaus davsclaus added a commit that referenced this issue Nov 21, 2015
@davsclaus davsclaus Add to FAQ about #1903 0ad486a
@davsclaus davsclaus added a commit that closed this issue Nov 22, 2015
@davsclaus davsclaus Fixed #1903 by filtering out AMQ add/remove mbeans of client connecto…
…rs that would cause the tree to be updated. Added JVM system property that can be used to turn on old behavior.
7c205fa
@davsclaus davsclaus closed this in 7c205fa Nov 22, 2015
@davsclaus
Member

Yay we now filter out those events that would cause this update. The next release should be good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment