New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Wrong version of jolokia #1725
Comments
What version of hawtio are you using? And are you sure its jolokia problem? |
|
That sounds strange since jolokia per design itself is completely agnostic about the MBean registered, i.e. about the content it serves. So I doubt that Jolokia has any influence about the list of MBeans returned (regardless of version). Could you post somehow (pastebin, gist or so) the output of 'http://host:port/hawtio/jolokia/list' of both, the 1.2.1 and 1.2.2 installations ? (BTW, is it the same server only with the agents exchanged ?) (BTW, 1.2.3 is out since yesterday) |
I've upgraded hawtio to 1.2.3 BTW (I wonder if that fixes the login to secure jolokia machines using the Google Chrome Extension? :). We'll be cutting a release soon I expect... |
Sorry, not yet. But it is on my Christmas Wishlist (which is the list where I put in things which I'm going tackle in the 'calm' time before christmas ;-) |
@rhuss OK thanks! I'll put it on my list for Santa :) |
POST .../hawtio-default-offline-1.4.29-orig/jolokia/?maxDepth=7&maxCollectionSize=500&ignoreErrors=true&canonicalNaming=false whereas POST .../hawtio-default-offline-1.4.29-orig/jolokia/?maxDepth=7&maxCollectionSize=500&ignoreErrors=true&canonicalNaming=false |
sorry : whereas POST .../hawtio-1.4.29-modified/jolokia/?maxDepth=7&maxCollectionSize=500&ignoreErrors=true&canonicalNaming=false |
Hmm, it hard to say what happens. You did only exchange jolokia.jar in the modified version ? Could you try with |
setting ignoreErrors=false didn't change anything in the resulted json |
Can we start from the top-level ( |
the size is never the same when comparing between versions. |
version 1.4.30 of hawtio which has just been released doesn't resolve the issue of reduced camel set and missing jmx entries although it uses jolokia v 1.2.3. |
Have you tried bumping up the maxDepth setting in hawtio's preferences? This and the maxCollectionSize setting have a direct correlation to what hawtio fetches when it grabs the JMX tree and other JMX responses. Also, how is your camel context configured, or more importantly can you supply an example that replicates the problem so it can be investigated by the team? Thanks! |
@hchehade could please try the folowing approach:
Sorry, since your data is confidential I have no other approach yet. As @gashcrumb said, ideally an isolated example would be perfect in order to reproduce the issue. |
you can reproduce the issue if you install hawtio on a bare weblogic server. |
Which version of Weblogic ? |
12.1 |
I wonder if this is a WebLogic issue only, eg running hawtio on Tomcat et all seems working fine. |
it is not an issue of hawtio. It is rather an issue of jolokia. If you install hawtio on tomcat and access a remote jolokia agent on weblogic, you will have the same issue. |
At the beginning of next week I can have a look at a Weblogic installation 12.1 and will check this out. |
So were you able to reproduce the issue?? |
Sorry not yet. Still have to install a weblogic 12.1 since I only have a WLS 11g (10.3.6.0) at hand. Unfortunately I don't have much time right now to install a Weblogic 12.1, so it would be really helpful if you could look in your output and examine what and where the difference is. As I said above, starting with maxDepth = 1 it shouldn't be to difficult to examine the output. Could you please try recipe i described previously ? |
I have recreated this with Jetty as well. Downgrading jolokia to 1.2.1 and voila I have routes. With jolokia 1.2.3 I get no routes if I increase the number of routes in Camel. We have a camelContext which contains about 1200 routes, and when I reduce the number to 100 with jolokia 1.2.3 I can see the servers routes. When I load more, nothing under camel at all. With 1.2.1 everything is happy. |
I observed the same (tomcat 7.0.52 with Hawtio 1.4.27), |
Ok, ok. I really need a reproducible test case. Can your share your code somehow so that I can reproduce it for me ? It's really hard to fish in muddy waters .... |
I will have a thorough look at the diff between 1.2.1 and 1.2.2 but of course having it in my debugger would be far easier .... 'hope we can nail it down till the year changes ;-) |
Recreating this is going to be fun. Here is sorta how we create our 1200 routes :) We have a bean that is fired by a timer that runs once. The bean itself creates a new class that is a route. That class is then added to the the Here is some example code that runs inside of the bean that a timer could fire off just once, after the camelContext starts:
You could look at creating timers that fire every few ms and just log messages. public class MyRoute extends RouteBuilder {
} Our code is firing off polling consumers, so a thread is added for each. I am not sure it timers will add enough load, but it is a start. @davsclaus probably is asking wth we are doing, but running this across multiple routes is going to eventually require almost sharding the load, which is not a bridge we want to cross now. |
In addition to this: we noticed that changing the maxDepth parameter within the Hawtio->preferences->Jolokia is not used in the actual xmlhttprequest to a remote jolokia. We noticed this by using firebug to inspect the params that are sent. Could this explain the problem, especially when the number of objects at a certain level in the jmxtree is very large? We are connecting from hawtio version 1.4.37 towards a remote jolokia (running in activemq). |
Same problem as @hchehade, most of the entries inside the org.apache.camel JMX domain are missing. We are running about 1000 routes inside Apache ServiceMix 5.4.0 with hawt.io version 1.4.46. When I downgrade hawt.io to version 1.4.4 it works as expected. One more information, when issuing a GET request to http://:8181/hawtio/jolokia/list I get different response sizes. Jolokia serialization issues? |
Was this problem related to jolokia 1.2.2+ upgraded to a newer json simple master 1.2.1 1.2.2 So upgrading from json simple 1.1 to 1.1.1 maybe likely caused this problem? If anyone able to consistently reproduce, then you are welcome to try with jolokia 1.2.1 and then force it to use json simple 1.1.1 so that would be the only change, and then see if that causes the problem. And if so we could get it reported to json simple, and maybe jolokia in the mean time could downgrade to 1.1 so it works |
Can we get it documented which version of jolokia to use with which version of hawtio :) I will open another boog |
actually it is working fine with 1.2.1 |
I just created a SNAPSHOT from 1.2.2 with json-simple-1.1.1 replaced by json-simple-1.1 (nothing changed otherwise). You can refer to it as The WAR agent can be directly downloaded from maven central. @chrislovecnm , @davidflam, @hchehade, @fliot: Since I really can't reproduce it myself (sorry, don't have the appropriate amount of Camel routes ;-) I need your help. Could you please try this agent (or the JVM variant from the same location, which also contains the json-1.1-simple.jar) with your setup ? This would be awesome ! Thanks a lot ! |
Since this is a Jolokia issue I opened an issue over there (rhuss/jolokia#187). Please let's continue there. |
Camel routes display (around 200 routes), perfectly works on my build (hawtio 1.4.45, jolokia 1.2.1). I just tested with jolokia 1.2.2-json-simple-1.1-SNAPSHOT, and unfortunatelly I must advise it doesn't work (as it doesn't work with jolokia 1.2.2). |
Thanks for the feedback, this is quite helpful. I will have closer look to the code diff then. |
Indeed the serialization stuff has been changed slightly between 1.2.1 and 1.2.2. However, I still can't repdoruce the issue. I introduced a large lists in the AttributeChecking MBean and tried it with really large lists, but they are only truncated if Since I'm not that fluent in Camel yet, could someone please create a small, self-contained WAR which creates a large number of routes during init (1000 or so) so that I could examine the situation locally ? 'would be very helpful and shouldn't be much work. thanks .... |
I created a bit camel project based on the Apache Camel Servlet Tomcat example. Its hosted in github at You can build from source and deploy the war to tomcat. I also noticed that in hawtio the number of routes listed are not all of them, but a bit strange with routes from the begging, and then a chunk of routes from the end etc. The app has one main route which is the the helloRoute. And then when you add new routes they are named route-1, route-2 ... |
And you can also list all the routes using, and then compare with jolokia |
Thanks a lot, @davsclaus ! Good news: I could reproduce the problem with 500 routes. Interestingly, the list of route MBeans is not truncated but there are gaps in the list (e.g Hope I can nail it down over the weekend so that we finally have a solution for that. |
BTW, its start with 167 added routes and is completely deterministic and reproducible (route-92 is missing then in 1.2.2 but not 1.2.1). Nice somehow. |
@rhuss and I though only OSGi could drive you mad with non deterministic behavior based on factors from the current lunar position and whether its a leap year |
Ok, got it finally. The point is, that between 1.2.1 and 1.2.2 a bug was fixed accidentally (well, doesn't happen very often ;-): Jolokia has some options which influences the JSON serialization. One of them is This truncation worked for any collections in 1.2.1 except for Since the As soon as the number of MBeans within a domain exceed This is of course not a good solution. There are several ways how to fix that:
Any ideas or suggestions on this ? |
For a quick workaround with jolokia 1.2.2 and 1.2.3 you can change the
|
Yay Yeah I think increasing the default to 10000 is better. IMHO as ppl likely dont know about this limit and then instead they wonder why some data is not there. With ActiveMQ or Camel they can be verbose in JMX, eg for Camel a route can result in N mbeans, 1 for the route, N for its processors, N for its endpoints. And for ActiveMQ you have a mbean per queue / topic. So hitting 1000 is actually common for production usage. So having a limit of 10000 is better, but still even on the lower side. A broker could potential have many queues. |
I think the LIST should not have any limit by default, eg its there to list all the mbeans. Its a more special operation you know can return a lot of data. And if you want to limit it, maybe the list should have (if not already) a nice filter api so you can filter only the mbean kinds you want. And yeah maybe also a different upper limit option you can use, and then some way to know in the response if the upper limit was hit. |
Yes, you are right, one should really exclude LIST operation from the However, restricting LIST in the serialization is not so easy, since the serialization mechanism for a return value is completely decoupled from the actual operation. What's about the following, super easy to implement solution: Removing the hard limit on the collection size (which is BTW the only hard limit currently set by default):
If there are no objections, I will choose this route, since this is the easiest, backwards compatible, solution for this pain point. |
+1 |
@rhuss did you have time to work on jolokia recently? |
I'm just back today and plan to do a release this week with the proposed fix. |
Truncation of collections can be tuned with the 'maxCollectionSize' config parameter. The default was infinite for the JVM agent and Mule agent but 1000 for the WAR agent. This turned out to be too less (see hawtio/hawtio#1725 (comment)). Now the limitation has been removed, which shouldn't be a big problem for most. You can always add your own limitation by setting the parameter in web.xml. Fixes #187.
Fixed now |
Jolokia V122 is causing multitude of errors. Many entries in JMX and Camel are missing.
Setting the dependency to jolokia V121 would resolve these problems.
The text was updated successfully, but these errors were encountered: