Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

outputWriterOpenTSDB not working on latest release #443

Open
vfork001 opened this issue Apr 7, 2016 · 7 comments · Fixed by #444
Open

outputWriterOpenTSDB not working on latest release #443

vfork001 opened this issue Apr 7, 2016 · 7 comments · Fixed by #444
Labels

Comments

@vfork001
Copy link

vfork001 commented Apr 7, 2016

same config works in ver 253, but not 254

[07 Apr 2016 15:44:29] [ServerScheduler_Worker-1] 163428 ERROR (com.googlecode.jmxtrans.jmx.JmxUtils:62) - Could not submit query ProcessQueryThread(log=org.slf4j.impl.Log4jLoggerAdapter(com.googlecode.jmxtrans.jmx.ProcessQueryThread), server=Server(pid=null, host=triumph08, port=8999, url=service:jmx:rmi:///jndi/rmi://triumph08:8999/jmxrmi, cronExpression=null, numQueryThreads=2), query=Query(objectName=java.lang:type=Memory, keys=[], attr=[HeapMemoryUsage, NonHeapMemoryUsage], typeNames=[], resultAlias=heap, useObjDomainAsKey=false, allowDottedKeys=false, useAllTypeNames=false, outputWriterInstances=[BaseOutputWriter(typeNames=[], debugEnabled=false, settings={}, valueTransformer=com.googlecode.jmxtrans.model.results.IdentityValueTransformer@7baec269)])). You could try to size the 'queryProcessorExecutor' to a larger size.
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@476b0f0c rejected from java.util.concurrent.ThreadPoolExecutor@394ac3b8[Running, pool size = 10, active threads = 10, queued tasks = 1000, completed tasks = 143]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)

@gehel
Copy link
Member

gehel commented Apr 8, 2016

Any chance you could send a few thread dumps to help diagnose the issue? I know we have some bugs in the way connection pools are handled, so this is probably related... just want to make sure...

@gehel gehel added the bug label Apr 8, 2016
@vfork001
Copy link
Author

vfork001 commented Apr 8, 2016

jmxtrans.txt

@vfork001
Copy link
Author

vfork001 commented Apr 8, 2016

this is using OutWriterOpenTSDB
also, same config when i switched to OutputWriterOpenTSDBFactory, it can't parse config file.

@gehel
Copy link
Member

gehel commented Apr 9, 2016

Could you also send the config file(s)? This seems to be a different issue...

gehel pushed a commit that referenced this issue Apr 10, 2016
Instead of querying averything at the 60 second mark (or whatever the
runPeriod is), spread them a bit.

Also increased the queue size of executor services to something more
reasonnable.

This should fix #443.
@gehel
Copy link
Member

gehel commented Apr 10, 2016

This should be fixed by #444 which will be merged shortly and available as SNAPSHOT. Feel free to re-open if that not the case.

@vfork001
Copy link
Author

still not working, same error
[11 Apr 2016 09:02:07] [jmxtrans-result-5] 72612 WARN (com.googlecode.jmxtrans.jmx.ResultProcessor$1:60) - Could not write results [Result(attributeName=AvailableProcessors, className=com.
....

@gehel gehel reopened this Apr 11, 2016
@dwo
Copy link
Contributor

dwo commented Sep 9, 2016

FWIW: we had the same issue on 254, so upgraded to 259. This solved the issue, but only when we used the deprecated OpenTSDBWriter instead of OpenTSDBWriterFactory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants