Glowroot central collector stops ingesting due to non-existent backlog

[rweintraub-cycleon]

hello,

we've recently upgraded our central collector to 0.14.1 from 0.14.0-beta.3
on our tst & pre environment this worked without any issues however in our production environment the central collector stops ingesting any data from the agents after a short while (arround 5 minutes) and then never come back. Our cluster was able to previously handle the load of our agents quite well while it being under a constant high load.

I'm not sure what our next steps should be in this case.
do we expect the 0.14.1 release to have higher load then 0.14.0-beta.3?
also did the backlog behavior change?
previously i remember the central collector only ignoring incoming data if the cassandra cluster was constantly at 100% for a long time which currently is not the case for our cluster.

kind regards,
Roman

[Nowheresly]

Hello
thanks for the report.
Mostly, the way to interact with cassandra has been updated with the new datastax driver since version 0.14.
Infinispan has been updated too.
Do you think it would be possible to post a thread dump of glowroot central when it seems to be stuck?

[rweintraub-cycleon]

i can try to do that. since this is currently on our production system this is a bit disruptive. (i've downgraded for now)
is there anything else that you'd need or want data wise to do an analysis?
ideally i can pull everything in once go so that we only have one more additional downtime

[Nowheresly]

A new beta has been released.

https://github.com/glowroot/glowroot/releases/tag/v0.14.2-beta.3

Please let us know if it helps.