Statistics generation and analysis (Hibernate, Infinispan)

[dm-vestmark]

We would like some guidance on two issues pertaining to hibernate statistics in a clustered environment.

• How to generate statistics for a node in the cluster and not cluster-wide
• How to use these statistics to understand trends in cache usage

We are looking into an apparent sporadic cache performance degradation over a period of several days in a production environment. We are able to generate statistics at the cluster level using the following additions to our config files:

standalone-ha.xml :
<cache-container name="VmManagedBatchedMda" marshaller="JBOSS" modules="org.infinispan.hibernate-cache" statistics-enabled="true">
<invalidation-cache name="entity" remote-timeout="120000" statistics-enabled="true">
<heap-memory size="100"/> <expiration max-idle="300000" lifespan="300000"/>
</invalidation-cache>

and setting hibernate.cache.infinispan.statistics to true.
"[hibernate.cache.infinispan.statistics]": true

How to generate statistics for a node in the cluster and not cluster-wide
These additions have produced only statistics at the cluster level and not the local node level. The cluster-wide statistics do not allow us to analyze why a specific node(s) experience performance degradation based on cache usage. Thus, we need to generate and analyze statistics at the node level.
When we do this in a non-clustered environment (with standalone.xml and a ), we get a properly working Statistics mbean for each Hibernate entity. We get a properly working ClusterCacheStats bean in a cluster, but the values are the same on each cluster node. The Statistics mbean is still there, but we get an org.infinispan.commons.CacheException: java.lang.NullPointerException trying to access every single JMX attribute from the Statistics bean. Are the local Statistics supposed to work for an invalidation cache and report the local statistics (hits, misses, evictions) for each cluster node? And how does the ClusterCacheStats aggregate the numbers – is it adding up the hits, misses, evictions, and invalidations across all cluster nodes and reporting the sum?
How to use these statistics to understand trends in cache usage
The statistics also need to be captured for usage analysis over time. Is there some easy way to log the statistics over time for multiple entities? Or is the only way to configure our JMX tools to capture and track multiple attributes for multiple mbeans corresponding to the various entities we are interested in?
The Environment: Wildfly version: 26.0.1.Final
Inifinispan version: 12.1.7.Final
Hibernate version: 5.4.18.Final

[dm-vestmark]

Hi Community, any assistance would be much appreciated

[vmgbrown]

FWIW, this is a more detailed version of a similar question I posted several months ago:
#10933

[tristantarrant]

@pferraro can you help investigating this ?

[vmgbrown]

jmx-log.txt

[dm-vestmark]

Log provided in the jmx-log.txt

[tristantarrant]

Are you deploying the hawt.io war file ? If so, what version is it ?

[tristantarrant]

Add the following line:

<module name="org.eclipse.microprofile.metrics.api"/>

to modules/system/layers/base/org/infinispan/main/module.xml

and use Hawt.io 3.0

[tristantarrant]

@dm-vestmark did you try this?

[tristantarrant]

Would it be possible for you to try with a more recent WildFly ?

[tristantarrant]

The problem is because statistics in Infinispan 12 rely on Microprofile Metrics but WildFly is not exposing those classes to the org.infinispan:main module thus causing a ClassNotFoundException

[vmgbrown]

@tristantarrant Is that a limitation of WildFly 26? Are those classes exposed by WildFly 27 or later?

[tristantarrant]

It's a bug with WildFly 26. You can patch it as I described in a comment above. The bug is also present in all subsequent releases of WildFly. However, I doubt you could upgrade anyway, since WildFly 27+ require JakartaEE 9 deployments and don't work with JavaEE deployments.

[vmgbrown]

I see. So the suggestion to modify module.xml is the patch. Is Hawtio 3.0 also required? We have not been able to get it to work with WildFly 26.

[tristantarrant]

I simply deployed the hawtio 3.0 war to WF 26 with no issues. If older hawt.io works for you, that's fine.