Defer initialization of JGroups after logging is set up by Quarkus

[ahus1]

Closes #29129

[vmuzikar]

@ahus1 Thank you for the PR! The change makes sense to me but let's wait for @mabartos as he has more context about the original change.

[mabartos]

@ahus1 Thanks for the investigation around it, good job!

This is unfortunate that these 'revert' changes are required, as they removed the no-work CPU gap as shown here (don't focus on the red circle, but rather on the CPU utilization during the time ~2.3s-4.2s):

It was expected that the CPU usage would be higher because these tasks for different threads leverage more parallel executions than in the figure above.

However, as you mentioned, the issue is more critical with the trace logs, assumedly written into the DelayedHandler.

I'm just wondering if we have some possibilities to avoid providing changes in this PR.
What about setting the log level for the logger in the org.jgroups.stack.Protocol class before its initialization and avoiding using the trace level?

At the time of executing the build step, we already have initialized the whole configuration, so if the trace level for JGroups is required, we can allow it.

To summarize it, we could:

1. Check if the trace level for JGroups is required from the configuration, otherwise, change the level to info or whatever
2. Set the level for the org.jgroups.stack.Protocol log via its config option?
3. Start the new cache thread
4. Do not move the build step into the RUNTIME_INIT phase :)

@ahus1 I haven't tried it, but it might be feasible, right?

[ahus1]

@mabartos - the pause you see, is that maybe the pause where the discovery of ISPN takes place? It only applies when the first node is started.

My usual comment about optimizations applies here: If it makes things more fragile and complex, it is a trade-off with maintainability, and I'd rather not do it. Even if we fix it for the logging of the protocol, we might miss other locations, as the JGroups library is also "optimizing".

@pruivo - is the a way using the regular Java APIs to get hold of the instance of org.jgroups.protocols.TP to set the log level? If that is possible, one would also need to verify what kind of log framework JGroups detects here, as the Slf4j implementation doesn't allow for setting the log level and will throw an exception :-(

[pruivo]

@ahus1 after the CacheManager is instantiated:

var t = GlobalComponentRegistry.componentOf(cacheManager, Transport.class);
((JGroupsTransport)t).getChannel().getProtocolStack().getTransport().isTrace(false);

[mabartos]

@mabartos - the pause you see, is that maybe the pause where the discovery of ISPN takes place? It only applies when the first node is started.

Yep, AFAIK, during that phase, the cluster is analyzed and checks the existence of the coordinator. If there is any, the node will become the coordinator after exceeding the time. So, it should apply only to the first node, as you've mentioned.

Perhaps it'd be really better to stick with these changes you've provided, as the CPU gap for the first node is not a big deal. It'll probably be better in terms of maintainability, as you've mentioned.

[mabartos]

Based on my previous comment, I'm ok with these changes.