feat(vertx): use vertx async to improve performance

[andrewazores]

Related to #453

This PR contains some spin-off work from the Vertx 3->4 upgrade. It primarily focuses on using vertx.executeBlocking() in select code paths to ensure that operations do not block the main thread and will be performed by a worker thread instead. I also went through and removed all uses of .parallelStream() since they were unlikely to be helpful. Under the covers this uses the Runtime.availableProcessors() count as the parallelism factor (number of threads), which for most Cryostat deployments is likely to just be 1, so the thread scheduling overhead is not worth it. Even if more CPU is available most of these uses were on collections/streams that are unlikely to have very many elements, still not enough to make the thread scheduling overhead worthwhile. There is one case in OpenShiftAuthManager where I converted a parallelStream() into an executeBlocking since that case was essentially multiplexing Kubernetes RBAC API requests. It's still unlikely to be very many elements, but the code was already structured nicely for concurrency and the work makes more sense to push off onto the worker thread pool. It may happen to execute in parallel for a quick performance win, or it may not in which case it just degrades back to where it was before the patch.

[jan-law]

It's been a while since I tested something on OpenShift, so I'm not sure if I set this up correctly.

When I go to Recordings for the cryostat container, the loading spinner appears for around 30s before the JMX authentication popup appears. Logs show an authentication error as well as a timeout exception:

Jun 06, 2022 8:55:55 PM io.cryostat.core.log.Logger info
INFO: connection attempt failed.

Jun 06, 2022 8:55:55 PM io.cryostat.core.log.Logger info
INFO: Connection for service:jmx:rmi:///jndi/rmi://10.128.2.13:9091/jmxrmi closed
Jun 06, 2022 8:55:55 PM io.cryostat.core.log.Logger error
SEVERE: Exception thrown
java.util.concurrent.ExecutionException: org.openjdk.jmc.rjmx.ConnectionException caused by java.lang.SecurityException: Authentication failed! Invalid username or password
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
	at io.cryostat.rules.PeriodicArchiver.performArchival(PeriodicArchiver.java:154)
	at io.cryostat.rules.PeriodicArchiver.run(PeriodicArchiver.java:132)
	at io.cryostat.rules.RuleProcessor.lambda$activate$8(RuleProcessor.java:219)
	at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:889)
	at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:860)
	at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:50)
	at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:274)
	at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:22)
	at io.vertx.core.impl.AbstractContext.emit(AbstractContext.java:53)
	at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:22)
	at io.vertx.core.impl.VertxImpl$InternalTimerHandler.run(VertxImpl.java:883)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:176)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.openjdk.jmc.rjmx.ConnectionException caused by java.lang.SecurityException: Authentication failed! Invalid username or password
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:441)
	at io.cryostat.core.net.JFRConnection.attemptConnect(JFRConnection.java:228)
	at io.cryostat.core.net.JFRConnection.connect(JFRConnection.java:185)
	at io.cryostat.core.net.JFRConnection.getService(JFRConnection.java:111)
	at io.cryostat.recordings.RecordingArchiveHelper.getDescriptorByName(RecordingArchiveHelper.java:507)
	at io.cryostat.recordings.RecordingArchiveHelper.lambda$saveRecording$0(RecordingArchiveHelper.java:146)
	at io.cryostat.net.TargetConnectionManager.executeConnectedTask(TargetConnectionManager.java:156)
	at io.cryostat.recordings.RecordingArchiveHelper.saveRecording(RecordingArchiveHelper.java:142)
	... 20 more

Jun 06, 2022 8:56:01 PM io.cryostat.core.log.Logger error
SEVERE: Exception thrown
java.util.concurrent.TimeoutException
	at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
	at io.cryostat.net.openshift.OpenShiftAuthManager.validateToken(OpenShiftAuthManager.java:272)
	at io.cryostat.net.openshift.OpenShiftAuthManager.validateHttpHeader(OpenShiftAuthManager.java:385)
	at io.cryostat.net.web.http.AbstractAuthenticatedRequestHandler.validateRequestAuthorization(AbstractAuthenticatedRequestHandler.java:120)
	at io.cryostat.net.web.http.AbstractAuthenticatedRequestHandler.handle(AbstractAuthenticatedRequestHandler.java:92)
	at io.cryostat.net.web.http.AbstractAuthenticatedRequestHandler.handle(AbstractAuthenticatedRequestHandler.java:69)
	at io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
	at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)

Here's what I ran to deploy cryostat on openshift:

cd ~/cryostat
git fetch aazores
git checkout vertx-async
mvn clean package
podman tag quay.io/cryostat/cryostat:latest quay.io/jalaw/cryostat:latest
podman push quay.io/jalaw/cryostat:latest
cd ~/cryostat-operator
oc login ... <cluster bot creds>
DEPLOY_NAMESPACE=$(oc project -q) oc apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.yaml
DEPLOY_NAMESPACE=$(oc project -q) make deploy CORE_IMG="quay.io/jalaw/cryostat:latest"
make create_cryostat_cr

[andrewazores]

Hmm, thanks for checking that. I must have broken something around the OpenShiftAuthManager in one of the later force-pushes. I'll check it out.

[andrewazores]

@jan-law I've decided to just drop the async implementation from that part of the OpenShiftAuthManager that was causing the check to time out and fail. In the end that was trying to do the Cryostat application-level to k8s resource mapping check in parallel, and then parallelize each of those access checks against the Kubernetes RBAC API server. In practice, there won't actually be very many of those access checks occurring per Cryostat API request, so it is unlikely to really benefit much from parallelism. In my CRC setup I did some profiling of Cryostat itself (using the cryostat.jfc template) and found that each k8s RBAC check performed here is only taking on average around ~20ms to complete, and API requests either have no such checks to perform or have perhaps 2-3 resources to check for each API request at most.

[hareetd]

Works well on OpenShift.

[jan-law]

Looks good!