Remote development mode: out of sync application

[vsevel]

Describe the bug

When using remote development mode, I sometimes get in a situation where I make the change, call the endpoint, and get the old behavior. a few seconds later a call to the same endpoint will get me the new behavior.
I assume the first call happens before the remote dev client had time to inform the remote side that a patch was coming.
In other cases, calling the endpoint blocks for some time, then renders the new behavior, which is expected.
The only complaint is that getting the result from the endpoint takes in my situation 20 seconds. I checked that there was plenty of resources available on the worker node, and that the cpu limit was high enough. The application is taking less than 50 millicores. so in my situation cpu does not seem to be the limiting factor.
It might very well depend on the application and the environment, but this type of delay will certainly push away developers and limit the usefulness of this feature as a result.

Expected behavior

Ideally, after the change is made, calling the endpoint should block until we are ready to get the new behavior.
Restart should ideally be fast in dev mode.

Actual behavior

Calling the endpoint after the change sometimes gives me the old behavior. Then after some time I will get the new one.
It takes a long time for the application to restart.

How to Reproduce?

I am using a test application with the following extensions:

2024-05-07 18:27:43,314 INFO [io.quarkus] (Quarkus Main Thread) {} Installed features: [cache, cdi, config-yaml, confluent-registry-avro, hibernate-validator, ironjacamar, kafka-client, messaging, messaging-kafka, micrometer, narayana-jta, opentelemetry, rest, rest-client, rest-client-jackson, rest-jackson, scheduler, security, smallrye-context-propagation, smallrye-fault-tolerance, smallrye-health, smallrye-openapi, swagger-ui, vault, vertx]

The application is not doing much, except sending a kafka message every 5 secs, and receiving the message it sent.

Output of uname -a or ver

No response

Output of java -version

No response

Quarkus version or git rev

89b732b (future 3.11)

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response