Java task worker does not reconnect to server after unexpected server restart

[Snarr]

Context

When a Java task worker is connected to a server and the server unexpectedly restarts, the worker recognizes that the gRPC connection closed but fails to reconnect to the server. This makes the worker unusable until the worker itself restarts. In this case I was able to produce the error when running the example-basic workflow example that creates a wfSpec with the same name.

14:02:26 ERROR [LH] PollThread - Unexpected error from server
io.grpc.StatusRuntimeException: CANCELLED: RST_STREAM closed stream. HTTP/2 error code: CANCEL
        at io.grpc.Status.asRuntimeException(Status.java:539) ~[grpc-api-1.56.1.jar:1.56.1]
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:491) [grpc-stub-1.56.1.jar:1.56.1]
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567) [grpc-core-1.56.1.jar:1.56.1]
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71) [grpc-core-1.56.1.jar:1.56.1]
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735) [grpc-core-1.56.1.jar:1.56.1]
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716) [grpc-core-1.56.1.jar:1.56.1]
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.56.1.jar:1.56.1]
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) [grpc-core-1.56.1.jar:1.56.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:842) [?:?]
14:02:26 ERROR [LH] PollThread - Unexpected error from server
io.grpc.StatusRuntimeException: CANCELLED: RST_STREAM closed stream. HTTP/2 error code: CANCEL
        at io.grpc.Status.asRuntimeException(Status.java:539) ~[grpc-api-1.56.1.jar:1.56.1]
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:491) [grpc-stub-1.56.1.jar:1.56.1]
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567) [grpc-core-1.56.1.jar:1.56.1]
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71) [grpc-core-1.56.1.jar:1.56.1]
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735) [grpc-core-1.56.1.jar:1.56.1]
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716) [grpc-core-1.56.1.jar:1.56.1]
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.56.1.jar:1.56.1]
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) [grpc-core-1.56.1.jar:1.56.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:842) [?:?]
<===========--> 87% EXECUTING [16m 30s]
> :example-basic:run

Steps To Reproduce

1. In a terminal, first launch a LittleHorse server instance

./local-dev/do-server.sh

2. In a separate terminal, launch a LittleHorse worker instance for example-basic

./gradlew example-basic:run

3. Kill the LittleHorse server terminal

4. View the Worker instance terminal and see the error

5. Relaunch the LittleHorse server

6. In a separate terminal, use lhctl to launch a wfRun of the example-basic wfSpec

lhctl run example-basic

7. Notice no update or response from the Worker after the LittleHorse server relaunched

Expected Behavior

The Worker should attempt to reconnect to the server after the connection has closed, establishing a new connection if the server is found and logging that to the terminal.

Screenshots

No response

Additional Context

No response

Components

Java SDK