"UNIMPLEMENTED: unknown service" when available service changes

[sangaman]

Problem description

I am making calls to a gRPC server that sometimes switches the available service at a given host and port. If ServiceA is active, attempting calls from ServiceB will give UNIMPLEMENTED errors as expected, and vice-versa.

However, if I start a node process and make a call against ServiceA, then the server switches to ServiceB and I make a call against that, if it switches back to ServiceA I get UNIMPLEMENTED errors on all calls. This is despite calling the close method on the client for a given service before switching between services.

One weird thing is that when debugging this issue in VSCode, if I set a breakpoint while debugging the issue goes away as soon as the breakpoint is hit and I'm able to make calls to ServiceA again.

Reproduction steps

In my case I am testing the code in https://github.com/ExchangeUnion/xud against a server running the code at https://github.com/lightningnetwork/lnd which provides both a WalletUnlockerService as well as a LightningService (services B and A above, respectively). I'll give generic steps below but can offer more specific steps if needed of what I'm actually doing to reproduce the problem. In my setup I can reproduce the issue every time.

1. Start a gRPC server providing ServiceA.
2. Start a node process, create a client for ServiceA and perform a call.
3. Restart the gRPC server and have it provide ServiceB on the same port.
4. Close the ServiceA client in the same node process and create a client and make a call against ServiceB.
5. Disable ServiceB on the server and enable ServiceA on the same port.
6. Close the ServiceB client in the same node process and recreate a client and make a call against ServiceA.
7. The call will fail with an UNIMPLEMENTED error

Restarting the node process resolves the error, it can successfully make calls against ServiceA again.

Environment

• Ubuntu 19.04 amd64
• Node 10.15.3 installed via nvm
• gRPC@1.21.1

Additional context

Some discussion specific to our project:

• xud doesn't refresh "lnd is WaitingUnlock" state ExchangeUnion/xud#1090 (comment)
• xud stuck with lnd "error": "12 UNIMPLEMENTED: unknown service lnrpc.Lightning" state ExchangeUnion/xud#1140

[murgatroid99]

First, I have to say that we would recommend against this pattern of starting and stopping different clients and servers on the same host and port. The gRPC library is written with the assumption that different server instances on the same port are identical, and that any connection it can establish on that port is the server it's looking for. Would it be possible to instead run the two servers on different ports, serve both servers from a single server?

In this specific case, are you sure that the Service B server has finished shutting down before starting the second client for Service A? If not, that client may actually be connected to the Service B server.

[sangaman]

Thanks for the response. Unfortunately I don't have control over the way the server implements these services - by design there's a single RPC port for these two services and they are never available at the same time - although I can recommend that they change that. One service provides a call to "unlock" the other service, and when the server application is restarted it comes back in the "locked" state again.

I did make one mistake (now corrected) in the steps above in that we don't necessarily make a call against ServiceB in step 4.

As to your question, I'm actually not certain. In some cases I would create a client for ServiceA, attempt a call, and then close that client if the call errored with UNIMPLEMENTED with this all happening before ServiceA was actually available. But then when ServiceA does become available, recreating the client and attempting a call would still error with UNIMPLEMENTED.

I will try running some more tests to see if the issue presents itself if I don't make a call against ServiceB before it's available and update this issue. Thanks again for you time.

[murgatroid99]

It's not about whether ServiceA is available yet, but about whether ServiceB is still running. The gRPC library binds ports with SO_REUSEPORT, so both of those servers can be running. You need to make sure exactly one server is running at a time for this setup to work consistently.

Even if you need to switch off which server is running, it would really simplify things if you could run them on different ports.