Connecting to shutdown servers hangs client a randomly time to emit error 14 UNAVAILABLE #2031
Comments
|
Can you share the full log that you took those excerpts from? There may be information in other parts of it that will help understand what exactly is happening here. |
|
@murgatroid99 I splitted the log in two sections, to note de hang time of 5 seconds in this example. But there are times that hangs 30 seconds and more frequently than this. Sorry for the long log. I tried to connect lot of times, as you can see: First partHangs here !!!! 5 seconds later continue... Second part 5 seconds later |
|
I think there is a bug there but it's hard to tell what the problem is exactly from the logs. Can you share your test code so that I can run it through a debugger? |
|
Yes, sure. I was simplified the code to only one client, because causes the same issue. dvdriverClientClass.jsconst grpc = require('@grpc/grpc-js');
const protoLoader = require('@grpc/proto-loader');
const { promisify } = require('util');
function promisifyAll(client) {
const to = {};
for (var k in client) {
if (typeof client[k] != 'function') continue;
to[k] = promisify(client[k].bind(client));
}
return to;
}
class DVDriverClient {
constructor(serverURL, protoPath) {
const packageDefinition = protoLoader.loadSync(
protoPath,
{
keepCase: true,
longs: String,
enums: String,
defaults: true,
oneofs: true
});
const dvdriver = grpc.loadPackageDefinition(packageDefinition).dvdriver;
this.client = new dvdriver.DriverService(serverURL,
grpc.credentials.createInsecure()
);
// Convert gRPC's callback APIs to await friendly promises
const {
getCompany,
getAffiliation,
addAffiliation,
deleteAffiliation,
sendCommand
} = promisifyAll(this.client);
this.srvGetCompany = getCompany;
this.srvGetAffiliation = getAffiliation;
this.srvAddAffiliation = addAffiliation;
this.srvDeleteAffiliation = deleteAffiliation;
this.srvSendCommand = sendCommand;
}
async getCompany(params) {
return await this.srvGetCompany(params);
}
async getAffiliation(params) {
return await this.srvGetAffiliation(params);
}
async addAffiliation(params) {
return await this.srvAddAffiliation(params);
}
async deleteAffiliation(params) {
return await this.srvDeleteAffiliation(params);
}
async sendCommand(params) {
return await this.srvSendCommand(params);
}
}
module.exports = {DVDriverClient};index.jsconst PROTO_PATH = __dirname + '/protos/dvdriver.proto';
const {DVDriverClient} = require('./dvdriverClientClass');
const clients = new Map();
const GRPC = {
loadClients: function () {
return new Promise((resolve, reject) => {
// here I was simplified this to only one company (server),
// because it's the same for testing purposes
const client = new DVDriverClient('192.168.10.20:50052', PROTO_PATH);
clients.set(1, client);
resolve()
});
},
// Itera por todas las empresas
broadcastRequest: async (companyId, method, params) => {
try {
if (clients.has(companyId)) {
//console.log('Query', companyId)
const response = await clients.get(companyId)[method](params);
//console.log('OK', companyId)
return response;
}
} catch (error) {
console.log('Company ID', companyId, ':', error.message)
}
},
getCompanies: async () => {
const output = {};
for (const companyId of clients.keys()) {
output[companyId] = await GRPC.broadcastRequest(companyId, 'getCompany', null);
}
return output;
},
getAffiliations: async (appid) => {
const output = {}
for (const companyId of clients.keys()) {
output[companyId] = await GRPC.broadcastRequest(companyId, 'getAffiliation', appid);
}
return output
}
}
module.exports = GRPC;Then in another file I require const grpcClient = require('../../grpc');
grpcClient.loadClients();In this file I receive requests from an API REST, to call various grpc unary calls. So the client is load once. Then I do calls to grpc, such as: const companiesSettings = await grpcClient.getCompanies()
console.log(companiesSettings)const affiliations = await grpcClient.getAffiliations({appid: appid});
console.log(affiliations)and so on. This is my proto: // Proto file para comunicar dvdriver con cada nodo (empresa)
// dvdriver es el servidor, y los nodos los clientes
syntax = "proto3";
option optimize_for = SPEED;
package dvdriver;
message Empty {}
message Company {
bool modulo_despacho = 1;
bool modulo_taximetro = 2;
bool modulo_q_r = 3;
bool taximetro_auto = 4;
bool libre_ocupado_externo = 5;
bool panico_externo = 6;
bool agregarse_base = 7;
bool boton_cancelar_despacho = 8;
bool boton_pasajero_a_bordo = 9;
string whatsapp = 10;
}
message AffiliationQuery {
string appid = 1;
}
message AffiliationResponse {
message Affiliation {
Company company = 1;
uint32 affiliated = 2;
bool locked = 3;
message Link {
bool free = 1;
string alias = 2;
string movil = 3;
}
repeated Link links = 4;
}
Affiliation affiliation = 1;
}
message AffiliationRequest {
uint32 id = 1;
map<string, string> info = 2;
}
message AffiliationDelete {
uint32 id = 1;
string appid = 2;
}
message Command {
uint32 id = 1;
string command = 2;
string appid = 3;
map<string, string> options = 4;
}
service DriverService {
rpc GetCompany (Empty) returns (Company) {}
rpc GetAffiliation (AffiliationQuery) returns (AffiliationResponse) {}
rpc AddAffiliation (AffiliationRequest) returns (Empty) {}
rpc DeleteAffiliation (AffiliationDelete) returns (Empty) {}
rpc SendCommand (Command) returns (Empty) {}
}
For testing you need to do lot of calls (ex EDIT: Removed deadlines in code for testing purposes |
|
Your code sets a two second deadline for the methods you mentioned. It is never going to hang. |
Ignore these deadlines. These are there only to stop the hang as a workaround. Please, test it without these deadlines. EDIT: I edited the code and removed the deadlines. Sorry |
|
Hello @murgatroid99 . Were you able to reproduce the issue? |
|
I was not able to reproduce it, but fortunately I think I found the bug anyway. I published |
|
Well, I can confirm that 1.5.4 fix this issue! I tested a lot of times with 1.5.3 and 1.5.4. Seams also that 1.5.3 fails only when I try to connect to an IP, not a FQDN. But I am not sure 100% about this. But definitely 1.5.4 fix this issue, and also the 14 error response is more fastest. Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Problem description
I have configured 4 clients, to make unary calls to 4 servers. Each client is an instance from a class, and stored in a js Map. Some calls need to do on all servers, and another calls only calls one. Because few servers are shutdown, I send the same call (with each client corresponding to the instance with the ip:port of each server) and get for those are shutdown
error 14 UNAVAILABLE. That's fine! But randomly some clients hangs waiting for the connection error. So, every call to a shutdown server giveme error 14 at randomly time, from miliseconds, to 30 seconds!!! Why? I suppose the error should emited instantaneous. This cause me issues, because I need to check all servers before continue with de application code.Reproduction steps
Create a simple grpc client and try to make an unary call to any no existent server ip:port. You get and error 14. Repeat the same call few times, and you will see that sometimes client hangs for seconds.
Environment
Additional context
I was simplified my client to only one to reproduce the same issue. I think this issue is relate with those are around there suche as #1591 #1815 or https://stackoverflow.com/questions/61565913/why-does-my-node-js-grpc-client-take-3-seconds-to-send-a-request-to-my-python-gr
Of course, I was setup a deadline for the client to avoid hangs waiting for the error 14, but this is a workaround. A 1 second deadline is ok, but it is not a fix.
This is the console output. As you can see, there is 30 seconds waiting to connect. I want to fail instantly
Only for reference, this is the console output when I instanciate the client class:
Thanks for the help and this module!
Regards,
Normando
The text was updated successfully, but these errors were encountered: