Connection closed too early under load: ECONNRESET

[fpavageau]

I have a problem that started with version 3.1.6, and still exists in 3.1.8: under enough load, it looks like some connections to the target service are closed too early.

The only log I get from the microgateway looks like:

2020-11-17T21:25:24.699Z [error][api.example.com][1][example-nonprod][non-prod][some/api/path][][][67c658c0-291b-11eb-8fdb-f165ba50582a][microgateway-core][][GET][502][socket hang up][ECONNRESET][]

The target service is behind an AWS load balancer, and the access log for the LB shows that the connection was closed before the service could respond (which is logged as a 460 HTTP code for the service and 0 HTTP code for the LB: connection closed by the client). Apparently, the microgateway then responds with something that look like

{
  "message": "socket hang up",
  "code": "ECONNRESET"
}

and a 502 HTTP code (as logged).

The issue is only visible with enough traffic / load (it's almost never seen on our development environment), which could mean a problem with some resource shared between requests. I realise there's not much information to track the bug, I wonder if there's some log I could activate to provide more. Sticking to 3.1.5 solves the problem in the meantime, but that's not a long-term solution.

Given the limited changes between 3.1.5 and 3.1.6, there's a good chance that the issue is actually in one of the dependencies, such as microgateway-core. However, package.json in 3.1.6 seems to reference a branch of microgateway-core ("microgateway-core": "apigee/microgateway-core#for-3.1.6") instead of a real release. In 3.1.8, it does reference a release ("microgateway-core": "^3.1.9").

[wayne-m-miller]

Discovered the configure module is attempting to generate a URI with spaces in it.

The errant call can be found here:

Also found a workaround, specify the CLI url argument.

[fpavageau]

I just tested, the issue still exists in 3.2.1. I also changed the runtime from Node 12 to Node 14, it doesn't matter. I had hoped the issue listed as fixed in the release notes for 3.2.0 (173064680) was the same problem even if it was noted as starting with 3.1.7, but it's not.

We're still stuck at 3.1.5(and Node 12).