fetch requests fails as network errors on chrome console

[omar-dulaimi]

Prerequisites

• I am running the latest version. (up upgrade)
• I searched to see if the issue already exists.
• I inspected the verbose debug output with the -v, --verbose flag.
• Are you an Up Pro subscriber?

Description

Our function randomly stops working and keeps interrupting requests either immediately or after a while. Sometimes a lot less time to fail than the timeout.

here are some error logs on cloudwatch:
"message": "error creating handler: waiting for http://127.0.0.1:36785 to be in listening state: timed out after 15s"

RequestId: id Error: Runtime exited with error: exit status 1
Runtime.ExitError

Lambda execution failed with status 200 due to customer function error: RequestId: id Error: Runtime exited with error: exit status 1. Lambda request id: id

To give you more background, our function is a graphql server using graphql yoga. Here's how it's configured:

const server = new GraphQLServer({
    typeDefs: './src/schema.graphql',
    resolvers,
    middlewares: [permissions],
    resolverValidationOptions: {
        requireResolversForResolveType: false,
    },
    context: req => ({ ...req, db }),
});


server.start(
    {
        playground: false,
        validationRules,
        formatError,
    },
    ({ port }) => {
        console.log(`server started at port: ${port}`);
        console.log('process.env.PORT', process.env.PORT);
    }
);

now I understand that PORT is managed by up, so it's given as is. Once I noticed that the logs above print not the same port numbers, I tried to set the port of the server to process.env.PORT but that didn't solve the problem.

What's weird is that our production server has the same config without changing the port number, keeping it to the default of graphql yoga which is port 4000.

So how does that work?
each lambda invocation means a new PORT, right?
shouldn't the graphql server listen to the same port number, or would it work on the default 4000

Now all that above wasn't enough to pinpoint the exact location of the issue, It's still vague.

More error logs:

{"fields":{"app-name","commit":"123","error":"net/http: timeout awaiting response headers","id":"0","plugin":"relay","stage":"staging","version":"504"},"level":"warn","timestamp":"2021-02-06T20:11:50.190274865Z","message":"request timeout"}

2021/02/06 20:11:50 http: proxy error: net/http: timeout awaiting response headers

Any help is highly appreciated.

2021/02/06 20:11:50 http: proxy error: net/http: timeout awaiting response headers

[Network error]: {"name":"ServerParseError","response":{},"statusCode":502,"bodyText":""}

UPDATE:
just noticed that the server actually starts using the same port that UP defines at runtime:

    "message": "process.env.PORT 38369"
    "message": "server started at port: 38369"
    "message": "error creating handler: waiting for http://127.0.0.1:38369 to be in listening state: timed out after 15s"

So judging by the last error message, does it mean that the server/up is taking a lot of time to be in listening state ?
any way to debug/increase that 15second number?

[tj]

Unfortunately debugging that is pretty app-specific, since it means your app is either taking too long (say connecting to databases before, reading files etc) before listening on PORT. Sometimes it can be as simple as forgetting to pass process.env.PORT or perhaps passing it incorrectly to the server (if it expects a number instead of string for example).

[omar-dulaimi]

Hey @tj
Increasing the timeout did fix a big chunk of the issue to make development a little bit easier than before. But the problem still occurs randomly; first fetch sometimes, immediately sometimes, sometimes it's really slow to respond, etc.

Since you mentioned connecting the database, what we use is a Prisma 1 server hosted on EC2. That Prisma server in turns connects to our RDS database. Do you think the issue might be related to these parts?
I just don't want this issue to come to the production server once I merge the code there. Anything else I could do to better debug this?

[omar-dulaimi]

Locally it works perfectly fine though. So the requests passing through apigw and lambda run into unreported issue it seem.

[omar-dulaimi]

For whoever struggles with this in the future, make sure to remove docker completely. You're welcome!