ERROR: aws-serverless-express connection error with every requests

[hoangsetup]

"aws-serverless-express": "3.2.0",
serverless@1.29.2

It working fine in yesterday.

CloudWatch log
a0c895f5-fe7c-11e8-b3ae-4f8962debec2 ERROR: aws-serverless-express connection error
1544667186190,/aws/lambda/function,2018-12-13T02:13:06.190Z a0c895f5-fe7c-11e8-b3ae-4f8962debec2 { Error: socket hang up at createHangUpError (_http_client.js:299:15) at Socket.socketOnEnd (_http_client.js:391:23) at emitNone (events.js:111:20) at Socket.emit (events.js:208:7) at endReadableNT (_stream_readable.js:1064:12) at _combinedTickCallback (internal/process/next_tick.js:138:11) at process._tickDomainCallback (internal/process/next_tick.js:218:9) code: 'ECONNRESET' }

[brettstack]

Did you change anything on your end? e.g did you modify your API Gateway or Lambda Function?

[hoangsetup]

@brettstack No, nothing change. We have 2 stages on aws, both gets the same issue.

[keitakn]

The same error is occurring in our service.

"aws-serverless-express": "3.3.5",
"serverless": "1.32.0",

[brettstack]

Thanks for reporting. If you could provide additional information such as how you're implementing the handler (e.g is it wrapped in a promise?); Which version of Node.js runtime you're using, that might help.

[hoangsetup]

@brettstack
this is my handler, LambdaNode ver 8.10, I am using Typescript

exports.handler = (event, context, callback) => {
  try {
    context.callbackWaitsForEmptyEventLoop = false;
    context.succeed = (response) => {
      callback(null, response);
    };
    console.log("START REQUEST", event);
    return awsServerlessExpress.proxy(server, event, context);
  } catch (e) {
    console.info("===> Handler Error: ", e);
  }
};

[keitakn]

@brettstack
We are using the runtime of Node.js 8.10.

Here is our handler.

We also use TypeScript.

const server = awsServerlessExpress.createServer(app, undefined, binaryMimeTypes);

export const expressHandler = async (
  event: lambda.APIGatewayEvent,
  context: lambda.Context,
  callback: lambda.Callback,
) => {
  context.callbackWaitsForEmptyEventLoop = false;

  return new Promise((resolve) => {
    awsServerlessExpress.proxy(
      server,
      event,
      {
        ...context,
        succeed: process.env.IS_OFFLINE ? context.succeed : resolve,
      },
    );
  });
};

[hoangsetup]

Only request include Authorization in header has been crash.
my authorize function is working fine.

[brettstack]

Thanks for the additional details. Is it 100% failure rate for those?

[hoangsetup]

@brettstack Yup! it is 100%

[brettstack]

Which region(s)?

[brettstack]

I notice you both use context.callbackWaitsForEmptyEventLoop = false; and are overriding context.succeed. Might be relevant. We are investigating.

[hoangsetup]

@shwetaskatdare My region is ap-northeast-1

I am going to try remove callbackWaitsForEmptyEventLoop setting

[keitakn]

@brettstack Our region is ap-northeast-1.

We are using aws-serverless-express in multiple applications.

Some applications do not use context.callbackWaitsForEmptyEventLoop = false;.

The following is one of them.

import lambda from 'aws-lambda';
import awsServerlessExpress from 'aws-serverless-express';
import next from 'next';
import app from './app';

const appNext = next({ dir: './src', dev: false });
const server = awsServerlessExpress.createServer(app(appNext));

module.exports.handler = (event: lambda.APIGatewayEvent, context: lambda.Context, callback: lambda.Callback) => {
  appNext.prepare().then(() => {
    return awsServerlessExpress.proxy(server, event, context);
  });
};

This application is running with Next.js and aws-serverless-express.

This application also worked until yesterday, but aws-serverless-express connection error came to occur.

[fukuiretu]

Hi!

In our environment we run Nuxt.js but it stopped working with a similar error.

2018-12-13T03:29:15.120Z    4411feb2-fe87-11e8-a498-f1e00a009a2b    ERROR: aws-serverless-express connection error
2018-12-13T03:29:15.120Z    4411feb2-fe87-11e8-a498-f1e00a009a2b    { Error: socket hang up

Since erroneous response did not occur when accessing by erasing Cookie, I guess that the HTTP Request Header is involved.

region: ap-northeast-1
runtime: nodejs8.10
aws-serverless-express: 3.3.5

Partial snippets of code

// app.js
const express = require('express')
const { nuxt } = require('./core/nuxt')

const app = express()

app.use(nuxt.render)

module.exports.app = app

// handler.js
const awsServerlessExpress = require('aws-serverless-express')
const { app } = require('./app')

const server = awsServerlessExpress.createServer(app)

module.exports.handler = (event, context, callback) => {
  if (process.env.NODE_ENV !== 'production') {
    console.log('[info]', 'Event', JSON.stringify(event))
  }
  awsServerlessExpress.proxy(server, event, context)
}

I hope it becomes helpful 🙏

[hoangsetup]

@brettstack You can take a test on AWS env with a request what include a long header like that

{
"Content-Type": "application/json",
"testheader": "suntinculpaquiofficiadeseruntmollitanimidestlaborum.eyJzdWIiOiJlMTg4YTJiZC0zNzAzLTRhMDYtODA2YS0yYzU2OTNjODlhZjQiLCJldmVudF9pZCI6IjRiYTgxN2UzLWZlYWItMTFlOC1hZTA2LTIzMTY1ZWVkMmYzYSIsInRva2VuX3VzZSI6ImFjY2VzcyIsInNjb3BlIjoiYXdzLmNvZ25pdG8uc2lnbmluLnVzZXIuYWRtaW4iLCJhdXRoX3RpbWUiOjE1NDQ2ODcyMjksImlzcyI6Imh0dHBzOlwvXC9jb2duaXRvLWlkcC5hcC1ub3J0aGVhc3QtMS5hbWF6b25hd3MuY29tXC9hcC1ub3J0aGVhc3QtMV9FNXJuZ1RGZXYiLCJleHAiOjE1NDQ2OTA4MjksImlhdCI6MTU0NDY4NzIyOSwianRpIjoiNTFkMzU0YzItZDMyZC00NWRmLTlhYzUtYzU4MjJjOTc3ZmNlIiwiY2xpZW50X2lkIjoiMWhqa3A0ZGRqZTN1aWVjbDcxcDU1amIyNDYiLCJ1c2VybmFtZSI6InRoaW5oYmcifQ.PzbFkSa11PvIbM-bVQ94YGpqr1U2vu5H2lWeknUSGrhad2ptwKKm_nj6lOHdmSjgkrzEGRcd8mvHsjmKHidq68bC5tG_R3wNBstbV9luu_SeM-3WMIZ3PyXzIpszbQKdzbz3o3pXj1in6E1RUb7C-wgob9r6GbmVIVqtND1Bbxk4RFopt4nUU4VmfizOB0e_anKq8qMKNS6FMYVMxr_6xB-E0TB1fraDLur1pAP9ilpJALgfYX1nSzEMnSi6VgtDrXcuGjEfIUSG49kf6LOsWCJUD2-qF_X3xoS9TCIIY_Jmsv_Qe1CicI6A-i0n0XbziYcMQFC6tt3Mhfz6TsNtAg"
}

testheader header value is a string with length is 937

[tobinbc]

It is also 100% failure for us as of today (13/12/2018), no code changes since yesterday. Using Amplify library to create the lambda function / express.

I am using eu-west-1 region for this api gateway, lambda, dynamodb.

My current 'workaround' is switching all lambdas to use Node 6.10 (they are now working).

[SaMnCo]

Same here, with nodejs 8.10, version 3.3.5 in eu-west-1, also no code change.

100% failure rate, app down. We are going to test a few things from now and report if we find a work around.

[j0k3r]

Same issue here since few hours.
nodejs 8.10, eu-west-1, 3.3.5 too.

Are you guys using your Lambda behind API Gateway (and using a JS frontend, like React)?
It looks like here when we send a request using Chrome (version 70 or 71) it fails. Firefox is fine.

Looks like when cookies are defined, errors appears.

Can't this be a change related to AWS Lamba itself? 😞

[0xR]

Same here in EU-WEST-1

[cipriancaba]

we have 3 stages deployed in eu-west-1.. only staging works.. Using API Gateway with Cloudfront

[kkarimi]

Same here with 3.3.5 in eu-west-1. TypeScript -> node 8.10

[asiniscalchi]

Same here, with nodejs 8.10, version 3.3.5 in eu-west-1.

[RossCasey]

Same here, with nodejs 8.10, version 3.3.5 in eu-west-1 as well

[peterservisbot]

Same, with nodejs 8.10, version 3.3.5 in eu-west-1! 👎

[licalac]

Same here eu-west-1b !

Ionic 3 =>API Gateway => Lambda = DynamoDB was working like a charm :(

[cipriancaba]

Is everyone here using context.callbackWaitsForEmptyEventLoop = false ?

[0xR]

So I guess we'll receive service credit for this outage? This is a long time for a major outage.

@brettstack Is anyone investigating what changed on AWS side?

[cianclarke]

Minimally reproducible testcase - upload this lambda:
Archive.zip

With this invoke payload:
https://gist.github.com/cianclarke/7a2d284747f6323e348f0012401f566b

Also reproducible locally, but using Node 8.14. Run testcase.js in same repo:

Cians-MacBook-Pro-2:deleteme cianclarke$ nvm use 8.14
Now using node v8.14.0
Cians-MacBook-Pro-2:deleteme cianclarke$ node testcase.js
9781
h1
h2
h3
h4
h5
h6
h7
h8
h9
h10
h11
reqerr
{ Error: write EPIPE
    at WriteWrap.afterWrite [as oncomplete] (net.js:868:14) errno: 'EPIPE', code: 'EPIPE', syscall: 'write' }
ERROR: aws-serverless-express connection error
{ Error: write EPIPE
    at WriteWrap.afterWrite [as oncomplete] (net.js:868:14) errno: 'EPIPE', code: 'EPIPE', syscall: 'write' }
succeed
{ '0': { statusCode: 502, body: '', headers: {} } }

[david4City]

Same here with a TS +Babel => node 8.10 config. Running on eu-west-1.

[ylynn]

We involved AWS Lambda to investigate. They are working on a fix.

[zoellner]

running into similar issues on local development. Have isolated it to be related to the cookie size in our case (maybe overall header size?)

[zoellner]

I'm now able to reproduce the error consistently

After line https://github.com/awslabs/aws-serverless-express/blob/cb3b17d5fcfad911669e33433c191bb558945ed9/src/index.js#L136
i.e. when the request is initialized in forwardRequestToNodeServer, I watch the size of the header. Whenever JSON.stringify(req.getHeaders()).length >= 8180 the request fails with one of the errors below.

It might depend on the request itself a little bit, but the result seems to be consistent with https://nodejs.org/docs/latest-v8.x/api/http.html#http_http_maxheadersize which defaults to 8KB.

Most of the header size comes from headers['x-apigateway-event'] and headers['x-apigateway-context'], i.e. the hack to attach the event context to the request.

Possible Errors:

{ Error: socket hang up
    at createHangUpError (_http_client.js:342:15)
    at Socket.socketOnEnd (_http_client.js:437:23)
    at emitNone (events.js:111:20)
    at Socket.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:139:11)
    at process._tickDomainCallback (internal/process/next_tick.js:219:9) code: 'ECONNRESET' }

{ Error: write EPIPE
    at WriteWrap.afterWrite (net.js:868:14) errno: 'EPIPE', code: 'EPIPE', syscall: 'write' }

[neo-headz]

I get this error intermittently after switching my API Gateway's custom domain to a regional endpoint. This is to support a multi-region, latency-based routing policy. Maybe the headers are different between an edge optimized and a regional endpoint?? @zoellner I'm using your fork and it seems to fix this issue. Thanks!

[asurinsaka]

02/22/2019 aws-serverless-express@3.3.5, no serverless module used.
I get this error today.
After I tried to change the header in API Gateway from json to html and updated the zip file in lambda, then I got the error.

[willwarreniv]

Also got this error. I was able to work around it by changing my node version from v8.15 to v8.10. Not exactly the best fix, but it will allow you to get your work done.

[mbenedettini]

I've run into this with serverless-offline, trying to upload a file via a POST. Ended up switching to https://github.com/dougmoscrop/serverless-http with which this error is not showing.

[dpurrington]

This is still happening to me ~ .5% of requests. Not good. Using serverless-http fixed the issue for me as well.

Maybe load-related. I'm load testing using artillery:

   3   │   http:
   4   │     pool: 40
   5   │   phases:
   6   │     - duration: 10
   7   │       arrivalCount: 100
   8   │   defaults:
   9   │     headers:
  10   │       "Content-Type": "application/json"
  11   │ scenarios:
  12   │   - flow:
  13   │     - loop:
  14   │       - post:
  15   │           url: "/v1/events/query"
  16   │           json:
  17   │             uid: 1
  18   │       count: 200

[adrai]

Someone already tried nodejs10.x ? #234 (comment)

[david4City]

Hi, sorry to up this again, I'm running a TS +Babel => node 8.10 config on eu-west-1 and I'm currently getting this error :

ERROR: aws-serverless-express connection error
{ Error: socket hang up
at createHangUpError (_http_client.js:299:15)
at Socket.socketOnEnd (_http_client.js:391:23)
at emitNone (events.js:111:20)
at Socket.emit (events.js:208:7)
at endReadableNT (_stream_readable.js:1064:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickDomainCallback (internal/process/next_tick.js:218:9) code: 'ECONNRESET' }

Am I the only one concerned ?

[seeheee]

I try to link the dialog flow and AWS using the lambda function, but an error occurs.
Tell me how to fix it.

{
"errorType": "Runtime.ImportModuleError",
"errorMessage": "Error: Cannot find module 'aws-serverless-express'",
"trace": [
"Runtime.ImportModuleError: Error: Cannot find module 'aws-serverless-express'",
" at _loadUserApp (/var/runtime/UserFunction.js:100:13)",
" at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)",
" at Object. (/var/runtime/index.js:36:30)",
" at Module._compile (internal/modules/cjs/loader.js:776:30)",
" at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)",
" at Module.load (internal/modules/cjs/loader.js:653:32)",
" at tryModuleLoad (internal/modules/cjs/loader.js:593:12)",
" at Function.Module._load (internal/modules/cjs/loader.js:585:3)",
" at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)",
" at startup (internal/bootstrap/node.js:283:19)"
]
}

[adrai]

@seeheee probably your uploaded lambda function does not include the aws-serverless-express dependency

[seeheee]

How do you include it?

[adrai]

@seeheee how do you deploy your lambda function?
via AWS UI?
Claudia.js?
serverless?
CloudFormation?
SAM?

[seeheee]

I deploy your lambda function via AWS UI

[adrai]

so probably you haven’t included node_modules/aws-serverless-express

[seeheee]

I don't know how to include a moduleㅠㅠ

[adrai]

Create a zip of your whole app Il giorno 29 lug 2019, alle ore 14:22, seeheee <notifications@github.com<mailto:notifications@github.com>> ha scritto: I don't know how to include a moduleㅠㅠ — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#199>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAIJF4VSEHHY5DCZ5V5FFFLQB3ORXANCNFSM4GKDTF4Q>.

[svarughese91]

any resolutions for this? I am using node 10.X and I am getting this issue as well. I am using serverless deploy to deploy the lambda function. Any suggestions?

[dineshrawat11]

Same issue!! Any resolution?

[emmanuel-omegaalexis]

I'm seeing the same issue with node10. No code changes just move the lambda function from 8.10 to 10. 8.10 is EOL soon. Does anyone have a resolution?

[emmanuel-omegaalexis]

I'm seeing the same issue with node10. No code changes just move the lambda function from 8.10 to 10. 8.10 is EOL soon. Does anyone have a resolution? Thanks

[adrai]

perhaps give serverless-http a try....
Seems also more performant: https://github.com/fastify/aws-lambda-fastify#%EF%B8%8Fsome-basic-performance-metrics

[zoellner]

@emmanuel-omegaalexis you mentioned previously you're having issues with my fork. Can you let me know what the issue is and a snippet showing how you're calling the library?
I'm using my fork with node 8/10/12 without any problems. Note that my fork doesn't have any of the legacy resolver modes but just the promise mode.

[emmanuel-omegaalexis]

Hi @zoellner, I have seen others have had success with your fork so I assume it is my implementation.

const awsServerlessExpress = require('aws-serverless-express');
const app = require('./app');
const server = awsServerlessExpress.createServer(app);
 
 exports.handler = (event, context) => awsServerlessExpress.proxy(server, event, context, {
     apiGatewayEventWhitelist: ['requestContext']
	 });

I changed to serverless-http and it is working @adrai Thank you.

[mickey2012ex]

i encountered this issue when running sls offline

finally resolve it by installing aws cli
After installation go to cmd, type "aws configure"
enter AWS Access Key Id,AWS Secret Access Key, region
And the error is gone, the key value can be fake

See if it can help

[barakd]

Had the same issue, for me the issue was quite a tricky one, so I hope I'll save someone a bit time,
I had typescript export miss match, so my export default app;
had to be used as const server = awsServerlessExpress.createServer(app.default) and not const server = awsServerlessExpress.createServer(app)
the latter caused the socket timeout.