Resolved Lambda Issue Regarding Async Callbacks

[j]

Since I saw the line, context.callbackWaitsForEmptyEventLoop = false inside of the lambda code, I thought to myself, "Hm, I don't need to add that, cool!".

After deploying it, I found that any other method request to the server would hang if my lambda function timeout is long enough since my database connection stayed open. I'm able to simultaneously execute direct GraphQL queries, but anything else locks up (playground, options, etc).

My solution was to wrap the handler and manually set context.callbackWaitsForEmptyEventLoop:

const apolloHandler: lambda.APIGatewayProxyHandler = server.createHandler({
  cors: {
    origin: true,
    credentials: true
});

export const handler: lambda.APIGatewayProxyHandler = (
  event: lambda.APIGatewayProxyEvent,
  context: lambda.Context,
  callback: lambda.APIGatewayProxyCallback
) => {
  context.callbackWaitsForEmptyEventLoop = false;

  return apolloHandler(event, context, callback);
};

Perhaps "callbackWaitsForEmptyEventLoop" can be an option in createHandler so this hack doesn't need to be required.

[braidn]

@j do you have more examples of how you are doing this in your setup? Below is kind of what I am doing and it's an async callback, much like you are using. I am running into the same Lambda/hang timeout issue as you.

exports.handler = (event, context, callback) => {
  console.log("remaining time =", context.getRemainingTimeInMillis());
  console.log("functionName =", context.functionName);
  console.log("AWSrequestID =", context.awsRequestId);
  console.log("logGroupName =", context.logGroupName);
  console.log("logStreamName =", context.logStreamName);
  console.log("clientContext =", context.clientContext);

  context.callbackWaitsForEmptyEventLoop = false;

  startServer(event, context)
    .then(handler => {
      return handler(event, context, callback);
    })
    .catch(err =>
      callback(null, {
        statusCode: err.statusCode || 500,
        headers: { "Content-Type": "text/plain" },
        body: "Handlers Failed to Start"
      })
    );
};

I haven't tested GraphQL queries at the Lambda but, the playground and introspection will consume the entire timeout and do nothing. Not sure how to take your solution and fix my issue. Interested though!

[j]

@braidn Hmm, that's interesting.

You might have a CORS issue? Make sure you have the cors config as well as cors setup in api gateway.

Example Sam config (cors part):

Globals:
  Api:
    Cors:
      AllowMethods: "'DELETE,GET,HEAD,OPTIONS,PATCH,POST,PUT'"
      AllowHeaders: "'Content-Type, Authorization, X-Amz-Date, X-Api-Key, X-Amz-Security-Token'"
      AllowOrigin: "'*'"

And your apollo handler:

const apolloHandler: lambda.APIGatewayProxyHandler = server.createHandler({
  cors: {
    origin: true,
    credentials: true
});

Make sure you're on latest versions too.

Ever since I figured this out, Apex Ping is showing 100% uptime (before it was 0%)

[braidn]

@j I actually have these very settings / love that you all write out TypeScript in code examples.

As a resolution here (for my issue): My code above (the async startServer code) does work! However, there is either something on the firewall of the server that's being introspected during the stitchSchema async phase or a problem with Lambda not being able to communicate with the outside world (port blocking). These issues cause the Lambda to timeout and have nothing to do with Apollo Server or using async in a handler.

Thank you for the response Jordan!

👍

[j]

@braidn mine was most likely due to a MongoDB connection and that since the connection was "Lambda Cached" throughout it's lifetime, that any route that doesn't use context.callbackWaitsForEmptyEventLoop = false will fail.

[abernix]

There was a fix for callbackWaitsForEmptyEventLoop on OPTIONS requests which landed in #2638. Anyhow, I'm curious if this issue still needs investigation and should be left open or if we can close it? 😄

[j]

@abernix it would still be nice if you could easily attach async bootstrap code to a lambda handler.

[davidalekna]

Hey guys, I've been trying to solve my implementation with callbackWaitsForEmptyEventLoop but it just doesn't work for me... Everything works fine except rds (db) calls, take about 10 seconds to execute. I've tried context.callbackWaitsForEmptyEventLoop = false; variations but I always get same result... Not sure if anything changed but seems like callbackWaitsForEmptyEventLoop is not taking effect on my lambda function :/

[braidn]

@davidalekna what happens if you setup something like:

const runHandler = (event, context, handler) =>
  new Promise((resolve, reject) => {
    const callback = (error, body) => (error ? reject(error) : resolve(body));

    handler(event, context, callback);
  });

const main = async (event, context) => {
    server = await someNewApolloServer(event, context);
    handler = server.createHandler(serverOptions);
    const response = await runHandler(event, context, handler);
    return response;
};

and in the async someNewApolloServer function set the context of the server using something like:

    context: async ({ event, context }) => {
        someContext....
        context.callbackWaitsForEmptyEventLoop = false;
    }

This should calm down your RDS call times (or it does for me)

[davidalekna]

cheers @braidn, following setup has worked for me. Although very weird that we have to use this hack 🤔😬

const server = new ApolloServer(config);

function runApollo(event, context, apollo) {
  return new Promise((resolve, reject) => {
    const callback = (error, body) => (error ? reject(error) : resolve(body));
    apollo(event, context, callback);
  });
}

export async function handler(event, context) {
  const apollo = server.createHandler({
    cors: {
      origin: true,
      credentials: true,
      methods: 'GET, POST',
      allowedHeaders:
        'Origin, X-Requested-With, Content-Type, Accept, Authorization',
    },
  });

  return await runApollo(event, context, apollo);
}

[GimignanoF]

Any updates on this? I faced the same problem and using the davidalekna trick works, but of course a more elegant solution is preferred :/
Also, I noticed that the default Playground scheme url isn't right when using stages in lambda. For instance, if I publish a lambda in "dev" staging, than the urls for the playground and the graphql server will be something like <api_gateway_url>/dev/graphql. However the playground tries to fetch schema from <api_gateway_url>/graphql. So I have to change that manually every time i access the Playground if is not cached. Not a big deal, but still annoying ¯_(ツ)_/¯

[davidalekna]

@GimignanoF Apollo 3 is out, I think that is the reason why this is a bit dead, I hope it gets fixed soon

[braidn]

Scratching my head here, Apollo Server 3 is out? https://github.com/apollographql/apollo-server/milestone/16

[davidalekna]

Ah you are right, its Apollo Client, nothing todo with this 😬

[GimignanoF]

Scratching my head here, Apollo Server 3 is out? https://github.com/apollographql/apollo-server/milestone/16

Yep, that confused me to ahahahah I hope it gets released soon so they can move on to fix this

[glasser]

Apollo Server 4 replaces a hard-coded set of web framework integrations with a simple stable API for building integrations. As part of this, the core project no longer directly supports a Lambda integration. Check out the @as-integrations/aws-lambda package; this new package is maintained by Lambda users and implements support for the API Gateway v1 and v2 protocols. If you need to work with different AWS projects, you may be able to add that support to that package, or take another approach such as combining the @vendia/serverless-express package with Apollo Server 4's built-in expressMiddleware (this is the same approach that apollo-server-lambda takes in AS3). Sorry for the delay!