Lazy listener executions do not respect AWS Lambda's provisioned concurrency

[BSouzaDock]

I have a slack application running 100% on AWS and have been having problems for some time with expired_trigger_id error.

As is already known, it is necessary to respond to the slack API within 3 seconds to avoid this error. I even had to create a lambda to run every 3 minutes, invoking the main lambda to escape the cold start, but this is always the wrong solution.

AWS allows the lambdas to be loaded by default by activating the provisioned concurrency option and indicating how many containers you want to leave on, thus avoiding the cold start problem.

It turns out that I configured everything correctly, but when I call the slack API, instead of the request going to the lambda that is already running, it ends up invoking a new one, thus generating the expired_trigger_id error.

I'm in doubt if this is an AWS or Slack Bolt API problem, I even created a topic on stackoverflow: Provisioned lambda correctly configured but not working as it should

If anyone here can help me with this, I appreciate it.

[mwbrooks]

Hey @BSouzaDock 👋🏻 I'm sorry to hear that you're running into this hosting issue, especially when you're app is already running!

I'm not the right person to help you with this problem, but I just wanted to let you know that you're heard and the core Bolt Python team has eyes on your problem 👀 I'll share it with the team so others can jump in to help out!

I'd encourage others to read over the Stack Overflow post because @BSouzaDock has left some incredible detail in it.

[seratch]

Hey @BSouzaDock, according to AWS's document,

Provisioned Concurrency is not supported on the unpublished version of the function ($LATEST). Ensure your client application is not pointing to $LATEST before configuring provisioned concurrency.
https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html#configuring-provisioned-concurrency

pointing to $LATEST does not work with provisioned concurrency. So, you may need to point to the version 42 instead of $LATEST in this case.

[BSouzaDock]

Hello @seratch , thanks for the reply.

It turns out that me lambda is configured correctly exactly as it says in the documentation.

I created an Alias called Prd and pointed the API Gateway to it instead of the Latest version as you can see:

It has 5 lambdas provisioned in version 44

When the requests begin to arrive, the impression I have is that only 1 provisioned lambda receives the requests, and the others, even if started, cannot receive them, so a new lambda is invoked cold and generates the errors.

Despite multiple provisioned lambdas started, only 1 provisioned lambda is receiving requests.

[BSouzaDock]

I performed a very interesting test here, I believe it will help you understand the problem:

1. I created a workflow step and monitored the lambda logs after execution.
2. After sending the workflow, the api gateway redirected the request to the lambda alias Prd(v.45) correctly as shown in the printscreen below (red):
   

Briefly this is the code of the main method of my lambda:

# Start app
def main(event, context):
    show_income_event(event)
    slack_handler = SlackRequestHandler(app=app)

    return slack_handler.handle(event, context)

The show_income_event method is responsible for displaying the request data (body, method, etc.) as shown on the printscreen.

3. During the processing of the request by the bolt api in the line "return slack_handler.handle(event, context)", instead of continuing the execution within the current lambda or using a provisioned lambda, it is invoking a new lambda cold, as I show in printscreen (marked in green), and within this lambda the workflow step is processed. I believe this is the cause of the problem.

[seratch]

@BSouzaDock Are you using lazy listeners in the app and does the pattern invoke the lazy ones? If yes, the lazy listener feature may invoke $LATEST instead. I haven't checked if AWS Lambda runtime provides the lambda version in the event request / context data yet but if the data exists, this part can be improved for better support of provisioned concurrency.

[BSouzaDock]

@seratch Yes, I'm using lazy, and I always call lazy by default. This could be the problem.

You can get the lambda alias at runtime like this:

import os

def lambda_handler(event, context):
    function_name = os.environ['AWS_LAMBDA_FUNCTION_NAME']
    alias = function_name.split(':')[-1]

[seratch]

@BSouzaDock
Accessing the os env variables may work too but currently the lazy listener invocation uses ARN rather than the function name. The string is extracted this way: https://github.com/slackapi/bolt-python/blob/v1.16.4/slack_bolt/adapter/aws_lambda/handler.py#L59

And then, the string is passed this way: https://github.com/slackapi/bolt-python/blob/v1.16.4/slack_bolt/adapter/aws_lambda/lazy_listener_runner.py#L26

It seems that function_version is also available in the event context: https://docs.aws.amazon.com/lambda/latest/dg/python-context.html

We need to check if it's feasible to properly invoke a function with the given function version.

[seratch]

Also, if your issue is only the 3+ second expiration of trigger_id, I'd suggest moving the code parts that require trigger_id to ack function side. With that, you can run the code in provisioned funtion, so that the specific issue should be quickly resolved.