Long running CLI commands repeat every 2 minutes

[duncan-c]

Description:

We're using Bref with Laravel, and we have some artisan commands which take some time to complete (usually database operations).

We've seen that after a command has been running for two minutes it is triggered again (but the original one continues running), and then after another two minutes, it's triggered again, and so on, etc. This can continue infinitely, and can cause some real problems if the commands are doing database manipulations which then conflict (for example I just added a lot of duplicate data into production!).

As soon as the call to serverless bref:cli is manually stopped (or an error causes it to stop) the commands stop repeating.

How to reproduce:

Create an artisan command which just waits 5 minutes and stops:

$this->info('Command started!');
for ($i = 0; $i < 5; $i++) {
    sleep(60); // Wait for 1 minute
    $this->info(($i+1).' minute(s) have passed.');
}
$this->info('Command completed!');

Then deploy and run the command: serverless bref:cli --args="wait-5-mins"

The command will never complete itself, and the Lambda logs will show the command starting every 2 minutes:

Bref Version: 2.1.16

[duncan-c]

Turns out this is because the AWS CLI timeout is 2 minutes by default. Setting a larger (or 0) value in the AWS_CLIENT_TIMEOUT environment variable fixes this issue.

serverless/serverless#10935