"SQS delete message failed...receipt handle has expired" (Artillery Pro)

[sparkida]

This is occurring after running a test with a duration of 5 minutes (however the actual test runs quite a bit longer)

All virtual users finished
Summary report @ 08:13:56(-0800) 2019-02-21
  Scenarios launched:  601863
  Scenarios completed: 542521
  Requests completed:  542521
  RPS sent: 1369.65
  Request latency:
    min: 42
    max: 26049
    median: 99
    p95: 863
    p99: 1919
  Scenario counts:
    options: 601863 (100%)
  Codes:
    200: 542521
  Errors:
    EPROTO: 58991

{ SQSError: SQS delete message failed: Value h0zOmmlGhFbsSCWBFM/8P9pEW4b6533kFILe+DgaL59Jmwzzq6wWjTvln4m3IBf46DRhcshKcMsYCVh3d3MazSIa6KIF+XQ6MCQMc4AWt0erQx0wllCDHfqKL3MmCF7w0H5iPq17ZPdwJLPj9sqWtJFd2KRu6rN9LHr04k9Od7i+4sC4= for parameter ReceiptHandle is invalid. Reason: The receipt handle has expired.
    at Response.<anonymous> (/home/nick/fox/load/node_modules/sqs-consumer/index.js:198:24)
    at Request.<anonymous> (/home/nick/fox/load/node_modules/aws-sdk/lib/request.js:364:18)
    at Request.callListeners (/home/nick/fox/load/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/home/nick/fox/load/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/home/nick/fox/load/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/home/nick/fox/load/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/home/nick/fox/load/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /home/nick/fox/load/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/home/nick/fox/load/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/home/nick/fox/load/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/home/nick/fox/load/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
    at Request.emit (/home/nick/fox/load/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/home/nick/fox/load/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/home/nick/fox/load/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/home/nick/fox/load/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /home/nick/fox/load/node_modules/aws-sdk/lib/state_machine.js:26:10
  name: 'SQSError',
  message: 'SQS delete message failed: Value h0zOmmlGhFbsSCWBFM/8P9pEW4b6533kFILe+DgaL59Jmwzzq6wWjTvln4m3IBf46DRhcshKcMsYCVh3d3MazSIa6KIF+XQ6MCQMc4AWt0erQx0wllCDHfqKL3MmCF7w0H5iPq17ZPdwJLPj9sqWtJFd2KRu6rN9LHr04k9Od7i+4sC4= for parameter ReceiptHandle is invalid. Reason: The receipt handle has expired.' }

[sparkida]

Just realized that this zombied the test/cluster, all 20 tasks have been running and I had to manually stop them, but "All virtual users finished" was issued and the CLI connection was terminated.

[hassy]

Thanks for the bug report @sparkida! Just to clarify - you ran the test, the CLI printed the "finished" message above followed by an error stack trace, and following that you could still see 20 ECS tasks running which had to be killed manually?

[sparkida]

@hassy that's correct, this happens intermittently and has been happening since we started testing a couple months ago

[iwong3]

Same thing is happening here, although my test is supposed to run for 7 hours, but stops after 1 hour.