Fix flaky lambda concurrency test

[joe4dev]

Motivation

A flaky Lambda test occasionally blocks the pipeline.

Thank you @bentsku for raising this https://localstack-cloud.slack.com/archives/C02U7DN1B4M/p1695076802171769

Changes

• Increase reserved concurrency in test case
• Document race condition
• Add extra debug logging for concurrency exceptions

Discussion

a) The safest fix is to increase ReservedConcurrentExecutions=3 but then we only occasionally test the border case.
b) We could slow down the test and use sleeps to have a high likelyhood of succeeding
c) We separate the mixed async/sync invoke into a separate test case (or after asserting the logs).

[coveralls]

coverage: 80.004% (-3.0%) from 82.998% when pulling 1c5fc7a on fix-flaky-lambda-concurrency-test into 1b16827 on master.

[github-actions]

LocalStack Community integration with Pro

       2 files         2 suites   1h 10m 29s ⏱️
2 214 tests 1 717 ✔️ 497 💤 0 ❌
2 215 runs  1 717 ✔️ 498 💤 0 ❌

Results for commit 1c5fc7a.

[dominikschubert]

a) The safest fix is to increase ReservedConcurrentExecutions=3 but then we only occasionally test the border case.
b) We could slow down the test and use sleeps to have a high likelyhood of succeeding
c) We separate the mixed async/sync invoke into a separate test case (or after asserting the logs).

Let's do a) for now with c) in the long run 👍