Auto-discover unit-test projects in CI

[GarrettBeatty]

Problem

CI runs unit tests via the run-unit-tests target in buildtools/build.proj, which hardcoded only 4 projects:

• SnapshotRestore.Registry.Tests
• Amazon.Lambda.RuntimeSupport.UnitTests
• Amazon.Lambda.Annotations.SourceGenerators.Tests
• Amazon.Lambda.DurableExecution.Tests

As a result, many unit-test projects were never executed in CI, including Amazon.Lambda.AspNetCoreServer.Test, Amazon.Lambda.Core.Tests, Amazon.Lambda.AspNetCoreServer.Hosting.Tests, EventsTests.NET8, Amazon.Lambda.Logging.AspNetCore.Tests, Amazon.Lambda.DynamoDBEvents.SDK.Convertor.Tests, Amazon.Lambda.DurableExecution.Testing.Tests, Amazon.Lambda.DurableExecution.Analyzers.Tests, PowerShellTests, and TestFunctionCSharp. New test projects also silently went unrun unless someone remembered to append them here.

Change

Replace the hardcoded list with buildtools/run-unit-tests.ps1, which discovers unit-test projects by convention — any *.csproj that references Microsoft.NET.Test.Sdk (or sets <IsTestProject>true</IsTestProject>), excluding *.IntegrationTests.csproj (those run in a separate phase via run-integ-tests-parallel.ps1). This mirrors the existing integ-test discovery pattern.

Discovery is content-based rather than name-based because the repo's test projects follow no single naming convention (*.Test, *.Tests, EventsTests.NET8, PowerShellTests, ...). Non-test fixture projects (TestServerlessApp, TestWebApp, HandlerTest, ...) don't reference the test SDK, so they're cleanly excluded.

The script now discovers 14 projects (the original 4 plus the 10 that were being skipped) and exits non-zero listing any that fail.

[GarrettBeatty]

CI run surfaced pre-existing failures (now fixed)

The first CI run went red — as designed, discovery turned on test projects that were never executed in CI before, exposing their pre-existing failures (Amazon.Lambda.AspNetCoreServer.Test: 106 fails; Hosting.Tests: 1). None were caused by the build.proj/script change itself. Fixed in e11f4ac, split by root cause:

Real test bugs (deterministic — reproduced locally)

• BuildStreamingPreludeTests / ResponseStreamingPropertyTests (14 tests): the StandalonePreludeBuilder helper subclassed the HTTP API v2 function, whose BuildStreamingPrelude override collapses headers into the single-value Headers collection — but the tests assert on multi-value MultiValueHeaders (v1/REST semantics, as their section comments describe). Changed the helper to derive from the v1 APIGatewayProxyFunction. These were introduced with the response-streaming feature (Merge response streaming feature branch #2354) and never ran in CI, so the mismatch went unnoticed.

CI-environment robustness (passed locally, failed only in CI)

• inotify limit (~90 tests): failed with The configured user limit (128) on the number of inotify instances has been reached. The AspNetCoreServer tests each start an ASP.NET Core host; the container default was exhausted once all host-starting projects ran. Raised fs.inotify.max_user_instances in ci.buildspec.yml.
• TestMinimalAPI.TestMapPostComplexType: spawns dotnet run out-of-process and failed with a bare exit code 129. Now captures stdout/stderr and handles the WaitForExit timeout so failures are diagnosable and the child can't leak.
• AddAWSLambdaBeforeSnapshotRequestTests: raced a fixed 500 ms delay for the before-snapshot callback (flaky under load). Now polls for the callback.

All AspNetCoreServer.Test (193) and the snapshot tests pass locally on net8.0. Re-running CI to confirm.

[GarrettBeatty]

Unit tests now run in parallel + fixed flakes it exposed

Reworked run-unit-tests.ps1 to mirror run-integ-tests-parallel.ps1: build all discovered projects once serially (avoids the shared ProjectReference build-output race), then run dotnet test --no-build in parallel (throttle 5) with streamed project-prefixed output. All projects run and every failure is reported in one pass (failed projects' output reprinted as a clean block), then exits non-zero listing failures.

Note: this replaces the earlier fail-fast behavior — like the integ script, it now surfaces all failures rather than short-circuiting.

Running projects concurrently surfaced two pre-existing intra-assembly flakes (xUnit parallelizes collections within an assembly by default), both fixed:

1. Hosting.Tests — many tests mutate the process-global AWS_LAMBDA_FUNCTION_NAME; AddAWSLambdaHosting_NotInLambda_DoesNotRegisterHostingOptions (needs it unset) raced tests that set it. Added AssemblyInfo.cs with [assembly: CollectionBehavior(DisableTestParallelization = true)].
2. RuntimeSupport.UnitTests — PositiveHandlerTestsAsync (~1 in 5 under load) failed with "Can't find method name in console text": ExecHandlerAsync points a static logger action on the shared Amazon.Lambda.Core assembly at a per-invocation StringWriter, and LambdaBootstrap's background RunAsync writes through it, so concurrent tests cross writers. Disabled in-assembly parallelization there too, and replaced two tight busy-spin wait loops with await Task.Delay(1). Verified stable across 6 consecutive full-suite runs.

All fixes verified locally on net8.0/net10.0. (The only local-only failure is NativeAOTTests.EnsureNoTrimWarningsDuringPublish, which needs the C++ native linker absent on my dev box but present in CI.)

[GarrettBeatty]

Root-caused the CI hang ('security token expired')

That run wasn't a test failure — all 286 unit tests passed. The build hung for ~1h50m after the tests finished (17:26 → 19:15) until the 2-hour AWS role credential expired, which surfaced as The security token included in the request is expired.

Cause: 13 of 14 unit-test projects printed a completion line; the one that didn't was Amazon.Lambda.AspNetCoreServer.Test. Its TestSnapStartInitialization fired a LambdaBootstrap.RunAsync polling loop against a dead endpoint (localhost:123) as fire-and-forget (_ = bootstrap.RunAsync(cts.Token)). After the assert, the leaked loop kept retrying (the hung log shows GET /api/SnapStart repeating), so dotnet test never exited and the parallel ForEach-Object never completed.

Fix (9ae885ec):

• Keep the RunAsync task; assert SnapStartController.Invoked first, then cancel and await the task so no background work outlives the test. The dead-endpoint loop surfaces OperationCanceledException or HttpRequestException on teardown — both expected, both swallowed.
• Added AssemblyInfo.cs disabling in-assembly parallelization (same approach as Hosting.Tests and RuntimeSupport.UnitTests); this project shares process-global AWS_LAMBDA_* env vars and the static SnapStartController.Invoked flag.

Verified 193/193 on net8.0 and net10.0, stable across repeated runs, no hang.

[GarrettBeatty]

Second hang root-caused → run AspNetCoreServer.Test serially

The previous run hung the same way (unit tests started, then ~1h50m of silence until the credential expired). Same project: Amazon.Lambda.AspNetCoreServer.Test was again the only one of 14 that never printed a completion line — it stalled after ~64 of 193 tests, with a GET /api/SnapStart request started but never finished.

Root cause (deeper than the last fix): TestSnapStartInitialization drives SnapStart init, which invokes the before-snapshot hooks accumulated in the process-global Amazon.Lambda.Core.SnapshotRestore registry. Every test that constructs an APIGatewayHttpApiV2ProxyFunction (dozens of them) enqueues a hook there, and each hook runs in-process ASP.NET requests. When this test finally drains that global queue it replays them all; under the CPU/host contention of the parallel pool that in-process request work deadlocked. The project passes reliably on its own (193/193, ~8s).

Fix (d1bcd3a9): run-unit-tests.ps1 now splits discovered projects into a parallel pool and a serial list. All are built once up front; the pool runs concurrently with --no-build; then the serial projects (currently just AspNetCoreServer.Test) run one at a time afterward. The earlier hardening (SnapStart test tears down its bootstrap; DisableTestParallelization on this assembly) stays as defense-in-depth.

Verified locally end-to-end: the full script completes, AspNetCoreServer.Test passes serially with no hang. (The only local failure is NativeAOTTests.EnsureNoTrimWarningsDuringPublish, the C++-linker issue that's local-only and green in CI.)