Skip to content

Tuning down log-level for spawn events to reduce mem. usage of default test config#211

Merged
sarsko merged 1 commit intoawslabs:mainfrom
dylanjwolff:spawn-log-level
Sep 8, 2025
Merged

Tuning down log-level for spawn events to reduce mem. usage of default test config#211
sarsko merged 1 commit intoawslabs:mainfrom
dylanjwolff:spawn-log-level

Conversation

@dylanjwolff
Copy link
Copy Markdown
Contributor

The test-log crate uses INFO level logging by default. Thus anything at that level will be logged by default when running tests. The logs are also buffered in memory until the test has completed, which can lead to high memory consumption for tests producing verbose logs, in addition to increasing the test runtime.

While logging has already been turned off by default in CI before this PR (RUST_LOG=off), the default behavior with no additional environment variables causes the tests to run extremely slowly due to verbose logging. This is not a good experience for new developers of Shuttle and generally seems like a bad default.

The primary culprit is logging task creation at INFO level; we have several tests which do millions of shuttle iterations, each containing multiple task spawns. This PR moves that logging to DEBUG, so it is off by default during testing, but can be easily re-enabled by setting RUST_LOG=debug. With this change, running the shuttle test suite on my machine goes from:

cargo test --release: from ~244s and >20 GB of RAM to ~183s and <1GB of RAM


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@sarsko
Copy link
Copy Markdown
Contributor

sarsko commented Sep 4, 2025

Its wild to me that this one change causes a 20x reduction in memory usage and a 25-ish% perf improvement. I am fine with the change, though I do like this event being INFO, as it is an important event. I’m wondering if the correct solution is to vend a better tracing subscriber (and our own test macro which installs it)

@sarsko sarsko merged commit 152e1b3 into awslabs:main Sep 8, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants