Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

memory_store_tests::ensure_full_copy_of_bytes_is_made_test regularly failing #352

Closed
chrisstaite-menlo opened this issue Oct 30, 2023 · 2 comments · Fixed by #528
Closed

Comments

@chrisstaite-menlo
Copy link
Collaborator

During the ASAN build the test memory_store_tests::ensure_full_copy_of_bytes_is_made_test regularly fails.

failures:

---- memory_store_tests::ensure_full_copy_of_bytes_is_made_test stdout ----
thread 'memory_store_tests::ensure_full_copy_of_bytes_is_made_test' panicked at 'Memory usage increased by 1.3929403332736068 perc, which is more than 1.2 perc', cas/store/tests/memory_store_test.rs:109:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
@aaronmondal
Copy link
Member

I keep encountering this as well during CI testing. I raised the limit from 1.1 to 1.2 already, but asan causes so much increase in memory usage that we'd probably need something like 1.6-1.8 here. I'm torn whether that's the right approach though since it would reduce the usefulness of the test for non-asan runs.

Wdyt about disabling it for the asan workflow entirely? Or we could try to make memory usage of turbo cache more predictable?

Another interesting option that I believe @allada already played around with is to use a different allocator altogether which likely has a big influence on this test. There is a good chance that tcmalloc or mimalloc could give us lower memory usage and an overall performance improvement essentially for free.

@allada
Copy link
Member

allada commented Oct 30, 2023

I'd personally prefer to just disable the test for all sanatizers. This test is a very specific regression test, so I think it's ok to disable it under certain scenarios.

allada added a commit that referenced this issue Dec 19, 2023
Gathers stats over memory usage rather than one sample.

closes #352
allada added a commit that referenced this issue Dec 20, 2023
Gathers stats over memory usage rather than one sample.

closes #352
allada added a commit that referenced this issue Dec 20, 2023
Gathers stats over memory usage rather than one sample.

closes #352
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants