-
Notifications
You must be signed in to change notification settings - Fork 306
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FATES P32x2
test appears to fail reliably when using I2000FatesCruRsGs
compset
#1518
Comments
@glemieux I'm changing the type from tests to bug because this looks likely to be a real issue to me, not just something test-specific. |
Discussing this a little more with with @rgknox, he suggested that the issue could possibly be the result of a race condition since the failure mode occurs when there is an attempted deallocation of a patch. Perhaps this hypothesis has some weigh that the faster running river stub version of this test is reliably hitting the race condition, where as the slower running mosart version hits this only rarely? What would be considered a good method in which to debug a potential race condition? |
Given that you said this is the only FATES threading test, my first suspicions would be with threading. You could see if wrapping the relevant code in an an I would probably also take this opportunity to increase your testing with multiple threads. It's really easy to introduce an accidental threading issue in CTSM, and I wonder if the same is true of FATES. And (like a lot of issues, but maybe more so) it is a lot easier to catch a threading issue when it is first introduced rather than trying to find it months or years later. |
@adrifoster ran into this (in possibly a slightly different way) in her recent testing. Since this is not super common, our feeling (from discussion with @adrifoster @ekluzek and @glemieux ) is that, when we run into this, we'll try rerunning the test to get it to pass. What @adrifoster saw in test
without any stack backtrace. |
Since #1275 the |
Brief summary of bug
Trying to run
ERP_D_P32x2_Ld3.f19_g17.I2000Clm50FatesCruRsGs.cheyenne_intel.clm-FatesColdDef
results in a weird RUN failure during deallocation of a patch. However, if the test is run usingI2000Clm50FatesCru
it seem to run more reliably.General bug information
CTSM version you are using: ctsm5.1.dev058-36-g8b6c8f0c
Does this bug cause significantly incorrect results in the model's science? No
Configurations affected:
Details of bug
This test builds successfully, but fails the run with the following error in cesm.log:
This was discovered in the course of PR #1275 in an attempt to reduce the workload for the test times by removing compsets that use MOSART with most fates testmods. @rgknox and I believe we have seen this issue before, but it has usually cleared up upon resubmission. As such we had usually chalked it up to machine instability
Note that this is the only multi-threading test in the fates test suite.
The text was updated successfully, but these errors were encountered: