Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Termination detection fault with dtd #634

Open
abouteiller opened this issue Feb 7, 2024 · 2 comments
Open

Termination detection fault with dtd #634

abouteiller opened this issue Feb 7, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@abouteiller
Copy link
Contributor

Describe the bug

Seen only once on #321, need to see if it also happens on master

To Reproduce

29302 Command: "/apps/spacks/2023-08-14/opt/spack/linux-rocky9-x86_64/gcc-11.3.1/openmpi-4.1.5-2rgaqk2wseegpmbdbbygvwrljccjaqsk/bin/mpiexec" "-n" "4" "dsl/did/dtd_test_task_insertion"
...
 29329 dtd_test_task_insertion: /home/bouteill/parsec/dplasma/parsec/parsec/mca/termdet/local/termdet_local_module.c:114: parsec_termdet_local_termination_dete
       cted: Assertion `tp->tdm.monitor == PARSEC_TERMDET_LOCAL_TERMINATED' failed.
 29330 [leconte:4113702] *** Process received signal ***
 29340 [leconte:4113702] [ 7] /home/bouteill/parsec/dplasma/build.cuda/parsec/parsec/libparsec.so.4(+0xb042f)[0x7fa93d0fc42f]
 29341 [leconte:4113702] [ 8] /home/bouteill/parsec/dplasma/build.cuda/parsec/parsec/libparsec.so.4(parsec_release_dtd_task_to_mempool+0x32)[0x7fa93d0ce596]
 29342 [leconte:4113702] [ 9] /home/bouteill/parsec/dplasma/build.cuda/parsec/parsec/libparsec.so.4(__parsec_complete_execution+0xc6)[0x7fa93d0b7f70]
 29343 [leconte:4113702] [10] /home/bouteill/parsec/dplasma/build.cuda/parsec/parsec/libparsec.so.4(__parsec_task_progress+0x12e)[0x7fa93d0b80ca]
 29344 [leconte:4113702] [11] /home/bouteill/parsec/dplasma/build.cuda/parsec/parsec/libparsec.so.4(__parsec_context_wait+0x2ee)[0x7fa93d0b8c0a]
 29345 [leconte:4113702] [12] /home/bouteill/parsec/dplasma/build.cuda/parsec/parsec/libparsec.so.4(+0x49343)[0x7fa93d095343]
@abouteiller abouteiller added the bug Something isn't working label Feb 7, 2024
@abouteiller
Copy link
Contributor Author

this is somewhat rare but can be reproduced for sure with ctest --repeat until-fail:100

Another variant

dtd_test_task_insertion: /home/bouteill/parsec/dplasma/parsec/parsec/mca/termdet/local/termdet_local_module.c:106: parsec_termdet_local_taskpool_state: Assertion `0' failed.

@abouteiller
Copy link
Contributor Author

I have it on master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant