Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

analyzer_manager could stuck on _run_and_wait_for_reducing #667

Closed
shchen-idmod opened this issue Mar 10, 2020 · 1 comment
Closed

analyzer_manager could stuck on _run_and_wait_for_reducing #667

shchen-idmod opened this issue Mar 10, 2020 · 1 comment
Labels
Analyzers bug Something isn't working
Milestone

Comments

@shchen-idmod
Copy link
Collaborator

This is frequently seeing on few bamboo agents.
for example for this bamboo job:
http://idm-bamboo:8085/browse/RIT-ID-207/log
it stuck on test:
test_analyzers_emod_comps.py: test_analyzer_preidmtools_exp case
this test is using experiment_id: 'f48e09d4-acd9-e911-a2be-f0921c167861' with 5 simulations.
from idmtools.log, I should see print this line 5 times:
- Finalizing f58e09d4-acd9-e911-a2be-f0921c167861
instead I only see from 1 simulation out of 5 for above line
And I also expect these few lines in idmtools.log which I did not see.

Finished finalizing results
Destroying analyzers
Cleaning up the cache

last line was
2020-03-09 11:58:36,629.629 /home/idm_bamboo_user/bamboo-agent/xml-data/build-dir/RIT-ID-ITIP3C/idmtools_core/idmtools/analysis/analyze_manager.py:301 _run_and_wait_for_reducing [DEBUG] (24672,(140129637955392) - Finalizing f58e09d4-acd9-e911-a2be-f0921c167861

attached log I scp from bamboo agent:
idmtools.log

Since job hung, I have to kill bamboo build. this test should finished in few seconds.
We seem more and more this behavior in bamboo.
I am not sure what cause this this hang. from log, it seems stuck on _run_and_wait_for_reducing method

@shchen-idmod shchen-idmod added bug Something isn't working Analyzers labels Mar 10, 2020
@shchen-idmod
Copy link
Collaborator Author

shchen-idmod commented Mar 11, 2020

close it. after clean up bamboo machine(clean all docker images and containers), machine free up lot memory and cache. now it seems no stuck on run.

@devclinton devclinton added this to the 1.0 milestone May 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Analyzers bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants