-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue with multiple subagents on Summit #2186
Comments
Thanks Matteo! Some notes:
The module env sourcing is nothing we can fix short of including your workaround into the resource config. It's a bit messy, but at least the path seems to be somewhat generic (i.e., stable) (fingers crossed). Finally, the |
another node: can you use BTW: another issue we saw: there is a naming mixup for the bootstrap stages: |
I did the following:
Debugging the agent is difficult for the following reasons:
Analysis of the logs (apparently) produced by prrte seems to indicate that the issue with starting the sub-agents does not depend on a failure to create the DVM. Logs indicated that it is successfully bootstrapped. I did not have the time to test whether the DVM is actually working. |
Thanks for the update, Matteo! Good point about including a comment on the source of created scripts. I also don't think that the startup problem is related to the DVM. I would guess that an environmental issue is at play (again), since identical commands behave diifferently in interactive and non-interactive mode... So another thing you could test: add this to |
I did the following:
Analysis of the logs (apparently) produced by prrte seems to indicate that the issue with starting the sub-agents does not depend on a failure to create the VDM. Logs indicated that it is successfully bootstrapped. I have:
Upon inspection, it become apparent that the We have:
The subagents are now coming up but they are trying to write to a write-only file system (on Summit):
I am going to address that and report back. Meanwhile, here some notes about why I found debugging the Agent time consuming:
|
This is now working. |
Thanks again for digging through this! |
Stack:
Currently, suagent support is broken. Debugging is ongoing, here some preliminary results.
Issue with
prte.py
PRTE sets a logical
cvd_id_mode
on Summit while a physical one on Lassen. The switch to a physicall mode broke PRRTE on Summit. Workaround:issues with
bootstrap_2.sh
:module
is not available on the batch/work nodes thereforemodule load
commands fail.Workaround for #1 and #2 in
src/radical/pilot/configs/resource_ornl.json
:Ongoing
We can successfully run
bootstrap_2.sh
by hand from the batch node but when executed by RP, the subagent(s) does not come online.The text was updated successfully, but these errors were encountered: