RP (0.35.1) failing on archer #780
Comments
|
This is reproducible. All attempts since the failed build have lead to the same error. |
|
ARCHER admins removed the default python module. |
|
I tried that commit for rp. devel for saga and radical utils. I still see the same error. client verbose: agent_0.err: bootstrap_1.err : |
Hmmm, this is a rather different error, isnt it? How do you configure the mongodb to use? |
|
I set Any reason why database_name is being deprecated ? |
For easier parsing, and for cases like this :) |
|
This is the first time I am seeing/noticing the warning :) But is the parameter deprecated or not supported anymore ? Should I set the database_name along with the database_url ? |
|
Yes, please set the database name as part of the URL... |
Technically you already did, as the "radicalpilot" part of the url is already the database name :-) |
|
Hmm.. yes ! Didn't even realize I used the extasy db url :-| Running it again now. |
|
client verbose remains same. agent_0.err: bootstrap_1.err agent.out: session created as follows: |
|
bootstrap_1.out is long, but can you post it somewhere, or attach it here? Thanks! |
|
bootstrap_1.out: https://gist.github.com/vivek-bala/84da5f9cdb47576339a9 |
|
Vivek, could you please change |
|
In now committed what I think is a fix in the branch |
|
Do I install devel saga and ru with fix/issue_780 ? |
|
Yes, its similar to devel modulo the fix. (I just merged devel back into it) |
|
Ok, I don't get the forwarding error anymore. I get the module error. Should I be testing the devel branch now ? (fixes for both mongodb+archer module changes) |
|
Did you pull after my message? |
|
I think so. I used the fix_issue_780 branch of rp. |
|
That should have the module fix. |
|
Also the number of files on the agent side are lesser. As in there are no agent_0.err or .out files. Is this expected ? |
|
Ok, looks like I might have pulled before the merge. I pulled again now, I see some changes. Wil give it another go |
|
Now I get, agent_0.out: agent_0.err: |
This was discussed in yesterdays meeting. You need to allocate enough nodes to allow for an agent node. On ARCHER this means at least 48 cores. (I think you now have 24) |
|
Ok, I guess then we can't use the short queue (24 core limit). I'll try again on the standard one. |
|
Per http://www.archer.ac.uk/documentation/user-guide/batch.php#sec-5.13 |
|
When I use the short queue with 48 cores, I got : agent_0.out: So I thought that 24 cores was the limit for "short". I get the same with the standard queue. Not sure why this is the case. |
|
Oh man, what a rabbithole. Educated guess, you don't have Can you do: If it works then I know what the problem is. |
|
Yup, now works with the profile variable. There seems to be some extra callbacks in the client side of the type: This is not in the executing script and I am not sure where this is coming from. But seems to take a lot of time (just for this callback). Client verbose: |
Ok, thanks. I will take care of that.
The dots are not callbacks, but are coming from the demo facility. That is fixed in RU feature/demo. |
Okk. The units took quite a while to complete (sleep 0). It seemed that these dots were produced as part of some loop. |
|
They come from the unit_manager.wait(), one dot per completed CU. |
|
Ok, you should now be able to run without RADICAL_PILOT_PROFILE. |
Merged
|
I've created a PR for the fixes, Andre can look at it and then roll a new RC. |
|
ps. You have 30 min left to get into the short queue until you'll have to wait until Monday! ;) |
|
Pulled and tried it again. I now get the following error: agent_0.err: |
|
Pilot goes from PendingActive to Failed. |
|
Its impressive the number and variety of problems that you run into ... This seems unrelated though. Are the credentials (still) valid and/or can you use the same credentials to run on a different resource? |
|
I know. At every run, I am thinking "now what's it going to be !" :) credentials of the mongodb right ? They seem to be ok. I ran the getting_started_local example with the same url. |
|
Im not really suspecting the agent here, especially given that you got past this point multiple times. |
FYI: ARCHER and RP are not a good match currently. Probably filesystem related. To be continued. |
|
Ok, this time the execution was successful. No changes to the script in all 3 trials. |
|
This will be a fun tutorial :/ |
Do we have any data on the reliability of "our" MongoDB vs the Extasy one? |
|
No - but I somehow doubt that this was an issue. I share your sentiments on the Archer FS though -- this is incredibly slow at times... |
I haven't had an issues with the extasy mongodb yet. All my experiments and tests (even some rp tests) are using the extasy url. Not sure if I have any quantitative data for that. |
|
Such intuition is good enough for now. |
|
Can we close this ticket thus? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I was able to RP scripts on archer this morning, but with no changes from the client side it now seems to fail. Could be some changes on archer (?).
Client side verbose:
Successful build: http://ci.radical-project.org/job/ExTASY-0.1/EXTASY_BRANCH=0.1,EXTASY_HOST=archer,EXTASY_WORKFLOW=cocoamber,PYTHON=System-CPython-2.7/lastBuild/console
Failed/Aborted build: http://ci.radical-project.org/job/ExTASY-0.1/EXTASY_BRANCH=0.1,EXTASY_HOST=archer,EXTASY_WORKFLOW=gromacslsdmap,PYTHON=System-CPython-2.7/121/console
agent.err:
agent.out:
The text was updated successfully, but these errors were encountered: