Why do I have waiting workers on this SWP run?

[RyanConway91]

I am running 420 models with ++sweep_chunk(99999) on 122 workers.
No runs have failed. Why is SWP only using 4 runners when there are 90 runs to complete?

[jtwhite79]

Hmm. I don't see any reason why...anything of note in the rmr file?

[wkitlasten]

I assume all 122 were running at one point (initially) and the resources requested in the submit file are available on the workers? Anything in the condor log files?

On Sat, 15 Oct 2022 at 1:34 AM, RyanConway91 ***@***.***> wrote: I am running 420 models with ++sweep_chunk(99999) on 122 workers. No runs have failed. Why is SWP only using 4 runners when there are 90 runs to complete? [image: image] <https://user-images.githubusercontent.com/25184697/195847509-3a4f23ec-e592-4d6d-8626-77c0dd46aa33.png> — Reply to this email directly, view it on GitHub <#218>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADSJXRFPHPVJAIJACLPVS7DWDFHNTANCNFSM6AAAAAARFGMHQQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- "Perfect spheres are pointless."

[RyanConway91]

Looks like the SWP run did not complete successfully.
End of the .rec file is:

End of the .rmr file is:

My SWP output csv is blank except for the obsnme headers

Looking at the panther_worker.rec file:

It seems like the connection was lost between the master and agents at some point. This even occurred for the agents running on the same server as the master

All 122 agents were initially running and successfully executed model runs. I am transferring files back with every model run. Oddly, results from all 420 runs were transferred back, so it seems like everything got ran despite not being recorded in the rmr or rec?

By condor log files you are referring to something related to the HTCondor right? I'm not using HTCondor (I tried to get it working in the past but couldn't figure it out. I'm on windows machines)

If I set ++sweep_chunk() to something smaller will it right partial results to the swp output csv or is the output all or nothing?

[jtwhite79]

hmmm. Seems like a lot of runs were going long so they were being rescheduled on available workers. But this shouldnt effect what sweep records.

The rec file for the sweep master doesnt have much - its not really needed - so Im not too worried about that. And it looks the worker rec file lost comms with the master at 7:43 but the end of the rmr file was ~5 hrs earlier?

The sweep chunk is basically how many runs to queue up, run, and write out at a time. So a chunk of 100 might be a nice place to start to see if you can get 100 runs done and written the sweep out file...otherwise sweep will wait until all runs have been queued and run (or failed to run) before writing the sweep output file.

[RyanConway91]

I added++overdue_resched_fac(2) and reran with a chunk of 100. PESTPP SWP went all the way through this time. Not sure what happened initially, must have been some local network issue...

Thanks!

[cnicol-gwlogic]

I think I had something similar hapoen recently. In my case, I think it was my virus/security software messing with file locks or something along those lines. Disabled it for the drive I was working from and it did not happen again.