Staggering and "infinite loop"

[Astelith]

I tried to manual staggering plot creation, my system, the 3975WR (32c/ 64t) with 256GB Ram, 6x 980 Pro Raid 0 (even overkill with this plotter), can sustain 3 plots at the same time if I start the new one right after Phase 1, do you think is possible to create something like swar plot manager? so to automate the spawn of a new plot after phase 1? And run it until we decide to stop? Thanks

[BitSec01]

@Astelith You can edit swars plotting manager to work with this. It's written in python so shouldn't be too hard. Only things that are going to be a hassle are rewriting the checks for when phases are done. But this plotter gives very good feedback to check when it's done etc

[Astelith]

Unfortunately I have no clue where to start, I'm really noob at this :D

[madMAx43v3r]

I will implement that yes, infinite / bounded loop, plus optional stagger of 2 or 3.

[digitalscream]

A better solution might be to just wrap it in a BASH script rather than expending dev resources on this. That way, you can use taskset to force it to just use cores on a specific CPU to avoid NUMA issues, and spin the final copy out to another process so it doesn't wait for it to complete.

Literally a 5 minute job, and I'm a guy who's crap at BASH ;)

Admittedly, this is Linux-only, but I'm sure some clever soul could do the same with Windows.

Also, from what I've noticed of running two in parallel (four comes next, when my new SAS disks are here), you can actually fire them off 10 minutes apart, and over the course of 24hrs they get faster as they stagger themselves (there's some interaction due to overall system load, I think, even when the cores are specified to prevent collision), and then reach an equilibrium point.

[fiveangle]

Came here to say same... ph1 can nearly effectively use all cores assigned, but ph2,3,4 are not quite as optimal, so launching next parallel job after ph1 completion of last-launched job has so far shown to be pretty ideal.

What would be even more ideal is to specify number of parallel jobs from 2-3 (or maybe arbitrary, but I think most systems would fall into max 2-3 due to increasing IOWait at higher parallel processes since ph2+3+4 times shouldn't ever extend beyond 2x ph1 time). Then, if no thread count is specified by the user, set thread count automatically. I've found this formula to work generally well as the starting point (oversubscribed) for setting initial thread count on my systems, then typically work down in threads from there:

(2*jobs-1/jobs)*physical_cores

Starting jobs with that number of threads, run parallel_jobs+1 number of jobs, observing completion speed of last parallel_jobs number of jobs, then decrement thread-count and observe resulting plot times, tabulating only the last parallel_jobs number of final plot times (essentially tossing out the switch-over job at the beginning), then do again to attain the hysteresis of last 2 same-direction increments for all parallel (staggered) jobs, which in turn will determine the direction of next increment... if plot times have dropped the last 2 increments, increment threads on the next parallel_plots number of jobs in the same direction. If plot times have increased the last 2 same-direction increments, increment threads in the opposite direction. This way, the number of threads assigned to each job will ultimately find optimal thread-count/job given the system it's running on and simply hover at that optimal point. This will also self-compensate for many system changes, most notably IOWait times increasing due to SSD conditioning after all NAND dies have been cycled through and are approaching steady-state.

I've hacked scripts to do all of this parallel launching, but continue to set thread-count via this process manually, but automating it would take this plotter to the next level of plug-n-play optimization.

Great work on your pipelining BTW. Looking forward to results on investigations of what small bits are different than the official plotter, and if those small changes affect POS lookups at all. Like you, I am already pretty much maxed on storage plotted with the official plotter, but I always find optimizing systems fascinating. Where were your genius optimizations 3 months ago ;)

But I have to say, even in its current simple state, implementing your plotter I was able to optimize to 8% more plots/day than my already-top-marks-plot-times/cpu-PassMark by staggering with the official plotter (see @fiveangle systems on storage_jm's community plotting spreadsheet). All with only a few hours of optimization rather than a few weeks, and with << less flash storage resources. Dynamite work ! 💥❤️