Support for non-immediate proof maintenance

[PetarMax]

When doing large real-world proofs with a high degree of parallelism, the bandwidth is overtaken by the writing of the proof to disk on every proof iteration, slowing the process considerably.

This PR introduces the ability to write to disk and perform the callback on every maintenance_iterations of the parallel proof, set by default to 32.

EDIT: Actually, perhaps it's a better idea to write the proof every max_workers iterations.

[nwatson22]

Looks good. Can you make it toggleable with a flag so we can choose to do the disk writing/status bar updating every step?

[PetarMax]

I think I prefer to have a separate parameter so that we can decouple it from the number of workers, as it is now. Is there anything else to do in pyk?

[nwatson22]

I think I prefer to have a separate parameter so that we can decouple it from the number of workers, as it is now. Is there anything else to do in pyk?

You could possibly add the same parameter/logic to Prover.advance_proof() and feed the option through to be able to be used with pyk prove, but that could also be done later in a separate PR since this is needed for kontrol now.

[nwatson22]

What is this change for?

[PetarMax]

I have not changed that file manually...

[PetarMax]

I think I prefer to have a separate parameter so that we can decouple it from the number of workers, as it is now. Is there anything else to do in pyk?

You could possibly add the same parameter/logic to Prover.advance_proof() and feed the option through to be able to be used with pyk prove, but that could also be done later in a separate PR since this is needed for kontrol now.

Yes, happy to.

[PetarMax]

Just to say - I am not sure how to test this, because the final outcome is the same, it's just that the proof should be faster. Happy to go along with any ideas in a follow-up PR.