Continue processing

[carlosedubarreto]

Is there a possibility to stop processing and continue later?
I tried forcing the process to stop and them start again, but it usually ends up without cotinuing processing, it seems that it sees that there was something running before and just wraps up.

I'm asking that in case I need to stop the process and if I could conitnue later, or in a case that I get a 'nan" value, if I could change the learning rate and run again not loosing all the previous process that was done

Thanks

[vye16]

There should be checkpointing during optimization, but I will check if there’s a bug. There are multiple stages of optimization so if it’s already finished one of them, it will just skip it and move on to the next one.

…

On Thu, Aug 3, 2023 at 3:22 PM Carlos Barreto ***@***.***> wrote: Is there a possibility to stop processing and continue later? I tried forcing the process to stop and them start again, but it usually ends up without cotinuing processing, it seems that it sees that there was something running before and just wraps up. I'm asking that in case I need to stop the process and if I could conitnue later, or in a case that I get a 'nan" value, if I could change the learning rate and run again not loosing all the previous process that was done Thanks — Reply to this email directly, view it on GitHub <#34>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLOKW2QICU4I66ILBMIVNTXTQQBZANCNFSM6AAAAAA3DMVCRI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[carlosedubarreto]

just to be sure.
for example, it does 4 steps. first it will do the full step 1 them step 2 and so on?

I got the impression that it did in chunks. for exmaple, it splited the frames in "n" parts. it would do the step 1 to 4 in the frist part, then step 1 to 4 in part 2 and so on...

the way you are saying, if I removed the step the it was not finished when I stopped the execution it would continue from the previous step?

It could be enough, I'll study it more.

[vye16]

Yes, in the last step of optimization, we optimize in chunks. We also have some optimization phases to help initialize the main `motion_chunks` phase of optimization. In any of these phases, if the optimization is interrupted, it will resume from the step it last saved a checkpoint at.

…

On Fri, Aug 4, 2023 at 1:57 AM Carlos Barreto ***@***.***> wrote: just to be sure. for example, it does 4 steps. first it will do the full step 1 them step 2 and so on? I got the impression that it did in chunks. for exmaple, it splited the frames in "n" parts. it would do the step 1 to 4 in the frist part, then step 1 to 4 in part 2 and so on... the way you are saying, if I removed the step the it was not finished when I stopped the execution it would continue from the previous step? It could be enough, I'll study it more. — Reply to this email directly, view it on GitHub <#34 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLOKW6PFSUR2H6QHM65T7TXTS2QRANCNFSM6AAAAAA3DMVCRI> . You are receiving this because you commented.Message ID: ***@***.***>

[carlosedubarreto]

So if I remove the last chunk (probably not completed if I manually stopped the process and I had some sort of error), theorically it should work?

I'll give it a try anyway