Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi post-processing: non-exclusive par-repair #291

Closed
hugbug opened this issue Oct 11, 2016 · 1 comment

Comments

Projects
None yet
1 participant
@hugbug
Copy link
Member

commented Oct 11, 2016

This issue defines a new feature resulted from discussion in #290.

Currently post-processor works out one item at a time. If an item requires repair that process can take very long. In that time the post-processing queue may grow and many items which do not require repair wait for their turn to be processed. Post-processing steps required for other (non damaged) items may not require that much CPU power (which is used by par-repair) and such items could be possibly processed parallel with the repairing item.

Tasks:

  • refactor code to make par-repair non global for easier management of post-processing steps;
  • make it possible to process more than one item at the same time;
  • create new option to enable/disable multi processing;
  • functional tests to cover new behavior;
  • fix global cout/cerr in par2-module to prevent crashes when executing two par2-instances;
  • options ParPauseQueue, UnpackPauseQueue, PostPauseQueue should work properly.

@hugbug hugbug added the feature label Oct 11, 2016

@hugbug hugbug added this to the v18 milestone Oct 11, 2016

hugbug added a commit that referenced this issue Oct 11, 2016

#291: refactor: made par-checker non-global
and more similar to other post-processing stages.

hugbug added a commit that referenced this issue Oct 11, 2016

#291: multi post-processing
During par-check/repair another pp-item can be post-processed at the
same time, as long as it doesn’t require par-check/repair.

hugbug added a commit that referenced this issue Oct 11, 2016

#291: new option "ParExclusive"
to configure simultaneous post-processing of a second item if currently
performing par-check/repair for the first item in pp-queue

hugbug added a commit that referenced this issue Oct 11, 2016

hugbug added a commit that referenced this issue Oct 12, 2016

#291: non-global cout/cerr in par2-module
Fixed global cout/cerr in par2-module to prevent crashes when executing
multiple par2-instances.

hugbug added a commit that referenced this issue Oct 12, 2016

#291: smoother transition between pp-stages
without intermediate “pp-queued” state shown in web-interface

hugbug added a commit that referenced this issue Oct 13, 2016

#291: new option "PostStrategy" to configure multi-post-processing
instead of option “ParExclusive”.

hugbug added a commit that referenced this issue Oct 14, 2016

#291: tweaking balanced pp-startegy
Other job can be run only during repair stage but not during
verification stage.

hugbug added a commit that referenced this issue Oct 15, 2016

#291: temporary pause during post-processing
Options ParPauseQueue, UnpackPauseQueue, PostPauseQueue work properly.
@hugbug

This comment has been minimized.

Copy link
Member Author

commented Oct 15, 2016

The implementation has gone beyond the original idea. With new option PostStrategy the multi-processing of pp-items can be configured in four ways:

screen shot 2016-10-15 at 13 04 26

hugbug added a commit that referenced this issue Oct 15, 2016

hugbug added a commit that referenced this issue Oct 15, 2016

#291: distinguishing pp-stages par-renaming and rar-renaming
to better handling temporary pause.

@hugbug hugbug closed this Oct 15, 2016

hugbug added a commit that referenced this issue Oct 15, 2016

hugbug added a commit that referenced this issue Oct 18, 2016

hugbug added a commit that referenced this issue Oct 18, 2016

hugbug added a commit that referenced this issue Dec 27, 2016

hugbug added a commit that referenced this issue Oct 9, 2017

#291: refactor: made par-checker non-global
and more similar to other post-processing stages.

hugbug added a commit that referenced this issue Oct 9, 2017

#291: multi post-processing
During par-check/repair another pp-item can be post-processed at the
same time, as long as it doesn’t require par-check/repair.

hugbug added a commit that referenced this issue Oct 9, 2017

#291: new option "ParExclusive"
to configure simultaneous post-processing of a second item if currently
performing par-check/repair for the first item in pp-queue

hugbug added a commit that referenced this issue Oct 9, 2017

#291: non-global cout/cerr in par2-module
Fixed global cout/cerr in par2-module to prevent crashes when executing
multiple par2-instances.

hugbug added a commit that referenced this issue Oct 9, 2017

#291: smoother transition between pp-stages
without intermediate “pp-queued” state shown in web-interface

hugbug added a commit that referenced this issue Oct 9, 2017

hugbug added a commit that referenced this issue Oct 9, 2017

#291: tweaking balanced pp-startegy
Other job can be run only during repair stage but not during
verification stage.

hugbug added a commit that referenced this issue Oct 9, 2017

#291: temporary pause during post-processing
Options ParPauseQueue, UnpackPauseQueue, PostPauseQueue work properly.

hugbug added a commit that referenced this issue Oct 9, 2017

hugbug added a commit that referenced this issue Oct 9, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.