Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add micro-batching to batch processor #2734

Closed
RafalSkolasinski opened this issue Dec 3, 2020 — with Board Genius Sync · 0 comments · Fixed by #3301
Closed

add micro-batching to batch processor #2734

RafalSkolasinski opened this issue Dec 3, 2020 — with Board Genius Sync · 0 comments · Fixed by #3301

Comments

Copy link
Contributor

Add micro-batching functionality to batch processor:

  • user able to provide new parameter: micro-batch size
  • batch processor to combine single prediction instances into batch
  • send batch as single request
  • split prediction results into single instances to maintain parity with not micro-batching mode

Benefits:

  • lower network overhead
  • possibility to benefit from vectorized implementations of underlying inference library
@RafalSkolasinski RafalSkolasinski added the triage Needs to be triaged and prioritised accordingly label Dec 3, 2020
@RafalSkolasinski RafalSkolasinski added this to the 1.6 milestone Dec 3, 2020
@RafalSkolasinski RafalSkolasinski added this to To do in 1.6.0 via automation Dec 3, 2020
@RafalSkolasinski RafalSkolasinski self-assigned this Dec 3, 2020
@RafalSkolasinski RafalSkolasinski moved this from To do to In progress in 1.6.0 Dec 3, 2020
@RafalSkolasinski RafalSkolasinski changed the title batch: add micro-batching to batch processor add micro-batching to batch processor Dec 3, 2020
@ukclivecox ukclivecox removed the triage Needs to be triaged and prioritised accordingly label Dec 3, 2020
@ukclivecox ukclivecox added this to To do in Sprint 1 via automation Jan 7, 2021
@ukclivecox ukclivecox removed this from In progress in 1.6.0 Jan 7, 2021
@ukclivecox ukclivecox moved this from To do to In progress in Sprint 1 Jan 7, 2021
@RafalSkolasinski RafalSkolasinski moved this from In progress to To do in Sprint 1 Jan 7, 2021
@RafalSkolasinski RafalSkolasinski added this to To do in Sprint 2 via automation Jan 7, 2021
@RafalSkolasinski RafalSkolasinski removed this from To do in Sprint 2 Jan 7, 2021
@RafalSkolasinski RafalSkolasinski added this to To do in Sprint 2 via automation Jan 7, 2021
@RafalSkolasinski RafalSkolasinski removed this from To do in Sprint 1 Jan 7, 2021
@RafalSkolasinski RafalSkolasinski added this to To do in Sprint 1 via automation Jan 7, 2021
@RafalSkolasinski RafalSkolasinski removed this from To do in Sprint 2 Jan 7, 2021
@ukclivecox ukclivecox removed this from To do in Sprint 1 Jan 14, 2021
@ukclivecox ukclivecox added this to Backlog in Backlog Jan 21, 2021
@axsaucedo axsaucedo changed the title add micro-batching to batch processor OSS-171: add micro-batching to batch processor Apr 26, 2021
@axsaucedo axsaucedo changed the title OSS-171: add micro-batching to batch processor add micro-batching to batch processor Apr 27, 2021
@axsaucedo axsaucedo added this to To do in MLOps Sprint 8 (Ending 29/05/2020) via automation May 24, 2021
@axsaucedo axsaucedo removed this from Backlog in Backlog May 24, 2021
@majolo majolo self-assigned this May 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

3 participants