Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve TaskHandler polling strategy #97

Open
gardusig opened this issue Jul 12, 2022 · 1 comment
Open

Improve TaskHandler polling strategy #97

gardusig opened this issue Jul 12, 2022 · 1 comment
Assignees
@gardusig
Labels
enhancement New feature or request

Comments

@gardusig
Copy link
Contributor

  • Add new parameter for batchSize
  • Each worker will have a standalone subprocess to poll indefinitely
    • For each polled task, start a new subprocess to execute and update it
@gardusig gardusig mentioned this issue Jul 12, 2022
Closed
@gardusig gardusig added the enhancement New feature or request label Jul 12, 2022
@hebrd
Copy link
Contributor

hebrd commented Jul 28, 2022

It would be great if subprocess count for each task could be configured independently, like TaskThreadCount in JAVA client

@gardusig gardusig self-assigned this Aug 6, 2022
@gardusig gardusig linked a pull request Sep 1, 2022 that will close this issue
Added batch polling and refactored worker parallelization #130
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gardusig gardusig

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Added batch polling and refactored worker parallelization conductor-sdk/conductor-python
2 participants
@gardusig @hebrd