Skip to content

feat: Separate push Taskworker from pull Taskworker#593

Merged
evanh merged 17 commits intomainfrom
evanh/feat/add-push-mode-workers
Apr 20, 2026
Merged

feat: Separate push Taskworker from pull Taskworker#593
evanh merged 17 commits intomainfrom
evanh/feat/add-push-mode-workers

Conversation

@evanh
Copy link
Copy Markdown
Member

@evanh evanh commented Apr 14, 2026

Moving the push taskbroker to a k8s Deployment means that the worker logic can simplify. Particularly the push worker doesn't need a list of hosts, just a single service host. Instead of having one Taskworker that supports both modes, create separate classes and clients for each.

This meant moving the multiprocess part of the TaskWorker to a separate class, that manages the queues and the various children processes and threads. This class is shared by both the Push worker and the original pull worker.
The push-based taskworker doesn't need all the features of the pull based taskbroker. Create a new simpler client that has a single RPC host to connect to, that the AlloyDB workers can use.

Note that this should be a non-breaking change: the is being moved around, not changed.

Moving the push taskbroker to a k8s Deployment means that the worker logic can simplify.
Particularly the push worker doesn't need a list of hosts, just a single service host. Instead of
having one Taskworker that supports both modes, create separate classes and clients for each.

This meant moving the multiprocess part of the TaskWorker to a separate class, that manages the
queues and the various children processes and threads. This class is shared by both the Push worker
and the original pull worker.
The push-based taskworker doesn't need all the features of the pull based taskbroker. Create a new
simpler client that has a single RPC host to connect t
@evanh evanh requested a review from a team as a code owner April 14, 2026 18:55
Comment thread clients/python/src/taskbroker_client/worker/client.py Outdated
Comment thread clients/python/src/taskbroker_client/worker/worker.py Outdated
Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/taskbroker_client/worker/client.py
Comment thread clients/python/src/taskbroker_client/worker/client.py Outdated
Comment thread clients/python/src/taskbroker_client/worker/worker.py Outdated
Comment thread clients/python/src/taskbroker_client/worker/worker.py Outdated
Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/taskbroker_client/worker/worker.py Outdated
Comment thread clients/python/src/taskbroker_client/worker/client.py
Comment thread clients/python/src/taskbroker_client/worker/client.py Outdated
Comment thread clients/python/src/taskbroker_client/worker/client.py Outdated
Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/examples/cli.py Outdated
Copy link
Copy Markdown
Member

@markstory markstory left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

Comment thread clients/python/src/taskbroker_client/worker/client.py Outdated
Comment thread clients/python/src/taskbroker_client/worker/client.py Outdated
Co-authored-by: Mark Story <mark@mark-story.com>
evanh and others added 4 commits April 20, 2026 15:35
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e3eed9a. Configure here.

Comment thread clients/python/src/taskbroker_client/worker/worker.py
Comment thread clients/python/src/taskbroker_client/worker/client.py Outdated
@evanh evanh merged commit a5e781b into main Apr 20, 2026
23 checks passed
@evanh evanh deleted the evanh/feat/add-push-mode-workers branch April 20, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants