Skip to content

Celery split for callbacks#1335

Merged
muhammad-ali-e merged 3 commits into
mainfrom
celery-split-for-callbacks
Jun 3, 2025
Merged

Celery split for callbacks#1335
muhammad-ali-e merged 3 commits into
mainfrom
celery-split-for-callbacks

Conversation

@muhammad-ali-e

@muhammad-ali-e muhammad-ali-e commented Jun 3, 2025

Copy link
Copy Markdown
Contributor

What

  • Split the Celery workers into two dedicated sets: one for processing file batches and another for handling callback tasks after batch completion.
  • Configured separate queues and worker apps for file processing and callbacks.
  • Adjusted Celery settings to optimize chord unlock retry intervals specifically for callback workers.

Why

  • To improve scalability by allowing independent scaling of file processing and callback workloads.
  • To reduce callback task retry frequency and resource contention by dedicating specific workers.

How

  • Created separate Celery apps or worker configurations for file processing and callback tasks.
  • Routed file batch processing tasks to the file-processing queue and callback tasks to the callback queue.
  • Tuned Celery chord unlock retry intervals

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

  • No, this PR only separates workers and queues but preserves existing task logic and workflows.

Database Migrations

Env Config

Relevant Docs

Related Issues or PRs

Dependencies Versions

Notes on Testing

Screenshots

Checklist

I have read and understood the Contribution Guidelines.

vishnuszipstack
vishnuszipstack approved these changes Jun 3, 2025
View reviewed changes

@vishnuszipstack vishnuszipstack left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@muhammad-ali-e added a NIT comment. everything else looks okay to me.

Comment thread backend/workflow_manager/workflow_v2/file_execution_tasks.py
chandrasekharan-zipstack
chandrasekharan-zipstack reviewed Jun 3, 2025
View reviewed changes
Comment thread backend/backend/workers/file_processing_callback/constants.py
chandrasekharan-zipstack
chandrasekharan-zipstack reviewed Jun 3, 2025
View reviewed changes
Comment thread backend/workflow_manager/workflow_v2/file_execution_tasks.py
chandrasekharan-zipstack
chandrasekharan-zipstack approved these changes Jun 3, 2025
View reviewed changes

@chandrasekharan-zipstack chandrasekharan-zipstack left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@muhammad-ali-e approved it but left some comments, please check

@github-actions

github-actions Bot commented Jun 3, 2025

Copy link
Copy Markdown
Contributor
filepath function $$\textcolor{#23d18b}{\tt{passed}}$$ SUBTOTAL
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_logs}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_cleanup}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_cleanup\_skip}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_client\_init}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_image\_exists}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_image}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config\_without\_mount}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_run\_container}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_image\_for\_sidecar}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_sidecar\_container}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$ $$\textcolor{#23d18b}{\tt{11}}$$ $$\textcolor{#23d18b}{\tt{11}}$$

@sonarqubecloud

sonarqubecloud Bot commented Jun 3, 2025

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@muhammad-ali-e muhammad-ali-e merged commit a59d8a8 into main Jun 3, 2025
6 checks passed
@muhammad-ali-e muhammad-ali-e deleted the celery-split-for-callbacks branch June 3, 2025 10:56
pk-zipstack pushed a commit that referenced this pull request Aug 20, 2025
@muhammad-ali-e
Celery split for callbacks (#1335)
73e0e02 
* splitting celery tasks for chord callbacks

* add celery retry value as env

* updated compose override file to easy to use of workers in single worker-callback
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@johnyrahul johnyrahul johnyrahul approved these changes
@chandrasekharan-zipstack chandrasekharan-zipstack chandrasekharan-zipstack approved these changes
@vishnuszipstack vishnuszipstack vishnuszipstack approved these changes
@hari-kuriakose hari-kuriakose Awaiting requested review from hari-kuriakose

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@muhammad-ali-e @johnyrahul @chandrasekharan-zipstack @vishnuszipstack