Skip to content

Worker check runner container status#1342

Merged
muhammad-ali-e merged 12 commits into
mainfrom
worker-check-runner-container-status
Jun 5, 2025
Merged

Worker check runner container status#1342
muhammad-ali-e merged 12 commits into
mainfrom
worker-check-runner-container-status

Conversation

@muhammad-ali-e

@muhammad-ali-e muhammad-ali-e commented Jun 5, 2025

Copy link
Copy Markdown
Contributor

What

  • Added support for polling container execution status from worker.
  • Implemented docker client methods to get pod status and remove container safely.
  • Moved ToolExecutionTracker status handling into the worker from runner.
  • Integrated FileExecutionTracker to track all stage and container names
  • Added callback-worker service to the Docker Compose file (was missing earlier).

Why

  • Polling from the worker allows us to asynchronously check container status until it completes.
  • Simplifies lifecycle handling for main and sidecar containers.

How

  • Added get_container_status and remove_container_by_name methods in the docker client.

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

  • No, Changes only in workflow execution

Database Migrations

  • No

Env Config

backend

  • FILE_EXECUTION_TRACKER_TTL_IN_SECOND
  • FILE_EXECUTION_TRACKER_COMPLETED_TTL_IN_SECOND
  • MAX_RUNNER_POLLING_WAIT_SECONDS
  • RUNNER_POLLING_INTERVAL_SECONDS

Runner

  • FILE_EXECUTION_TRACKER_TTL_IN_SECOND

Relevant Docs

Related Issues or PRs

Dependencies Versions

Notes on Testing

Screenshots

Checklist

I have read and understood the Contribution Guidelines.

ritwik-g
ritwik-g reviewed Jun 5, 2025
View reviewed changes
Comment thread unstract/tool-sandbox/src/unstract/tool_sandbox/helper.py
johnyrahul
johnyrahul reviewed Jun 5, 2025
View reviewed changes
Comment thread runner/src/unstract/runner/runner.py
johnyrahul
johnyrahul reviewed Jun 5, 2025
View reviewed changes
Comment thread unstract/core/src/unstract/core/tool_execution_status.py
@github-actions

github-actions Bot commented Jun 5, 2025

Copy link
Copy Markdown
Contributor
filepath function $$\textcolor{#23d18b}{\tt{passed}}$$ SUBTOTAL
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_logs}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_cleanup}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_cleanup\_skip}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_client\_init}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_image\_exists}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_image}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config\_without\_mount}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_run\_container}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_image\_for\_sidecar}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_sidecar\_container}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$ $$\textcolor{#23d18b}{\tt{11}}$$ $$\textcolor{#23d18b}{\tt{11}}$$

@sonarqubecloud

sonarqubecloud Bot commented Jun 5, 2025

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@muhammad-ali-e muhammad-ali-e merged commit 730f79e into main Jun 5, 2025
5 checks passed
@muhammad-ali-e muhammad-ali-e deleted the worker-check-runner-container-status branch June 5, 2025 14:20
pk-zipstack pushed a commit that referenced this pull request Aug 20, 2025
@muhammad-ali-e
Worker check runner container status (#1342)
71ea308 
* splitting celery tasks for chord callbacks

* add celery retry value as env

* updated compose override file to easy to use of workers in single worker-callback

* polling for container or pod status check from worker

* minor change in sample env value

* condition to avoid re processing the same file execution id

* minor changes in sample env

* fixed status updation of file execution tracker while duplicate task

* removed duplicate service from docker compose
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ritwik-g ritwik-g ritwik-g approved these changes
@johnyrahul johnyrahul johnyrahul approved these changes
@chandrasekharan-zipstack chandrasekharan-zipstack Awaiting requested review from chandrasekharan-zipstack
@hari-kuriakose hari-kuriakose Awaiting requested review from hari-kuriakose
@vishnuszipstack vishnuszipstack Awaiting requested review from vishnuszipstack

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@muhammad-ali-e @ritwik-g @johnyrahul