Skip to content

Conversation

@yedidyakfir
Copy link
Collaborator

Summary

  • Make swarm and chain workflow tasks idempotent to prevent duplicate execution on retries
  • Replace the old Dash-based visualizer with a new React frontend for task visualization
  • Reorganize and expand unit test coverage

Changes

  • Idempotency: Add idempotency guards to swarm tasks including fill_running_tasks, swarm_item_done, swarm_item_failed, and batch item operations
  • Signature model: Refactor signature model to support done and failed status, move task data via messages instead of context
  • Swarm workflows: Update swarm start, item done, and error handling to be idempotent with proper locking
  • Chain workflows: Update chain workflows to pass data via messages and support idempotent execution
  • Frontend: New React-based visualization UI with workflow tabs, graph canvas, and task info panel (replaces old Dash app)
  • API server: FastAPI server for serving signature data to the frontend
  • Tests: Reorganize tests into actions/, creation/, change_status/, idempotency/, publish/, and workflows/ folders with comprehensive coverage

Testing

  • Unit tests for all idempotency scenarios
  • Integration tests for swarm and chain workflows
  • Frontend API client tests

Closes #33

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 144 out of 156 changed files in this pull request and generated 7 comments.

Comments suppressed due to low confidence (1)

docs/documentation/callbacks.md:128

  • The text says the default return field is mageflow_results, but the example model still uses results: str. Update the example to use mageflow_results (or rename the field in the snippet) to match the new default.
    When no field is marked with ReturnValue, the return value of the function will be sent to the field named mageflow_results.
    ```python
    class SuccessMessage(BaseModel):
        results: str  # The return value of the function will be sent here
        field_int: int

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +40 to 43
async def chain_error_task(msg: EmptyModel, ctx: Context):
try:
chain_task_id = msg.model_dump()[CHAIN_TASK_ID_NAME]
task_data = HatchetInvoker(msg, ctx).task_ctx
Copy link

Copilot AI Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

chain_error_task is typed as EmptyModel but then immediately expects chain_task_id to be present in msg.model_dump()[CHAIN_TASK_ID_NAME]. With EmptyModel this will raise KeyError (and it also disagrees with the unit tests which pass ChainCallbackMessage). Consider changing the workflow input model to a message type that includes chain_task_id (e.g. ChainCallbackMessage) and set input_validator for ON_CHAIN_ERROR accordingly so the field is guaranteed to exist.

Copilot uses AI. Check for mistakes.
Comment on lines 43 to 46
if task_id:
current_task = await TaskSignature.get_safe(task_id)
await current_task.done()
task_success_workflows = current_task.activate_success(result)
Copy link

Copilot AI Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TaskSignature.get_safe(task_id) can return None (e.g., if the signature was already removed). await current_task.done() and current_task.activate_success(...) will then raise AttributeError. Add a None check (and decide whether to treat missing signatures as a no-op) before calling methods on current_task.

Copilot uses AI. Check for mistakes.
jobs:
create-github-release:
needs: check-tests
if: github.ref_type == 'tag' && github.event.base_ref == 'refs/heads/main'
Copy link

Copilot AI Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The job condition uses github.event.base_ref, which is typically empty for push events on tags. This will likely cause the release job to be skipped even when a tag is pushed. If you want to ensure the tag points to main, consider removing this check or validating that the tagged commit is contained in main via git merge-base --is-ancestor.

Suggested change
if: github.ref_type == 'tag' && github.event.base_ref == 'refs/heads/main'
if: github.ref_type == 'tag'

Copilot uses AI. Check for mistakes.
@yedidyakfir yedidyakfir merged commit f064f4f into develop Jan 29, 2026
14 checks passed
@yedidyakfir yedidyakfir deleted the feature/33-make-swarm-tasks-idempotent branch January 29, 2026 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make swarm tasks idempotent

2 participants