Implement per-page workflow system for sync-threads to overcome API limits #1941

devin-ai-integration · 2025-08-06T06:03:44Z

Implement per-page workflow system for sync-threads to overcome API limits

Summary

This PR implements a new coordinator-based workflow system to solve Cloudflare Workflow API request limitations when syncing user email threads. Instead of processing all pages sequentially in a single workflow instance (which hits API limits), the system now creates separate workflow instances for each page of threads.

Key Changes:

NEW: SyncThreadsCoordinatorWorkflow - orchestrates page-by-page processing
MODIFIED: SyncThreadsWorkflow - now supports both single-page and multi-page modes
UPDATED: Configuration files to register the new coordinator workflow
ENHANCED: Workflow triggering logic with fallback mechanism

The coordinator processes pages sequentially as requested: lists first page → creates page workflow → waits for completion → moves to next page until nextPageToken is null.

Review & Testing Checklist for Human

⚠️ HIGH RISK - Limited local testing due to environment issues. Critical items to verify:

End-to-end workflow execution: Deploy to staging and verify the coordinator actually creates page workflows sequentially and waits for completion before proceeding
API request distribution: Confirm that API calls are properly distributed across multiple workflow instances and stay within Cloudflare's subrequest limits
Error handling scenarios: Test what happens when individual page workflows fail - does the coordinator continue processing remaining pages gracefully?
Configuration deployment: Verify new SYNC_THREADS_COORDINATOR_WORKFLOW bindings are correctly deployed in all environments (dev/staging/prod)
Performance impact: Measure sync times - sequential processing may be significantly slower than the original approach

Recommended Test Plan:

Deploy to staging environment
Trigger sync for a large inbox (>1000 threads)
Monitor workflow execution in Cloudflare dashboard
Verify all pages are processed and threads are synced correctly
Test failure scenarios (network issues, API errors)

Diagram

%%{ init : { "theme" : "default" }}%%
graph TD
    Agent["apps/server/src/routes/agent/index.ts<br/>ZeroDriver.triggerSyncWorkflow()"]:::major-edit
    Coordinator["apps/server/src/workflows/<br/>sync-threads-coordinator-workflow.ts<br/>(NEW FILE)"]:::major-edit
    SyncWorkflow["apps/server/src/workflows/<br/>sync-threads-workflow.ts<br/>SyncThreadsWorkflow"]:::major-edit
    
    EnvTypes["apps/server/src/env.ts<br/>ZeroEnv interface"]:::minor-edit
    MainExports["apps/server/src/main.ts<br/>exports"]:::minor-edit
    WranglerConfig["apps/server/wrangler.jsonc<br/>workflow bindings"]:::minor-edit
    
    Agent -->|"triggers coordinator<br/>instead of direct sync"| Coordinator
    Coordinator -->|"creates page workflow<br/>for each page"| SyncWorkflow
    Coordinator -->|"waits for completion<br/>via polling"| SyncWorkflow
    
    EnvTypes -.->|"defines workflow types"| Agent
    MainExports -.->|"exports new coordinator"| Coordinator
    WranglerConfig -.->|"configures bindings"| Coordinator

    subgraph Legend
        L1[Major Edit]:::major-edit
        L2[Minor Edit]:::minor-edit
        L3[Context/No Edit]:::context
    end

    classDef major-edit fill:#90EE90
    classDef minor-edit fill:#87CEEB  
    classDef context fill:#FFFFFF

Notes

Session: https://app.devin.ai/sessions/4a78ee3f20d74ef6a82ddfcaba8d5dac (requested by Adam @MrgSub)
Environment Issues: Local testing was limited due to TypeScript compilation errors in the dev environment - changes were verified through CI only
Backward Compatibility: Original SyncThreadsWorkflow maintains multi-page mode for existing workflows
Fallback Mechanism: If coordinator workflow creation fails, system falls back to original workflow approach
Polling Implementation: Coordinator uses 1-second polling with 300-attempt timeout (5 minutes) to wait for page workflow completion

⚠️ Critical: This change fundamentally alters how email sync works. Thorough testing in staging environment is essential before production deployment.

- Create SyncThreadsCoordinatorWorkflow to spawn separate workflows per page - Modify SyncThreadsWorkflow to support single-page mode - Update ZeroDriver.triggerSyncWorkflow to use coordinator workflow - Add fallback to original workflow if coordinator fails - Update isSyncing method to check both coordinator and original workflows - Configure new coordinator workflow in wrangler.jsonc for all environments - Export new coordinator workflow from main.ts and add to env types This solves the API request limit issue by distributing requests across multiple workflow instances instead of processing all pages sequentially in a single workflow instance. Co-Authored-By: Adam <github@ajxb.net>

devin-ai-integration · 2025-08-06T06:03:47Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment and CI monitoring

jazzberry-ai · 2025-08-06T06:03:48Z

Bug Report

Name	Severity	Example test case	Description
Incomplete Thread Synchronization in Production	Medium	Enable the thread sync workflow in production. Observe that only the first page of threads is synchronized.	The `SyncThreadsCoordinatorWorkflow` relies on the `THREAD_SYNC_LOOP` environment variable to determine whether to iterate through all pages of threads. This variable is set to `"false"` in the production environment, causing the workflow to only process the first page. This results in incomplete thread synchronization. This setting will cause the coordinator workflow to behave the same as the original workflow, eliminating any gains from using the coordinator. The staging env also has this variable set to `"false"`.

_{Comments? Email us.}

greptile-apps

Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.

coderabbitai · 2025-08-06T06:03:51Z

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Explain this complex logic.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai explain this code block.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and explain its main purpose.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Join our Discord community for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

jazzberry-ai · 2025-08-06T19:05:00Z

Bug Report

Name	Severity	Example test case	Description
`isSyncing` always returns false	Critical	Trigger a sync workflow and check the UI.	The `isSyncing` method in `ZeroDriver` is commented out, causing the UI to always report that the agent is not syncing, even when it is. This prevents users from monitoring sync progress and could lead to data loss.

_{Comments? Email us.}

jazzberry-ai · 2025-08-06T19:09:40Z

Bug Report

Name	Severity	Example test case	Description
Premature termination due to transient listing errors	Medium	Simulate a temporary network issue or API rate limit during the listing of email threads.	The `SyncThreadsCoordinatorWorkflow` prematurely terminates if an error occurs while listing threads and getting the next page token. Instead, it should log the error and retry on the next iteration to recover from transient issues.

_{Comments? Email us.}

jazzberry-ai · 2025-08-06T19:14:39Z

Bug Report

Name	Severity	Example test case	Description
Missing Workflow ID	High	Call `ZeroDriver.triggerSyncWorkflow`.	The coordinator workflow creation will likely fail because of the missing ID.
Empty workflowId string	Low	Simulate an error during workflow creation in `SyncThreadsCoordinatorWorkflow`.	The `workflowId` field in the `pageWorkflowResults` array will be an empty string, leading to incomplete information.

_{Comments? Email us.}

jazzberry-ai · 2025-08-06T19:15:08Z

Bug Report

Name	Severity	Example test case	Description
`isSyncing` always returns false	Critical	Call `isSyncing` method on a `ZeroDriver` instance while a sync process is running.	The `isSyncing` method in the `ZeroDriver` class always returns `false`, which disables the ability to determine if a sync process is currently active. This can lead to concurrent sync processes, overloading the system and causing data inconsistencies.

_{Comments? Email us.}

jazzberry-ai · 2025-08-06T19:16:08Z

An error occured.

_{This error may be due to rate limits. If this error persists, please email us.}

jazzberry-ai · 2025-08-06T19:37:45Z

Bug Report

Name	Severity	Example test case	Description
`isSyncing` always returns `false`	High	Call the `isSyncing` function on a `ZeroDriver` instance while a sync workflow is running. It will return `false`.	The `isSyncing` function in `apps/server/src/routes/agent/index.ts` was modified to always return `false`. This prevents the system from correctly detecting when a sync workflow is in progress.

_{Comments? Email us.}

… clarity and consistency

jazzberry-ai · 2025-08-06T19:40:36Z

Bug Report

Name	Severity	Example test case	Description
Fixed timeout in `SyncThreadsCoordinatorWorkflow`	Medium	Create a connection with a folder containing many emails, such that syncing a single page takes longer than 5 minutes. Observe that the coordinator workflow times out and doesn't process all pages.	The fixed timeout in the coordinator workflow can lead to incomplete syncing if a page takes too long to process.
Misleading `pagesProcessed` in `SyncThreadsWorkflow` (`singlePageMode`)	Low	Run the sync workflow in `singlePageMode`. Observe `result.pagesProcessed` always reports `1`.	The `pagesProcessed` field is not incremented correctly.
Missing `nextPageToken` in `SyncThreadsWorkflow` (`singlePageMode`)	Low	Run the sync workflow in `singlePageMode`. Observe that `nextPageToken` is null.	The `nextPageToken` isn't exposed to the coordinator, which forces an extra API call.
`isSyncing` always returns false	Critical	Trigger a sync workflow and then call `isSyncing`. Observe that it returns `false`.	The `isSyncing` method provides incorrect information about the sync status.
Silent failures in `ZeroDriver.triggerSyncWorkflow`	Medium	Simulate an error during workflow creation (e.g., by invalidating the workflow binding). Trigger a sync workflow and observe that the error is only logged and the user is not notified. Do this such that the fallback also fails.	Silent failures can occur when workflow creation fails.

_{Comments? Email us.}

greptile-apps bot reviewed Aug 6, 2025

View reviewed changes

wip

4b16f3d

working

66c6ee0

MrgSub force-pushed the devin/1754459896-sync-threads-per-page-workflow branch from fda0c3b to 66c6ee0 Compare August 6, 2025 19:37

MrgSub approved these changes Aug 6, 2025

View reviewed changes

Refactor variable declarations in OutlookMailManager to use const for…

6e358d4

… clarity and consistency

MrgSub merged commit b9a5db5 into staging Aug 6, 2025
7 checks passed

MrgSub deleted the devin/1754459896-sync-threads-per-page-workflow branch August 6, 2025 19:44

Uh oh!

Implement per-page workflow system for sync-threads to overcome API limits #1941

Implement per-page workflow system for sync-threads to overcome API limits #1941

Uh oh!

Conversation

devin-ai-integration bot commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Implement per-page workflow system for sync-threads to overcome API limits

Summary

Review & Testing Checklist for Human

Diagram

Notes

Uh oh!

devin-ai-integration bot commented Aug 6, 2025

🤖 Devin AI Engineer

Uh oh!

jazzberry-ai bot commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bug Report

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot commented Aug 6, 2025

Review skipped

Chat

Support

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

Uh oh!

jazzberry-ai bot commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bug Report

Uh oh!

jazzberry-ai bot commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bug Report

Uh oh!

jazzberry-ai bot commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bug Report

Uh oh!

jazzberry-ai bot commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bug Report

Uh oh!

jazzberry-ai bot commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jazzberry-ai bot commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bug Report

Uh oh!

jazzberry-ai bot commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bug Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

devin-ai-integration bot commented Aug 6, 2025 •

edited

Loading

jazzberry-ai bot commented Aug 6, 2025 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)

jazzberry-ai bot commented Aug 6, 2025 •

edited

Loading

jazzberry-ai bot commented Aug 6, 2025 •

edited

Loading

jazzberry-ai bot commented Aug 6, 2025 •

edited

Loading

jazzberry-ai bot commented Aug 6, 2025 •

edited

Loading

jazzberry-ai bot commented Aug 6, 2025 •

edited

Loading

jazzberry-ai bot commented Aug 6, 2025 •

edited

Loading

jazzberry-ai bot commented Aug 6, 2025 •

edited

Loading