UN-2942 [HOT-FIX] API deployments now properly clean up execution directories after completion

[muhammad-ali-e]

What

• Added missing cleanup call to process_batch_callback_api function in workers/callback/tasks.py
• API deployments now properly clean up both API execution directories and workflow execution directories after completion

Why

• API deployments were not cleaning up execution directories after workflow completion
• ETL workflows had the cleanup logic, but API callback implementation was missing this critical step
• This caused accumulation of execution files on disk over time for API deployments
• The cleanup function _cleanup_execution_resources() already existed and worked correctly for ETL workflows

How

• Added cleanup_result = _cleanup_execution_resources(context) call after UI logs publishing (line 1574-1575)
• The cleanup function already handles API deployments correctly:
  • Skips cache cleanup for API deployments (preserves cache for subsequent requests)
  • Cleans both API execution directory (unstract/api/...) and workflow execution directory (unstract/execution/...)
  • Uses the same proven cleanup logic as ETL workflows

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

No, this PR cannot break existing features:

• This is a 2-line addition that adds missing functionality (bug fix)
• Uses the existing _cleanup_execution_resources() function that is already proven to work for ETL workflows
• No changes to API workflow execution logic, status updates, or pipeline handling
• No changes to cache behavior - cache is still preserved for API deployments as designed
• Only affects post-completion cleanup, which was completely missing before
• The cleanup operation is non-critical and failures are logged without breaking the callback flow
• Backward compatible - API deployments will simply start cleaning up directories (expected behavior)

Database Migrations

• No database migrations required
• No database schema changes

Env Config

• No environment configuration changes required
• No new environment variables added

Relevant Docs

• Related to worker architecture documentation in workers/ARCHITECTURE_DOCUMENTATION.md
• Cleanup logic documented in workers/callback/tasks.py function comments

Related Issues or PRs

• Jira Ticket: UN-2942
• Fixes issue where API deployment execution directories were not being cleaned up

Dependencies Versions

• No dependency version changes
• No new dependencies added

Notes on Testing

Testing Steps:

1. Deploy an API workflow with file processing
2. Check callback worker logs for cleanup messages:

   ℹ️  Cache cleanup skipped for API deployment (cache preserved)
   🧹 Starting directory cleanup for execution xxx (2 directories to clean)
   🧹 Cleaning api directory: unstract/api/...
   ✅ Successfully cleaned up api execution directory
   🧹 Cleaning workflow directory: unstract/execution/...
   ✅ Successfully cleaned up workflow execution directory
   

3. Verify directories are removed after execution completes
4. Verify API functionality still works correctly (cache preserved)
5. Compare with ETL workflow cleanup behavior (should be identical)

Screenshots

N/A - Backend/worker change with no UI impact

Checklist

I have read and understood the Contribution Guidelines.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch fix/UN-2942-HOT-FIX_api-deployment-directory-cleanup

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud