Skip to content

Process Drive folder uploads asynchronously in a worker#38

Merged
wailbentafat merged 3 commits into
developfrom
feat/async-drive-folder-import
Apr 11, 2026
Merged

Process Drive folder uploads asynchronously in a worker#38
wailbentafat merged 3 commits into
developfrom
feat/async-drive-folder-import

Conversation

@ademboukabes
Copy link
Copy Markdown
Collaborator

Summary

This PR moves Google Drive folder imports out of the request/response cycle and processes them asynchronously in a dedicated worker.

What changed

  • added a dedicated upload_group_worker
  • added a new NATS subject for upload group import requests
  • changed folder-based upload creation to return immediately after creating the upload group
  • moved heavy Google Drive folder import work to the worker
  • added upload group processing fields:
    • processing_status
    • processed_photo_count
    • failed_photo_count
    • error_message
  • updated upload group response schema to expose processing progress
  • added the migration for upload group processing state
  • added the new worker to run-workers
  • fixed storage_cleaner settings for Pydantic v2
  • fixed worker bootstrap issues by initializing Redis and MinIO properly

Why

Previously, folder-based Drive imports were processed synchronously inside the HTTP request, which made large folder uploads slow and blocking.

Now the API returns quickly, and the import is handled in the background by a worker.

@wailbentafat wailbentafat merged commit ac6e82f into develop Apr 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants