Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: separate celery workers #2940

Closed
wants to merge 89 commits into from
Closed

Conversation

AmineDiro
Copy link
Collaborator

Description

TO TEST thoroughly!

  • Complete refacto of celery worker tasks : new File abstraction, ...
  • Separated core logic from celery worker for better testing
  • Cleaned up unnecessary code block, abstractions ...
  • Fallback for audio files
  • Separated Dockerfiles for API and workers : ( 🔴 for DEV only)
  • Added some tests ...

TODO:

  • Dockerfile for production
  • fix quivr-api and quivr-core versions
  • More unit tests ( syncs, uploads ,etc)
  • @chloedia : some issues with megaparse returning empty page_content on scanned pdfs 🤔 ?

AmineDiro and others added 2 commits August 6, 2024 17:25
# Description

- Complete refacto of workers
- New docker image for worker 

NOTE: 

Syncs are under heavy refacto !

---------

Co-authored-by: chloedia <chloedaems0@gmail.com>
Co-authored-by: Stan Girard <stan@quivr.app>
Co-authored-by: Antoine Dewez <44063631+Zewed@users.noreply.github.com>
Co-authored-by: aminediro <aminedirhoussi@gmail.com>
Co-authored-by: Stan Girard <girard.stanislas@gmail.com>
Co-authored-by: Chloé Daems <73901882+chloedia@users.noreply.github.com>
@AmineDiro AmineDiro changed the title Feat/separate celery workers feat: separate celery workers Aug 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: backend Related to backend functionality or under the /backend directory size:XXL This PR changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants