-
Notifications
You must be signed in to change notification settings - Fork 0
fix: parallelize loader upserts #80
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: parallelize loader upserts #80
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR refactors the loader's Qdrant upsert pipeline to parallelize operations through background workers instead of awaiting inline, improving performance by allowing ingestion to continue while batches persist. It also fixes test synchronization and updates version metadata.
- Replaces synchronous task scheduling with bounded background workers for Qdrant upserts
- Updates concurrency test to use deterministic waiting instead of non-blocking queue operations
- Bumps project version from 0.26.53 to 0.26.55 across manifests
Reviewed Changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| mcp_plex/loader/init.py | Refactors upsert pipeline to use worker queue pattern with proper error handling |
| tests/test_loader_logging.py | Fixes test synchronization and adds assertion for new log message |
| pyproject.toml | Updates version metadata to 0.26.55 |
| docker/pyproject.deps.toml | Updates version metadata to 0.26.55 |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Coverage Report
|
||||||||||||||||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 4 out of 5 changed files in this pull request and generated 3 comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
What
Why
Affects
Testing
uv run pytestuv run ruff check .Documentation
https://chatgpt.com/codex/tasks/task_e_68e177c06d8c83289078fcc110cfccb0