switch to celery.chain for certain vector_search tasks

[shanbady]

What are the relevant tickets?

Closes https://github.com/mitodl/hq/issues/11848

Description (What does it do?)

This PR makes the contentfile embeddings (and similar) tasks in the vector_search app to use a celery chain instead of a group in order to resolve an issue where qdrant gets overwhelmed when many (100's) of runs are processed at once.

How can this be tested?

I have not found a simple way of seeing this overwhelming of qdrant locally but you can verify that the tasks continue to work as expected using celery.chain:

from vector_search.tasks import *
from learning_resources.models import *

run = LearningResourceRun.objects.first()

remove_run_content_files.apply_async([run.id])
embed_run_content_files.apply_async([run.id])
remove_unpublished_run_content_files.apply_async([run.id])

Additional Context

After this deploys I will try embedding the ~1300 runs we need to as part of the variant course workaround - If it continues being problematic we may need to adjust the throttle further

[github-actions]

OpenAPI Changes

No changes detected

View full changelog

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

[Copilot]

Pull request overview

This PR updates the vector_search Celery task orchestration to reduce Qdrant load spikes when embedding/removing large volumes of content-file vectors, by serializing chunked work via celery.chain and tightening the embeddings task rate limit.

Changes:

• Lower generate_embeddings task rate limit from 300/m to 200/m.
• Switch content-file embedding/removal task fan-out from celery.group (parallel) to celery.chain (sequential).
• Update unit tests to reflect the new Celery canvas primitive.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.

File	Description
vector_search/tasks.py	Adjusts rate limit and changes several content-file task canvases from group to chain to reduce concurrency against Qdrant.
vector_search/tasks_test.py	Updates assertions to expect celery.chain instead of celery.group in the affected tasks.

[shanbady]

leaving this as-is for now (may need to tweak further and potentially make it independant of the settings rate limit

[mbertrand]

LGTM, one minor change suggested which could potentially prevent errors under one condition (no contentfiles) flagged by copilot

[mbertrand]

Add this to remove_unpublished_run_content_files and remove_run_content_files too?

[mbertrand]

👍