removing unused payload fields

[shanbady]

What are the relevant tickets?

Closes https://github.com/mitodl/hq/issues/10782

Description (What does it do?)

This PR removes unnecessary fields from the payload index in Qdrant. clients in mit-learn and learn-ai that call the contentfile vector endpoint only use a few fields to filter by. In Qdrant - each field we decide to add to the payload index introduces significant storage and memory requirements in addition to performance hits since it needs to maintain new HNSW graphs so we should be intentional about what is actually added there.

This PR also contains a small bugfix for an issue where the learning resource responses were not sorted by score

How can this be tested?

On the live performance side of things I have validated it on the RC qdrant cluster.

1. checkout this branch
2. restart celery
3. run python manage.py generate_embeddings --courses
4. check the contentfiles collection info in your local qdrant dashboard and look at the "payload" list - ensure that the fields line up with the fields configured in the constants file

Checklist:

• Double-check that there are no calls being made to our vector search endpoint /api/v0/vector_content_files_search/ that uses any of the removed fields in this PR as a query parameter.

[Copilot]

Pull request overview

This PR aims to reduce Qdrant storage/memory overhead by removing unused payload indexes for the content_files embeddings collection, keeping only the fields intended for frequent filtering/faceting.

Changes:

• Removed multiple fields from QDRANT_CONTENT_FILE_INDEXES to reduce payload index footprint.
• Added an inline note to encourage intentional selection of indexed fields.

[github-actions]

OpenAPI Changes

5 changes: 0 error, 5 warning, 0 info

View full changelog

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

[abeglova]

👍