Log missing/invalid edx_module_id in content-file API requests

[mbertrand]

What are the relevant tickets?

Closes mitodl/hq#11873

Description (What does it do?)

Logs an error whenever an API request references an edx_module_id that has no backing content, so we get a
signal when AskTIM and other consumers are degraded by missing content.

For every requested id, existence is probed directly against the source of
truth, never inferred from empty search results. Two reasons are
distinguished:

• not_in_db — no ContentFile row (ETL/scrape gap)
• not_in_index — row exists but isn't embedded in Qdrant (embedding gap)

How can this be tested?

• Pick a real edx_module_id from your local data, and also deindex a

  from learning_resources.models import ContentFile
  from vector_search.tasks import *
  
  real_cf = ContentFile.objects.exclude(edx_module_id='').first()
  unindexed_cf = ContentFile.objects.exclude(edx_module_id='').last()
  generate_embeddings([real_cf.run.id], "content_file", False)
  real_edx_module_id = print(real_cf.edx_module_id)
  print(real_edx_module_id )
  
  remove_embeddings([unindexed_cf.id], "content_file")
  unindexed_edx_module_id = unindexed_cf.edx_module_id
  print(unindexed_edx_module_id)  

• Log in as an admin user and try the following URLs. The edx_module_id values will need to be url-encoded.

  REST endpoint — existing id is silent, bogus id logs not_in_db:

  http://open.odl.local:8065/api/v1/contentfiles/?edx_module_id=<REAL_ID>

  non-existent -> empty results AND an error log
  http://open.odl.local:8065/api/v1/contentfiles/?edx_module_id=does-not-exist

  Vector endpoint — only probes when there are no hits:

  non-existent id, no hits -> logs not_in_db
  http://open.odl.local:8065/api/v0/vector_content_files_search/?edx_module_id=does-not-exist

  real id that is in the DB but absent from Qdrant -> logs not_in_index
  http://open.odl.local:8065/api/v0/vector_content_files_search/?edx_module_id=<REAL_UNINDEXED_ID>

Watch the web container logs for lines like:

Missing ContentFile (not_in_db) for edx_module_id=does-not-exist [source=contentfiles_api]

[github-actions]

OpenAPI Changes

3 changes: 0 error, 3 warning, 0 info

View full changelog

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

[Copilot]

Pull request overview

Adds observability for degraded consumers (e.g., AskTIM) by logging (and emitting Sentry events via LoggingIntegration) when API requests reference edx_module_id values that are missing in the DB or missing from the Qdrant index, without breaking search behavior.

Changes:

• Introduces shared logging utilities to record missing ContentFile backing for requested edx_module_id values (not_in_db, not_in_index).
• Instruments the vector content-files search endpoint to probe DB/Qdrant when edx_module_id is provided and the search returns no hits (probe failures are swallowed to preserve endpoint availability).
• Instruments the REST ContentFileFilter.edx_module_id filter to log any requested IDs that have no backing ContentFile row, and adds automated test coverage for both REST and vector behavior.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
vector_search/views.py	Calls the missing-content probe when edx_module_id is provided and the vector search returns zero hits.
vector_search/views_test.py	Adds endpoint-level tests for logging, probe failure isolation, and skipping the probe when hits exist.
vector_search/utils.py	Implements the async probe that checks DB presence and (when applicable) Qdrant index presence via count(exact=True).
vector_search/utils_test.py	Adds unit tests covering not_in_db, not_in_index, silent success, and “unknown collection” skip behavior.
learning_resources/utils.py	Adds log_missing_content_file and present_edx_module_ids helpers used by REST and vector instrumentation.
learning_resources/utils_test.py	Adds a unit test asserting the log template/arguments for log_missing_content_file.
learning_resources/filters.py	Adds LoggedEdxModuleIdFilter and wires it into ContentFileFilter.edx_module_id.
learning_resources/filters_test.py	Adds REST filter tests to ensure missing IDs are logged and present IDs are not.

[shanbady]

approving since it does what it needs to.

I did leave a small comment about performance when using exact=True - however since the edx_module_id is an indexed payload i think this will be minimal.

I also tried manually running this count with exact=True on prod and it was pretty fast