Description
When the LLM index (AI features) is enabled, documents.tasks.bulk_update_documents can crash with Correspondent matching query does not exist (or DocumentType matching query does not exist) if the referenced correspondent or document type is deleted while the task's update_llm_index pass is running.
The metadata change itself is committed correctly (the foreign key is set to NULL as expected, no orphaned rows remain), but the LLM-index update for the affected documents is aborted, leaving their vector nodes stale until the next document_llmindex run. It is a small unguarded foreign-key access in build_document_node, present in both the latest release (3.0.3) and main.
Background & real-world use case (non-technical)
Paperless-ngx can build an LLM index — a semantic/vector index over your documents that powers the AI features (similar-document suggestions, RAG-based Q&A). Whenever a document changes, its entry in this index is refreshed so the AI stays in sync with the library.
A very common maintenance activity is tidying up the taxonomy — e.g. merging duplicate correspondents (say, "ACME Ltd" and "ACME Limited" into one) or removing a document type you no longer use. When this is done across many documents at once, Paperless queues background jobs that refresh each affected document, including its LLM-index entry.
The problem: if a correspondent or document type is deleted at roughly the same time these background refresh jobs run for documents that still reference it, the LLM-index refresh crashes for those documents. The documents themselves are fine — the correspondent/type is removed cleanly and nothing is lost — but the AI index for those few documents is left slightly out of date until the next index run, and the logs fill with an error traceback.
In short: this hits users who (a) have the AI/LLM index enabled and (b) do bulk taxonomy cleanup. It surfaced here during a large re-classification run that reassigned/merged correspondents and document types across the whole library.
Environment
- Paperless-ngx: 3.0.3 (also reproducible against
main by code inspection — see Notes)
- Deployment: Docker (official
ghcr.io image)
- AI features: LLM index enabled (embedding backend configured)
- Database: PostgreSQL
Traceback (abridged)
[ERROR] [celery.app.trace] Task documents.tasks.bulk_update_documents[…] raised unexpected:
DoesNotExist('Correspondent matching query does not exist.')
...
File "src/documents/tasks.py", line 332, in bulk_update_documents
update_llm_index(rebuild=False)
File "src/paperless_ai/indexing.py", line 378, in update_llm_index
nodes = build_document_node(document, chunk_size=chunk_size)
File "src/paperless_ai/indexing.py", line 184, in build_document_node
if document.correspondent
File ".../django/db/models/fields/related_descriptors.py", line 224, in get_object
...
django.db.models.fields.related_descriptors.RelatedObjectDoesNotExist:
Correspondent matching query does not exist.
The same crash occurs for document_type (indexing.py:187).
Root cause
build_document_node accesses the related objects with an unguarded lazy load:
"correspondent": document.correspondent.name if document.correspondent else None,
"document_type": document.document_type.name if document.document_type else None,
"storage_path": document.storage_path.name if document.storage_path else None,
if document.correspondent triggers a lazy DB fetch by correspondent_id. If that correspondent was deleted between the time the document was read for the index pass and the time build_document_node resolves the relation, the fetch raises RelatedObjectDoesNotExist instead of returning None, and the whole update_llm_index pass then aborts for the remaining documents in that task.
Steps to reproduce
- Enable the LLM index.
- Assign a correspondent (or document type) to one or more documents.
- Trigger a
bulk_edit on those documents (which enqueues bulk_update_documents) and delete that correspondent/document type at around the same time.
- The
bulk_update_documents task raises DoesNotExist in build_document_node.
Impact
- No data loss: the bulk metadata change is committed, FKs are nulled correctly, no orphaned rows.
- The LLM index nodes for the affected documents are not updated until the next
document_llmindex update/rebuild.
- Noisy
ERROR tracebacks during otherwise-normal bulk taxonomy maintenance.
Proposed fix
Either of:
- Re-fetch the document freshly at task start so the instance reflects the committed
NULL FK, or
- Guard the relation access in
build_document_node, treating a missing related object the same as None:
from django.core.exceptions import ObjectDoesNotExist
def _safe_related_name(document, field):
try:
rel = getattr(document, field)
except ObjectDoesNotExist:
return None
return rel.name if rel else NoneOption 2 is minimal and keeps node building resilient regardless of concurrency.
Notes
Description
When the LLM index (AI features) is enabled,
documents.tasks.bulk_update_documentscan crash withCorrespondent matching query does not exist(orDocumentType matching query does not exist) if the referenced correspondent or document type is deleted while the task'supdate_llm_indexpass is running.The metadata change itself is committed correctly (the foreign key is set to
NULLas expected, no orphaned rows remain), but the LLM-index update for the affected documents is aborted, leaving their vector nodes stale until the nextdocument_llmindexrun. It is a small unguarded foreign-key access inbuild_document_node, present in both the latest release (3.0.3) andmain.Background & real-world use case (non-technical)
Paperless-ngx can build an LLM index — a semantic/vector index over your documents that powers the AI features (similar-document suggestions, RAG-based Q&A). Whenever a document changes, its entry in this index is refreshed so the AI stays in sync with the library.
A very common maintenance activity is tidying up the taxonomy — e.g. merging duplicate correspondents (say, "ACME Ltd" and "ACME Limited" into one) or removing a document type you no longer use. When this is done across many documents at once, Paperless queues background jobs that refresh each affected document, including its LLM-index entry.
The problem: if a correspondent or document type is deleted at roughly the same time these background refresh jobs run for documents that still reference it, the LLM-index refresh crashes for those documents. The documents themselves are fine — the correspondent/type is removed cleanly and nothing is lost — but the AI index for those few documents is left slightly out of date until the next index run, and the logs fill with an error traceback.
In short: this hits users who (a) have the AI/LLM index enabled and (b) do bulk taxonomy cleanup. It surfaced here during a large re-classification run that reassigned/merged correspondents and document types across the whole library.
Environment
mainby code inspection — see Notes)ghcr.ioimage)Traceback (abridged)
The same crash occurs for
document_type(indexing.py:187).Root cause
build_document_nodeaccesses the related objects with an unguarded lazy load:if document.correspondenttriggers a lazy DB fetch bycorrespondent_id. If that correspondent was deleted between the time the document was read for the index pass and the timebuild_document_noderesolves the relation, the fetch raisesRelatedObjectDoesNotExistinstead of returningNone, and the wholeupdate_llm_indexpass then aborts for the remaining documents in that task.Steps to reproduce
bulk_editon those documents (which enqueuesbulk_update_documents) and delete that correspondent/document type at around the same time.bulk_update_documentstask raisesDoesNotExistinbuild_document_node.Impact
document_llmindex update/rebuild.ERRORtracebacks during otherwise-normal bulk taxonomy maintenance.Proposed fix
Either of:
NULLFK, orbuild_document_node, treating a missing related object the same asNone:Option 2 is minimal and keeps node building resilient regardless of concurrency.
Notes
mainstill has the unguarded ternary access inbuild_document_node, so upgrading does not resolve this.build_document_nodeappears not to be covered by either.