fix(dataset): block dataset deletion while documents are actively indexing

[manan-tech]

Fixes #38522
Related: #38518

Summary

DatasetService.delete_dataset fires dataset_was_deleted and removes the dataset row with no check for in-flight indexing. Deleting a dataset mid-ingest races the running indexing pipeline against clean_dataset_task: the pipeline keeps writing segments/chunks/vectors for rows the cleanup just removed, and its long-lived transactions can block the cleanup itself. #38518 reports the fallout at production scale — ~2.4M orphaned document_segments, ~830k child_chunks, and ~34 GB of orphaned pgvector tables.

Fix

delete_dataset now checks for actively indexing documents before doing anything, and raises a new DatasetIndexingInProgressError when found. Both delete endpoints (console and service API) map it to a 409 with a clear message, exactly like the existing DatasetInUseError handling on the same endpoints:

if DatasetService._has_documents_indexing(dataset.id, session):
    raise DatasetIndexingInProgressError(...)

The guard matches a document when:

• indexing_status ∈ DocumentService._INDEXING_STATUSES (parsing / cleaning / splitting / indexing — the same set the document list uses for its "indexing" display status), and
• is_paused IS NOT true

Both conditions are deliberate:

• Paused documents don't block deletion. Pausing is the escape hatch: if an indexing run is stuck (e.g., crashed worker), the user pauses the document(s) from the console and deletes. Without this exclusion a stuck run would block deletion forever.
• waiting (queued) documents don't block deletion. A queued task no-ops once the dataset rows are gone (_document_indexing loads the dataset first and returns if missing), so queued documents can't produce orphaned writes. Blocking on them would also strand console users — the UI has no pause control for queued documents (the pause button appears only for the four in-flight statuses).

#38522 sketches two options: pause-and-await, or reject with a clear error. This implements the reject option. Pause-and-await inside a synchronous HTTP handler means unbounded blocking of a web worker, and is_paused checkpoints are honored by the legacy indexing runner but not by the RAG-pipeline ingestion path that produced #38518 — so awaiting wouldn't reliably close the race anyway. Rejecting closes it at the entry point regardless of ingestion path.

The check-then-delete window (a document starting to index between the check and the commit) remains, as acknowledged in the issue; closing it fully needs dataset-level locking across all status-mutating paths and is out of scope here. #38519 (cleanup robustness) complements this guard for anything that slips through.

Changes

• services/dataset_service.py — _has_documents_indexing helper + guard in delete_dataset
• services/errors/dataset.py — DatasetIndexingInProgressError
• controllers/console/datasets/{error,datasets}.py, controllers/service_api/dataset/{error,dataset}.py — 409 mapping on both delete endpoints, service-API OpenAPI docs updated with the new 409 cause
• Frontend: no change needed — the shared fetch layer already toasts non-2xx message bodies on this endpoint (same path dataset_in_use uses today)

Testing

• New service-level unit tests: guard raises and does not signal/delete when a document is actively indexing; deletion proceeds when documents are paused/completed; guard query shape (in-flight statuses only, is_paused IS NOT true, EXISTS on documents.dataset_id)
• New controller tests for the 409 mapping: console (unit) and service API (containers integration, mirroring the existing test_delete_dataset_in_use)
• tests/unit_tests/services/test_dataset_service_dataset.py: 83 passed · tests/unit_tests/controllers/console/datasets/test_datasets.py: 71 passed
• ruff check / ruff format: clean · pyrefly check on all changed source files: 0 errors

[github-actions]

Pyrefly Diff

base → PR

--- /tmp/pyrefly_base.txt	2026-07-10 13:00:52.768681388 +0000
+++ /tmp/pyrefly_pr.txt	2026-07-10 13:00:43.786640538 +0000
@@ -7251,7 +7251,7 @@
 ERROR Argument `scoped_session[flask_sqlalchemy.session.Session]` is not assignable to parameter `session` with type `sqlalchemy.orm.session.Session` in function `services.data_migration.import_service.MigrationImportService.import_package` [bad-argument-type]
    --> tests/unit_tests/services/data_migration/test_import_service.py:999:49
 ERROR Argument `str` is not assignable to parameter `indexing_technique` with type `Literal['economy', 'high_quality']` in function `services.entities.knowledge_entities.rag_pipeline_entities.KnowledgeConfiguration.__init__` [bad-argument-type]
-   --> tests/unit_tests/services/dataset_service_test_helpers.py:447:28
+   --> tests/unit_tests/services/dataset_service_test_helpers.py:448:28
 ERROR Unexpected keyword argument `workspace_id` in function `services.enterprise.enterprise_service.DefaultWorkspaceJoinResult.__init__` [unexpected-keyword]
    --> tests/unit_tests/services/enterprise/test_enterprise_service.py:264:40
 ERROR Unexpected keyword argument `workspace_id` in function `services.enterprise.enterprise_service.DefaultWorkspaceJoinResult.__init__` [unexpected-keyword]
@@ -7545,35 +7545,35 @@
 ERROR Object of class `FromClause` has no attribute `create` [missing-attribute]
    --> tests/unit_tests/services/test_credit_pool_service.py:144:5
 ERROR Argument `SimpleNamespace` is not assignable to parameter `account` with type `Account` in function `services.dataset_service.DatasetService.create_empty_dataset` [bad-argument-type]
-   --> tests/unit_tests/services/test_dataset_service_dataset.py:348:61
+   --> tests/unit_tests/services/test_dataset_service_dataset.py:349:61
 ERROR Argument `SimpleNamespace` is not assignable to parameter `account` with type `Account` in function `services.dataset_service.DatasetService.create_empty_dataset` [bad-argument-type]
-   --> tests/unit_tests/services/test_dataset_service_dataset.py:373:25
+   --> tests/unit_tests/services/test_dataset_service_dataset.py:374:25
 ERROR Argument `SimpleNamespace` is not assignable to parameter `account` with type `Account` in function `services.dataset_service.DatasetService.create_empty_dataset` [bad-argument-type]
-   --> tests/unit_tests/services/test_dataset_service_dataset.py:417:25
+   --> tests/unit_tests/services/test_dataset_service_dataset.py:418:25
 ERROR Argument `SimpleNamespace` is not assignable to parameter `dataset` with type `Dataset` in function `services.dataset_service.DatasetService._update_pipeline_knowledge_base_node_data` [bad-argument-type]
-   --> tests/unit_tests/services/test_dataset_service_dataset.py:728:70
+   --> tests/unit_tests/services/test_dataset_service_dataset.py:729:70
 ERROR Argument `SimpleNamespace` is not assignable to parameter `dataset` with type `Dataset` in function `services.dataset_service.DatasetService._update_pipeline_knowledge_base_node_data` [bad-argument-type]
-   --> tests/unit_tests/services/test_dataset_service_dataset.py:738:70
+   --> tests/unit_tests/services/test_dataset_service_dataset.py:739:70
 ERROR Argument `SimpleNamespace` is not assignable to parameter `dataset` with type `Dataset` in function `services.dataset_service.DatasetService._update_pipeline_knowledge_base_node_data` [bad-argument-type]
-   --> tests/unit_tests/services/test_dataset_service_dataset.py:777:70
+   --> tests/unit_tests/services/test_dataset_service_dataset.py:778:70
 ERROR Argument `SimpleNamespace` is not assignable to parameter `dataset` with type `Dataset` in function `services.dataset_service.DatasetService._update_pipeline_knowledge_base_node_data` [bad-argument-type]
-   --> tests/unit_tests/services/test_dataset_service_dataset.py:800:74
+   --> tests/unit_tests/services/test_dataset_service_dataset.py:801:74
 ERROR Object of class `FakeAccount` has no attribute `current_tenant_id` [missing-attribute]
-   --> tests/unit_tests/services/test_dataset_service_dataset.py:879:9
+   --> tests/unit_tests/services/test_dataset_service_dataset.py:880:9
 ERROR Object of class `FakeAccount` has no attribute `current_tenant_id` [missing-attribute]
-   --> tests/unit_tests/services/test_dataset_service_dataset.py:919:9
+   --> tests/unit_tests/services/test_dataset_service_dataset.py:920:9
 ERROR Argument `None` is not assignable to parameter `embedding_model_provider` with type `str` in function `tests.unit_tests.services.dataset_service_test_helpers.DatasetServiceUnitDataFactory.create_dataset_mock` [bad-argument-type]
-   --> tests/unit_tests/services/test_dataset_service_dataset.py:996:38
+   --> tests/unit_tests/services/test_dataset_service_dataset.py:997:38
 ERROR Argument `None` is not assignable to parameter `embedding_model` with type `str` in function `tests.unit_tests.services.dataset_service_test_helpers.DatasetServiceUnitDataFactory.create_dataset_mock` [bad-argument-type]
-   --> tests/unit_tests/services/test_dataset_service_dataset.py:997:29
+   --> tests/unit_tests/services/test_dataset_service_dataset.py:998:29
 ERROR Object of class `FakeAccount` has no attribute `current_tenant_id` [missing-attribute]
-    --> tests/unit_tests/services/test_dataset_service_dataset.py:1066:9
+    --> tests/unit_tests/services/test_dataset_service_dataset.py:1067:9
 ERROR Object of class `FakeAccount` has no attribute `current_tenant_id` [missing-attribute]
-    --> tests/unit_tests/services/test_dataset_service_dataset.py:1103:9
+    --> tests/unit_tests/services/test_dataset_service_dataset.py:1104:9
 ERROR Argument `SimpleNamespace` is not assignable to parameter `user` with type `Account | None` in function `services.dataset_service.DatasetService.check_dataset_operator_permission` [bad-argument-type]
-    --> tests/unit_tests/services/test_dataset_service_dataset.py:1401:22
+    --> tests/unit_tests/services/test_dataset_service_dataset.py:1402:22
 ERROR Argument `SimpleNamespace` is not assignable to parameter `dataset` with type `Dataset | None` in function `services.dataset_service.DatasetService.check_dataset_operator_permission` [bad-argument-type]
-    --> tests/unit_tests/services/test_dataset_service_dataset.py:1409:25
+    --> tests/unit_tests/services/test_dataset_service_dataset.py:1410:25
 ERROR Object of class `FakeAccount` has no attribute `id` [missing-attribute]
   --> tests/unit_tests/services/test_dataset_service_document.py:93:9
 ERROR Object of class `FakeAccount` has no attribute `current_tenant_id` [missing-attribute]

[github-actions]

Pyrefly Type Coverage

Metric	Base	PR	Delta
Type coverage	52.85%	52.84%	-0.01%
Strict coverage	52.36%	52.36%	-0.01%
Typed symbols	32,803	32,805	+2
Untyped symbols	29,539	29,548	+9
Modules	2988	2988	0