refactor: replace db.paginate with plain SQLAlchemy pagination

[sergioperezcheco]

This PR replaces all db.paginate calls with a plain SQLAlchemy pagination helper.

Motivation

Following the direction of #38262, this removes the dependency on Flask-SQLAlchemy's db.paginate extension method. The replacement uses standard SQLAlchemy select with limit/offset and a scalar subquery for the total count, making the pagination logic explicit and portable.

What changed

Added api/libs/pagination.py with:

• PaginatedResult — a lightweight dataclass exposing items, total, page, per_page, pages, has_next, plus __iter__ for direct iteration
• paginate_query() — executes count and page queries using plain SQLAlchemy, falling back to db.session when no session is passed

All 12 source files that called db.paginate have been updated: 5 in dataset_service, 2 in app_service, 2 in annotation_service, 1 each in external_knowledge_service, plugin_migration, conversation controller, workspace controller, datasets_document controller, datasets_segments controller, service_api document controller, the schedule task, and the vector command.

No behavioral changes — the API responses and return types are preserved.

Closes #38262

[github-actions]

Pyrefly Diff

base → PR

--- /tmp/pyrefly_base.txt	2026-07-03 06:09:43.131180753 +0000
+++ /tmp/pyrefly_pr.txt	2026-07-03 06:09:33.529179998 +0000
@@ -7056,11 +7056,11 @@
 ERROR Argument `Literal['created_at']` is not assignable to parameter `k` with type `Never` in function `_typeshed._type_checker_internals.TypedDictFallback.pop` [bad-argument-type]
    --> tests/unit_tests/services/test_agent_drive_service.py:755:32
 ERROR Argument `dict[str, str]` is not assignable to parameter `args` with type `InsertAnnotationArgs` in function `services.annotation_service.AppAnnotationService.insert_app_annotation_directly` [bad-argument-type]
-   --> tests/unit_tests/services/test_annotation_service.py:511:69
+   --> tests/unit_tests/services/test_annotation_service.py:513:69
 ERROR TypedDict `AnnotationSettingDisabledDict` does not have key `embedding_model` [bad-typed-dict-key]
-    --> tests/unit_tests/services/test_annotation_service.py:1190:59
+    --> tests/unit_tests/services/test_annotation_service.py:1194:59
 ERROR TypedDict `AnnotationSettingDisabledDict` does not have key `embedding_model` [bad-typed-dict-key]
-    --> tests/unit_tests/services/test_annotation_service.py:1227:27
+    --> tests/unit_tests/services/test_annotation_service.py:1231:27
 ERROR Class member `_TrackingRateLimit.exit` overrides a member in a parent class but is missing an `@override` decorator [missing-override-decorator]
    --> tests/unit_tests/services/test_app_generate_service.py:560:17
 ERROR Class member `_TrackingRateLimit.exit` overrides a member in a parent class but is missing an `@override` decorator [missing-override-decorator]
@@ -7218,33 +7218,33 @@
 ERROR Argument `SimpleNamespace` is not assignable to parameter `dataset` with type `Dataset` in function `services.dataset_service.SegmentService.delete_child_chunk` [bad-argument-type]
    --> tests/unit_tests/services/test_dataset_service_segment.py:250:64
 ERROR Argument `SimpleNamespace` is not assignable to parameter `segment` with type `DocumentSegment` in function `services.dataset_service.SegmentService.update_child_chunk` [bad-argument-type]
-   --> tests/unit_tests/services/test_dataset_service_segment.py:913:49
+   --> tests/unit_tests/services/test_dataset_service_segment.py:915:49
 ERROR Argument `SimpleNamespace` is not assignable to parameter `document` with type `Document` in function `services.dataset_service.SegmentService.update_child_chunk` [bad-argument-type]
-   --> tests/unit_tests/services/test_dataset_service_segment.py:913:68
+   --> tests/unit_tests/services/test_dataset_service_segment.py:915:68
 ERROR Argument `SimpleNamespace` is not assignable to parameter `dataset` with type `Dataset` in function `services.dataset_service.SegmentService.update_child_chunk` [bad-argument-type]
-   --> tests/unit_tests/services/test_dataset_service_segment.py:913:87
+   --> tests/unit_tests/services/test_dataset_service_segment.py:915:87
 ERROR Argument `SimpleNamespace` is not assignable to parameter `dataset` with type `Dataset` in function `services.dataset_service.SegmentService.delete_child_chunk` [bad-argument-type]
-   --> tests/unit_tests/services/test_dataset_service_segment.py:927:60
+   --> tests/unit_tests/services/test_dataset_service_segment.py:929:60
 ERROR Argument `str` is not assignable to parameter `type` with type `Literal['basic', 'bearer', 'custom'] | None` in function `services.entities.external_knowledge_entities.external_knowledge_entities.AuthorizationConfig.__init__` [bad-argument-type]
    --> tests/unit_tests/services/test_external_dataset_service.py:118:60
 ERROR Argument `str` is not assignable to parameter `type` with type `Literal['api-key', 'no-auth']` in function `services.entities.external_knowledge_entities.external_knowledge_entities.Authorization.__init__` [bad-argument-type]
    --> tests/unit_tests/services/test_external_dataset_service.py:119:35
 ERROR Argument `None` is not assignable to parameter `api_settings` with type `dict[str, Any]` in function `services.external_knowledge_service.ExternalDatasetService.validate_api_list` [bad-argument-type]
-   --> tests/unit_tests/services/test_external_dataset_service.py:417:54
+   --> tests/unit_tests/services/test_external_dataset_service.py:421:54
 ERROR Argument `str | None` is not assignable to parameter `s` with type `bytearray | bytes | str` in function `json.loads` [bad-argument-type]
    --> tests/unit_tests/services/test_external_dataset_service.py:916:31
 ERROR Argument `list[dict[str, list[dict[str, bool | str]]]]` is not assignable to parameter `settings` with type `dict[str, Any] | None` in function `ExternalDatasetServiceTestDataFactory.create_external_knowledge_api_mock` [bad-argument-type]
-    --> tests/unit_tests/services/test_external_dataset_service.py:1128:82
+    --> tests/unit_tests/services/test_external_dataset_service.py:1118:82
 ERROR Argument `list[dict[str, list[dict[str, bool | str]]]]` is not assignable to parameter `settings` with type `dict[str, Any] | None` in function `ExternalDatasetServiceTestDataFactory.create_external_knowledge_api_mock` [bad-argument-type]
-    --> tests/unit_tests/services/test_external_dataset_service.py:1148:82
+    --> tests/unit_tests/services/test_external_dataset_service.py:1136:82
 ERROR Argument `list[dict[Unknown, Unknown]]` is not assignable to parameter `settings` with type `dict[str, Any] | None` in function `ExternalDatasetServiceTestDataFactory.create_external_knowledge_api_mock` [bad-argument-type]
-    --> tests/unit_tests/services/test_external_dataset_service.py:1175:67
+    --> tests/unit_tests/services/test_external_dataset_service.py:1161:67
 ERROR Argument `list[dict[str, list[dict[str, bool | str]]]]` is not assignable to parameter `settings` with type `dict[str, Any] | None` in function `ExternalDatasetServiceTestDataFactory.create_external_knowledge_api_mock` [bad-argument-type]
-    --> tests/unit_tests/services/test_external_dataset_service.py:1195:67
+    --> tests/unit_tests/services/test_external_dataset_service.py:1181:67
 ERROR `None` is not subscriptable [unsupported-operation]
-    --> tests/unit_tests/services/test_external_dataset_service.py:1487:16
+    --> tests/unit_tests/services/test_external_dataset_service.py:1471:16
 ERROR `None` is not subscriptable [unsupported-operation]
-    --> tests/unit_tests/services/test_external_dataset_service.py:1488:16
+    --> tests/unit_tests/services/test_external_dataset_service.py:1472:16
 ERROR Argument `Literal['invalid']` is not assignable to parameter `session_factory` with type `Engine | sessionmaker[Unknown] | None` in function `services.file_service.FileService.__init__` [bad-argument-type]
   --> tests/unit_tests/services/test_file_service.py:48:41
 ERROR Argument `list[FromClause]` is not assignable to parameter `tables` with type `Sequence[Table] | None` in function `sqlalchemy.sql.schema.MetaData.create_all` [bad-argument-type]

[github-actions]

Pyrefly Type Coverage

Metric	Base	PR	Delta
Type coverage	52.27%	52.31%	+0.04%
Strict coverage	51.78%	51.82%	+0.04%
Typed symbols	31,977	31,985	+8
Untyped symbols	29,478	29,436	-42
Modules	2969	2970	+1

[asukaminato0721]

fix ci

[asukaminato0721]

error[UP046]: Generic class `PaginatedResult` uses `Generic` subclass instead of type parameters
  --> libs/pagination.py:14:23
   |
13 | @dataclass
14 | class PaginatedResult(Generic[T]):
   |                       ^^^^^^^^^^
15 |     """Minimal pagination container backed by plain SQLAlchemy queries.
   |
help: Use type parameters

[sergioperezcheco]

thanks for catching that, fixed. switched to Python 3.12 type parameter syntax and the ruff UP046 error should be gone now.

[asukaminato0721]

remove the patch db

FAILED api/tests/unit_tests/services/test_external_dataset_service.py::TestExternalDatasetServiceFetchRetrieval::test_fetch_external_knowledge_retrieval_wraps_transport_errors - AttributeError: <module 'services.external_knowledge_service' from '/home/runner/work/dify/dify/api/services/external_knowledge_service.py'> does not have the attribute 'db'