Skip to content

feat: evaluation batch test#35800

Merged
FFXN merged 174 commits into
deploy/devfrom
deploy/lzy
May 5, 2026
Merged

feat: evaluation batch test#35800
FFXN merged 174 commits into
deploy/devfrom
deploy/lzy

Conversation

@FFXN
Copy link
Copy Markdown
Contributor

@FFXN FFXN commented May 5, 2026

Important

  1. Make sure you have read our contribution guidelines
  2. Ensure there is an associated issue and you have been assigned to it
  3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

Screenshots

Before After
... ...

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran make lint && make type-check (backend) and cd web && pnpm exec vp staged (frontend) to appease the lint gods

JohnJyong and others added 30 commits January 30, 2026 17:35
…ile and downloading evaluation dataset file/evaluation result file.
…ile and downloading evaluation dataset file/evaluation result file.
…ation

# Conflicts:
#	api/controllers/console/snippets/payloads.py
#	api/controllers/console/snippets/snippet_workflow.py
#	api/services/snippet_service.py
…ation

# Conflicts:
#	api/controllers/console/evaluation/evaluation.py
…de_run_result_mapping] in evaluation_service.
FFXN added 23 commits April 30, 2026 10:21
# Conflicts:
#	api/tests/unit_tests/controllers/console/app/test_workflow.py
# Conflicts:
#	api/controllers/console/evaluation/evaluation.py
#	api/services/evaluation_service.py
# Conflicts:
#	api/tests/unit_tests/controllers/console/app/test_workflow_app_log_api.py
@FFXN FFXN requested a review from QuantumGhost as a code owner May 5, 2026 13:02
@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. 💪 enhancement New feature or request labels May 5, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 5, 2026

Pyrefly Type Coverage

Metric Base PR Delta
Type coverage 39.35% 39.35% 0.00%
Strict coverage 38.83% 38.83% 0.00%
Typed symbols 20,092 20,092 0
Untyped symbols 31,384 31,384 0
Modules 2582 2582 0

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 5, 2026

Pyrefly Diff

base → PR
--- /tmp/pyrefly_base.txt	2026-05-05 13:04:01.323325840 +0000
+++ /tmp/pyrefly_pr.txt	2026-05-05 13:03:46.472160053 +0000
@@ -17,11 +17,11 @@
 ERROR Cannot find module `core.workflow.file` [missing-import]
     --> controllers/console/datasets/datasets.py:1395:9
 ERROR Argument `list[dict[str, Any]]` is not assignable to parameter `default_metrics` with type `list[Mapping[str, Any]]` in function `services.evaluation_service.EvaluationService.serialize_console_default_metrics` [bad-argument-type]
-   --> controllers/console/evaluation/evaluation.py:336:84
+   --> controllers/console/evaluation/evaluation.py:346:84
 ERROR Argument `list[dict[str, Any]]` is not assignable to parameter `default_metrics` with type `list[Mapping[str, Any]]` in function `services.evaluation_service.EvaluationService.serialize_console_default_metrics` [bad-argument-type]
-   --> controllers/console/evaluation/evaluation.py:374:84
+   --> controllers/console/evaluation/evaluation.py:384:84
 ERROR Argument `list[DefaultMetric]` is not assignable to parameter `default_metrics` with type `list[DefaultMetric | Mapping[str, Any]]` in function `services.evaluation_service.EvaluationService.filter_console_default_metrics` [bad-argument-type]
-   --> controllers/console/evaluation/evaluation.py:663:90
+   --> controllers/console/evaluation/evaluation.py:673:90
 ERROR Object of class `MissingRouter` has no attribute `get` [missing-attribute]
   --> controllers/console/init_validate.py:31:2
 ERROR Object of class `MissingRouter` has no attribute `post` [missing-attribute]
@@ -643,47 +643,47 @@
 ERROR Class member `DuplicateDocumentIndexingTaskProxy.PRIORITY_TASK_FUNC` overrides parent class `BatchDocumentIndexingProxy` in an inconsistent manner [bad-override]
   --> services/document_indexing_proxy/duplicate_document_indexing_task_proxy.py:15:5
 ERROR Object of class `Cell` has no attribute `font` (not declared in `__slots__`) [missing-attribute]
-   --> services/evaluation_service.py:207:13
+   --> services/evaluation_service.py:212:13
 ERROR Object of class `Cell` has no attribute `fill` (not declared in `__slots__`) [missing-attribute]
-   --> services/evaluation_service.py:208:13
+   --> services/evaluation_service.py:213:13
 ERROR Object of class `Cell` has no attribute `alignment` (not declared in `__slots__`) [missing-attribute]
-   --> services/evaluation_service.py:209:13
+   --> services/evaluation_service.py:214:13
 ERROR Object of class `Cell` has no attribute `border` (not declared in `__slots__`) [missing-attribute]
-   --> services/evaluation_service.py:210:13
+   --> services/evaluation_service.py:215:13
 ERROR `Literal[10]` is not assignable to attribute `width` with type `Float` [bad-assignment]
-   --> services/evaluation_service.py:213:43
+   --> services/evaluation_service.py:218:43
 ERROR `Literal[20]` is not assignable to attribute `width` with type `Float` [bad-assignment]
-   --> services/evaluation_service.py:215:70
+   --> services/evaluation_service.py:220:70
 ERROR Object of class `Cell` has no attribute `border` (not declared in `__slots__`) [missing-attribute]
-   --> services/evaluation_service.py:220:13
+   --> services/evaluation_service.py:225:13
 ERROR Object of class `Cell` has no attribute `alignment` (not declared in `__slots__`) [missing-attribute]
-   --> services/evaluation_service.py:223:17
+   --> services/evaluation_service.py:228:17
 ERROR Object of class `Cell` has no attribute `font` (not declared in `__slots__`) [missing-attribute]
-   --> services/evaluation_service.py:261:13
+   --> services/evaluation_service.py:266:13
 ERROR Object of class `Cell` has no attribute `fill` (not declared in `__slots__`) [missing-attribute]
-   --> services/evaluation_service.py:262:13
+   --> services/evaluation_service.py:267:13
 ERROR Object of class `Cell` has no attribute `alignment` (not declared in `__slots__`) [missing-attribute]
-   --> services/evaluation_service.py:263:13
+   --> services/evaluation_service.py:268:13
 ERROR Object of class `Cell` has no attribute `border` (not declared in `__slots__`) [missing-attribute]
-   --> services/evaluation_service.py:264:13
+   --> services/evaluation_service.py:269:13
 ERROR `Literal[10]` is not assignable to attribute `width` with type `Float` [bad-assignment]
-   --> services/evaluation_service.py:266:43
+   --> services/evaluation_service.py:271:43
 ERROR `Literal[30]` is not assignable to attribute `width` with type `Float` [bad-assignment]
-   --> services/evaluation_service.py:267:43
+   --> services/evaluation_service.py:272:43
 ERROR `Literal[30]` is not assignable to attribute `width` with type `Float` [bad-assignment]
-   --> services/evaluation_service.py:268:43
+   --> services/evaluation_service.py:273:43
 ERROR Object of class `Cell` has no attribute `border` (not declared in `__slots__`) [missing-attribute]
-   --> services/evaluation_service.py:273:13
+   --> services/evaluation_service.py:278:13
 ERROR Object of class `Cell` has no attribute `alignment` (not declared in `__slots__`) [missing-attribute]
-   --> services/evaluation_service.py:276:17
+   --> services/evaluation_service.py:281:17
 ERROR Argument `list[Mapping[str, Any]]` is not assignable to parameter `default_metrics` with type `list[DefaultMetric | Mapping[str, Any]]` in function `EvaluationService.filter_console_default_metrics` [bad-argument-type]
-   --> services/evaluation_service.py:656:86
-ERROR Argument `list[DefaultMetric]` is not assignable to parameter `default_metrics` with type `list[DefaultMetric | Mapping[str, Any]]` in function `EvaluationService._get_expected_output_column_names` [bad-argument-type]
-   --> services/evaluation_service.py:676:73
+   --> services/evaluation_service.py:661:86
+ERROR Argument `list[DefaultMetric]` is not assignable to parameter `default_metrics` with type `list[DefaultMetric | Mapping[str, Any]]` in function `EvaluationService._get_expected_output_columns` [bad-argument-type]
+   --> services/evaluation_service.py:681:68
 ERROR Argument `list[tuple[_CellGetValue, ...]]` is not assignable to parameter `rows` with type `list[list[Any] | tuple[Any, ...]]` in function `EvaluationService._rows_to_dataset_items` [bad-argument-type]
-    --> services/evaluation_service.py:1106:44
+    --> services/evaluation_service.py:1129:44
 ERROR Argument `list[list[str]]` is not assignable to parameter `rows` with type `list[list[Any] | tuple[Any, ...]]` in function `EvaluationService._rows_to_dataset_items` [bad-argument-type]
-    --> services/evaluation_service.py:1133:43
+    --> services/evaluation_service.py:1156:43
 ERROR Argument `bool | str | Any` is not assignable to parameter `token` with type `str` in function `core.helper.encrypter.encrypt_token` [bad-argument-type]
    --> services/model_load_balancing_service.py:623:71
 ERROR `handled_tenant_count` was assigned in the current scope before the nonlocal declaration [unknown-name]

@FFXN FFXN merged commit a55171a into deploy/dev May 5, 2026
11 checks passed
@FFXN FFXN deleted the deploy/lzy branch May 5, 2026 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

💪 enhancement New feature or request size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants