refactor(api): type WaterCrawl API responses with TypedDict by bittoby · Pull Request #33700 · langgenius/dify

bittoby · 2026-03-18T20:10:01Z

Summary

Types the WaterCrawl API request and response dicts in client.py and provider.py using explicit TypedDicts as part of the broader effort (#32863) to replace bare dict/Mapping annotations across api/core/rag. client.py and provider.py had untyped dict for both crawl configuration options (spider_options, page_options) and all response shapes, making the WaterCrawl API contract invisible to the type system. Typing these enables IDE autocompletion on response fields and catches field-name mistakes at type-check time rather than at runtime.

Changes

client.py: Add SpiderOptions TypedDict (max_depth, page_limit, allowed_domains, exclude_paths, include_paths) and PageOptions TypedDict (all page-level crawl settings); update create_crawl_request() and scrape_url() param types
provider.py: Add WatercrawlDocumentData, CrawlJobResponse, WatercrawlCrawlStatusResponse TypedDicts; update crawl_url(), get_crawl_status(), get_crawl_url_data(), scrape_url(), _structure_data(), and _get_results() return types; annotate spider_options and page_options local variables
website_service.py: Wrap TypedDict returns with dict() at the dict[str, Any] boundary (basedpyright requirement)

Test plan

make lint passes
make type-check passes (basedpyright + pyrefly + mypy — 0 errors)
uv run --project api pytest api/tests/unit_tests/core/rag/extractor/watercrawl/ — 27 tests pass

Part of #32863 (core/rag/extractor/watercrawl/)

…xtractor

gemini-code-assist · 2026-03-18T20:10:22Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the type safety and clarity of the WaterCrawl API integration by replacing untyped dict annotations with explicit TypedDict definitions. This change improves code maintainability, enables better IDE autocompletion, and allows for compile-time detection of API contract mismatches, preventing potential runtime errors.

Highlights

TypedDict Definitions: Introduced TypedDict definitions for WaterCrawl API request configuration (SpiderOptions, PageOptions) and response structures (WatercrawlDocumentData, CrawlJobResponse, WatercrawlCrawlStatusResponse) in client.py and provider.py.
Function Signature Updates: Updated function signatures in client.py to use the new SpiderOptions and PageOptions TypedDicts for crawl configuration parameters, and in provider.py to apply TypedDict return type annotations for various methods.
Type Compatibility Fix: Implemented dict() wrapping for TypedDict returns in website_service.py to ensure compatibility with basedpyright's strict type checking for dict[str, Any] boundaries.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

github-actions · 2026-03-18T20:11:55Z

Pyrefly Diff

base → PR

--- /tmp/pyrefly_base.txt	2026-03-18 20:11:36.786117604 +0000
+++ /tmp/pyrefly_pr.txt	2026-03-18 20:11:27.652002056 +0000
@@ -399,44 +399,44 @@
 ERROR Argument `dict[str, bytes | str]` is not assignable to parameter `headers` with type `Headers | Mapping[bytes, bytes] | Mapping[str, str] | Sequence[tuple[bytes, bytes]] | Sequence[tuple[str, str]] | None` in function `httpx._api.request` [bad-argument-type]
    --> core/rag/extractor/notion_extractor.py:368:21
 ERROR Cannot index into `Generator[Unknown]` [bad-index]
-   --> core/rag/extractor/watercrawl/client.py:188:54
+   --> core/rag/extractor/watercrawl/client.py:210:54
 ERROR Cannot index into `bytes` [bad-index]
-   --> core/rag/extractor/watercrawl/client.py:188:54
+   --> core/rag/extractor/watercrawl/client.py:210:54
 ERROR Cannot index into `list[Unknown]` [bad-index]
-   --> core/rag/extractor/watercrawl/client.py:188:54
+   --> core/rag/extractor/watercrawl/client.py:210:54
 ERROR `None` is not subscriptable [unsupported-operation]
-   --> core/rag/extractor/watercrawl/client.py:188:54
+   --> core/rag/extractor/watercrawl/client.py:210:54
 ERROR Object of class `Generator` has no attribute `get`
 ERROR Cannot index into `Generator[Unknown]` [bad-index]
-  --> core/rag/extractor/watercrawl/provider.py:47:12
+  --> core/rag/extractor/watercrawl/provider.py:70:12
 ERROR Cannot index into `bytes` [bad-index]
-  --> core/rag/extractor/watercrawl/provider.py:47:12
+  --> core/rag/extractor/watercrawl/provider.py:70:12
 ERROR Cannot index into `list[Unknown]` [bad-index]
-  --> core/rag/extractor/watercrawl/provider.py:47:12
+  --> core/rag/extractor/watercrawl/provider.py:70:12
 ERROR `None` is not subscriptable [unsupported-operation]
-  --> core/rag/extractor/watercrawl/provider.py:47:12
+  --> core/rag/extractor/watercrawl/provider.py:70:12
 ERROR Object of class `Generator` has no attribute `get`
 ERROR Object of class `Generator` has no attribute `get`
 ERROR Object of class `Generator` has no attribute `get`
 ERROR Object of class `Generator` has no attribute `get`
-ERROR Argument `Generator[Unknown] | bytes | dict[Unknown, Unknown] | list[Unknown] | Unknown | None` is not assignable to parameter `result_object` with type `dict[Unknown, Unknown]` in function `WaterCrawlProvider._structure_data` [bad-argument-type]
-  --> core/rag/extractor/watercrawl/provider.py:87:37
+ERROR Argument `Generator[Unknown] | bytes | dict[Unknown, Unknown] | list[Unknown] | Unknown | None` is not assignable to parameter `result_object` with type `dict[str, Any]` in function `WaterCrawlProvider._structure_data` [bad-argument-type]
+   --> core/rag/extractor/watercrawl/provider.py:110:37
 ERROR Cannot index into `Generator[Unknown]` [bad-index]
-   --> core/rag/extractor/watercrawl/provider.py:110:20
+   --> core/rag/extractor/watercrawl/provider.py:135:20
 ERROR Cannot index into `bytes` [bad-index]
-   --> core/rag/extractor/watercrawl/provider.py:110:20
+   --> core/rag/extractor/watercrawl/provider.py:135:20
 ERROR Cannot index into `list[Unknown]` [bad-index]
-   --> core/rag/extractor/watercrawl/provider.py:110:20
+   --> core/rag/extractor/watercrawl/provider.py:135:20
 ERROR `None` is not subscriptable [unsupported-operation]
-   --> core/rag/extractor/watercrawl/provider.py:110:20
+   --> core/rag/extractor/watercrawl/provider.py:135:20
 ERROR Cannot index into `Generator[Unknown]` [bad-index]
-   --> core/rag/extractor/watercrawl/provider.py:116:16
+   --> core/rag/extractor/watercrawl/provider.py:141:16
 ERROR Cannot index into `bytes` [bad-index]
-   --> core/rag/extractor/watercrawl/provider.py:116:16
+   --> core/rag/extractor/watercrawl/provider.py:141:16
 ERROR Cannot index into `list[Unknown]` [bad-index]
-   --> core/rag/extractor/watercrawl/provider.py:116:16
+   --> core/rag/extractor/watercrawl/provider.py:141:16
 ERROR `None` is not subscriptable [unsupported-operation]
-   --> core/rag/extractor/watercrawl/provider.py:116:16
+   --> core/rag/extractor/watercrawl/provider.py:141:16
 ERROR Pyrefly detected conflicting types while breaking a dependency cycle: `str | Any | None` is not assignable to `None`. Adding explicit type annotations might possibly help. [bad-assignment]
    --> core/rag/extractor/word_extractor.py:371:13
 ERROR Pyrefly detected conflicting types while breaking a dependency cycle: `str | Any | None` is not assignable to `None`. Adding explicit type annotations might possibly help. [bad-assignment]
@@ -3346,6 +3346,8 @@
    --> tests/unit_tests/core/datasource/test_datasource_manager.py:573:34
 ERROR Object of class `StreamChunkEvent` has no attribute `node_run_result` [missing-attribute]
    --> tests/unit_tests/core/datasource/test_datasource_manager.py:624:12
+ERROR `in` is not supported between `Literal['Single Page']` and `None` [not-iterable]
+   --> tests/unit_tests/core/datasource/test_website_crawl.py:989:16
 ERROR Argument `Iterator[DatasourceMessage]` is not assignable to parameter `messages` with type `Generator[DatasourceMessage]` in function `core.datasource.utils.message_transformer.DatasourceFileMessageTransformer.transform_datasource_invoke_messages` [bad-argument-type]
   --> tests/unit_tests/core/datasource/utils/test_message_transformer.py:28:26
 ERROR Object of class `BlobChunkMessage` has no attribute `text`

gemini-code-assist

Code Review

This pull request is a solid step towards improving type safety by introducing TypedDict for WaterCrawl API responses. The changes are well-implemented across client.py, provider.py, and website_service.py. My review includes a few suggestions to further enhance type specificity by replacing bare dict annotations with dict[str, Any], which aligns with the goals of this refactoring.

api/core/rag/extractor/watercrawl/provider.py

api/services/website_service.py

github-actions · 2026-03-18T20:24:19Z

Pyrefly Diff

base → PR

--- /tmp/pyrefly_base.txt	2026-03-18 20:24:03.306032143 +0000
+++ /tmp/pyrefly_pr.txt	2026-03-18 20:23:54.020880598 +0000
@@ -399,44 +399,44 @@
 ERROR Argument `dict[str, bytes | str]` is not assignable to parameter `headers` with type `Headers | Mapping[bytes, bytes] | Mapping[str, str] | Sequence[tuple[bytes, bytes]] | Sequence[tuple[str, str]] | None` in function `httpx._api.request` [bad-argument-type]
    --> core/rag/extractor/notion_extractor.py:368:21
 ERROR Cannot index into `Generator[Unknown]` [bad-index]
-   --> core/rag/extractor/watercrawl/client.py:188:54
+   --> core/rag/extractor/watercrawl/client.py:210:54
 ERROR Cannot index into `bytes` [bad-index]
-   --> core/rag/extractor/watercrawl/client.py:188:54
+   --> core/rag/extractor/watercrawl/client.py:210:54
 ERROR Cannot index into `list[Unknown]` [bad-index]
-   --> core/rag/extractor/watercrawl/client.py:188:54
+   --> core/rag/extractor/watercrawl/client.py:210:54
 ERROR `None` is not subscriptable [unsupported-operation]
-   --> core/rag/extractor/watercrawl/client.py:188:54
+   --> core/rag/extractor/watercrawl/client.py:210:54
 ERROR Object of class `Generator` has no attribute `get`
 ERROR Cannot index into `Generator[Unknown]` [bad-index]
-  --> core/rag/extractor/watercrawl/provider.py:47:12
+  --> core/rag/extractor/watercrawl/provider.py:70:12
 ERROR Cannot index into `bytes` [bad-index]
-  --> core/rag/extractor/watercrawl/provider.py:47:12
+  --> core/rag/extractor/watercrawl/provider.py:70:12
 ERROR Cannot index into `list[Unknown]` [bad-index]
-  --> core/rag/extractor/watercrawl/provider.py:47:12
+  --> core/rag/extractor/watercrawl/provider.py:70:12
 ERROR `None` is not subscriptable [unsupported-operation]
-  --> core/rag/extractor/watercrawl/provider.py:47:12
+  --> core/rag/extractor/watercrawl/provider.py:70:12
 ERROR Object of class `Generator` has no attribute `get`
 ERROR Object of class `Generator` has no attribute `get`
 ERROR Object of class `Generator` has no attribute `get`
 ERROR Object of class `Generator` has no attribute `get`
-ERROR Argument `Generator[Unknown] | bytes | dict[Unknown, Unknown] | list[Unknown] | Unknown | None` is not assignable to parameter `result_object` with type `dict[Unknown, Unknown]` in function `WaterCrawlProvider._structure_data` [bad-argument-type]
-  --> core/rag/extractor/watercrawl/provider.py:87:37
+ERROR Argument `Generator[Unknown] | bytes | dict[Unknown, Unknown] | list[Unknown] | Unknown | None` is not assignable to parameter `result_object` with type `dict[str, Any]` in function `WaterCrawlProvider._structure_data` [bad-argument-type]
+   --> core/rag/extractor/watercrawl/provider.py:110:37
 ERROR Cannot index into `Generator[Unknown]` [bad-index]
-   --> core/rag/extractor/watercrawl/provider.py:110:20
+   --> core/rag/extractor/watercrawl/provider.py:135:20
 ERROR Cannot index into `bytes` [bad-index]
-   --> core/rag/extractor/watercrawl/provider.py:110:20
+   --> core/rag/extractor/watercrawl/provider.py:135:20
 ERROR Cannot index into `list[Unknown]` [bad-index]
-   --> core/rag/extractor/watercrawl/provider.py:110:20
+   --> core/rag/extractor/watercrawl/provider.py:135:20
 ERROR `None` is not subscriptable [unsupported-operation]
-   --> core/rag/extractor/watercrawl/provider.py:110:20
+   --> core/rag/extractor/watercrawl/provider.py:135:20
 ERROR Cannot index into `Generator[Unknown]` [bad-index]
-   --> core/rag/extractor/watercrawl/provider.py:116:16
+   --> core/rag/extractor/watercrawl/provider.py:141:16
 ERROR Cannot index into `bytes` [bad-index]
-   --> core/rag/extractor/watercrawl/provider.py:116:16
+   --> core/rag/extractor/watercrawl/provider.py:141:16
 ERROR Cannot index into `list[Unknown]` [bad-index]
-   --> core/rag/extractor/watercrawl/provider.py:116:16
+   --> core/rag/extractor/watercrawl/provider.py:141:16
 ERROR `None` is not subscriptable [unsupported-operation]
-   --> core/rag/extractor/watercrawl/provider.py:116:16
+   --> core/rag/extractor/watercrawl/provider.py:141:16
 ERROR Pyrefly detected conflicting types while breaking a dependency cycle: `str | Any | None` is not assignable to `None`. Adding explicit type annotations might possibly help. [bad-assignment]
    --> core/rag/extractor/word_extractor.py:371:13
 ERROR Pyrefly detected conflicting types while breaking a dependency cycle: `str | Any | None` is not assignable to `None`. Adding explicit type annotations might possibly help. [bad-assignment]
@@ -3346,6 +3346,8 @@
    --> tests/unit_tests/core/datasource/test_datasource_manager.py:573:34
 ERROR Object of class `StreamChunkEvent` has no attribute `node_run_result` [missing-attribute]
    --> tests/unit_tests/core/datasource/test_datasource_manager.py:624:12
+ERROR `in` is not supported between `Literal['Single Page']` and `None` [not-iterable]
+   --> tests/unit_tests/core/datasource/test_website_crawl.py:989:16
 ERROR Argument `Iterator[DatasourceMessage]` is not assignable to parameter `messages` with type `Generator[DatasourceMessage]` in function `core.datasource.utils.message_transformer.DatasourceFileMessageTransformer.transform_datasource_invoke_messages` [bad-argument-type]
   --> tests/unit_tests/core/datasource/utils/test_message_transformer.py:28:26
 ERROR Object of class `BlobChunkMessage` has no attribute `text`

refactor(api): replace dict[str, Any] with TypedDicts in watercrawl e…

a281817

…xtractor

bittoby requested review from JohnJyong and QuantumGhost as code owners March 18, 2026 20:10

dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. refactor labels Mar 18, 2026

gemini-code-assist bot reviewed Mar 18, 2026

View reviewed changes

api/core/rag/extractor/watercrawl/provider.py Outdated Show resolved Hide resolved

api/services/website_service.py Outdated Show resolved Hide resolved

api/services/website_service.py Outdated Show resolved Hide resolved

api/services/website_service.py Outdated Show resolved Hide resolved

refactor(api): apply dict[str, Any] suggestions from PR review

e6416d1

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Mar 18, 2026

asukaminato0721 approved these changes Mar 19, 2026

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Mar 19, 2026

asukaminato0721 merged commit 9ff0d9d into langgenius:main Mar 19, 2026
14 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(api): type WaterCrawl API responses with TypedDict#33700

refactor(api): type WaterCrawl API responses with TypedDict#33700
asukaminato0721 merged 2 commits intolanggenius:mainfrom
bittoby:refactor/rag-watercrawl-typeddict

bittoby commented Mar 18, 2026

Uh oh!

gemini-code-assist bot commented Mar 18, 2026

Uh oh!

github-actions bot commented Mar 18, 2026

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

bittoby commented Mar 18, 2026

Summary

Changes

Test plan

Uh oh!

gemini-code-assist bot commented Mar 18, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

github-actions bot commented Mar 18, 2026

Pyrefly Diff

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 18, 2026

Pyrefly Diff

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants