fix: add bounded timeout to Firecrawl RAG extractor requests - #39596
Open
SpiliosDimakopoulos wants to merge 7 commits into
Open
fix: add bounded timeout to Firecrawl RAG extractor requests#39596SpiliosDimakopoulos wants to merge 7 commits into
SpiliosDimakopoulos wants to merge 7 commits into
Conversation
…ΘΜΟ>) 'Runing docker container' -> 'Running docker container'.
…ΘΜΟΣ>) ar-SA: unescape the quick-start-guide link and point it at the real heading slug (#البداية-السريعة). hi-IN: point the community link at the real bilingual heading slug (#समुदाय-और-संपर्क-community--contact). Verified both slugs with github-slugger.
validate_credentials() -> _get_request() called httpx.get() with no explicit timeout, unlike the Jina/Firecrawl/Nacos/Marketplace auth providers which were recently hardened with bounded httpx.Timeout values. A slow or hanging WaterCrawl endpoint could block the worker indefinitely during credential validation. Updated the existing unit test to assert the new timeout kwarg.
FirecrawlApp._post_request()/_get_request() called httpx.post()/get() with no explicit timeout inside a 502-retry loop, unlike the equivalent WaterCrawl RAG extractor client (core/rag/extractor/watercrawl/client.py) which already uses a bounded httpx.Timeout. A slow or hanging Firecrawl endpoint could block the worker indefinitely during scrape/crawl/map/search requests. Verified locally that existing behavior (200 responses, 502-retry-then-success, retry exhaustion) is unchanged; no test assertions check call kwargs so no test updates were needed.
SpiliosDimakopoulos
requested review from
JohnJyong,
QuantumGhost,
crazywoola and
laipz8200
as code owners
July 26, 2026 14:45
dosubot
Bot
added
the
size:S
Contributor
Pyrefly Type Coverage
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FirecrawlApp._post_request()/_get_request() call httpx.post()/get() with no explicit timeout inside a 502-retry loop, unlike the equivalent WaterCrawl RAG extractor client (core/rag/extractor/watercrawl/client.py), which already uses a bounded httpx.Timeout(30.0, connect=5.0). A slow or hanging Firecrawl endpoint could otherwise block the worker indefinitely during scrape/crawl/map/search requests. Adds the same bounded timeout constant and passes it to both outbound calls. Verified the 502-retry-then-success and retry-exhaustion paths are unchanged; existing tests only assert on return values/call counts, not kwargs, so no test changes were needed.