[fix](fe) Fix broken pipe risk on stream load redirect with unconsumed request body#63381
Closed
wenzhenghu wants to merge 23 commits into
Closed
[fix](fe) Fix broken pipe risk on stream load redirect with unconsumed request body#63381wenzhenghu wants to merge 23 commits into
wenzhenghu wants to merge 23 commits into
Conversation
* Fix BE UT core dump. * Enhance comments for pointer assignment logic Add comment explaining conditional check for pointer assignment * Add comments. --------- Co-authored-by: Wen Zhenghu <wenzhenghu.zju@gmail.com>
…are mixed in VFileScanner (#30) * [fix](scan) Fix missing predicate filter when Native and JNI readers are mixed in VFileScanner * code format * fix be ut test
…Catalog interface (#38)
Add the missing java.net.URI import to fix FE compilation in RestBaseController.
Handle IOException in stream load forward redirect response generation. Use a bounded idle time window for redirect drain and add delayed-arrival regression coverage.
Remove the RedirectView status getter assertion from LoadActionTest because the current Spring version does not expose getStatusCode().
Add a configurable idle wait window for stream load redirect drain. Refactor the repeated idle wait path, return INTERRUPTED on sleep interruption, and update FE unit tests accordingly.
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Increase the default bounded drain byte limit to 1 GB and the default idle wait window to 1 second for stream load redirect handling, and keep FE tests aligned with the new defaults.
### Release note
Adjust the default FE stream load redirect drain limits to 1 GB and 1 second.
### Check List (For Author)
- Test: Manual check
- No need to test (default value adjustment and FE test reset update only)
- Behavior changed: Yes (the default stream load redirect bounded drain limits are larger)
- Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #63325
Problem Summary:
Problem
3.1.3, FE usesJetty 12, and this introduced a compatibility change in the Stream Load redirect path.307 Temporary Redirectbefore the request body is fully consumed. UnderJetty 12, this behavior is more likely to cause early connection close or reset while the client is still writing the request body.HTTP/1.1streaming clients may observe errors such asBrokenPipeErrororConnectionResetErrorwhen sending Stream Load requests through FE.Jetty 12upgrade in Doris3.1.3and later.Fix
307 Temporary Redirectresponse.1GBdrain limit and a1000msidle wait window.New Configurations
jetty_server_max_unconsumed_request_content_reads-1means unlimited,0use Jetty default value, and a positive value sets the maximum number of read attempts.0.Jetty 12upgrade when FE returns a response before the request body is fully consumed.stream_load_redirect_bounded_drain_max_bytes307for a Stream Load redirect.0disables this compatibility logic.1GB.stream_load_redirect_bounded_drain_max_idle_time_ms0disables the extra idle wait.1000ms.Test Result / Validation
HTTP/1.1chunked Stream Load reproduction used during issue analysis.Expect: 100-continue,Transfer-Encoding: chunked, and paced body streaming to maximize the redirect race window.3.0(9030/ FE8030):payload_mb=1,chunk_kb=1,sleep_ms=0payload_mb=8,chunk_kb=16,sleep_ms=10307 Temporary Redirect.3.1.4instance (9034/ FE8034):BrokenPipeError.jetty_server_max_unconsumed_request_content_reads = -1stream_load_redirect_bounded_drain_max_bytes = 16777216stream_load_redirect_bounded_drain_max_idle_time_ms = 1000307 Temporary Redirect.BrokenPipeErrororConnectionResetErrorwas observed after the config took effect.16MBto1GB, while keeping the default idle wait at1000ms, so the compatibility path is enabled by default with a more generous drain window.Release note
None
Check List (For Author)
Test
Behavior changed:
1GB.Does this need documentation?
Check List (For Reviewer who merge this PR)