-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
connection: Remember transport socket read resumption requests and replay them when re-enabling read. #13772
connection: Remember transport socket read resumption requests and replay them when re-enabling read. #13772
Conversation
…play them when re-enabling read. Signed-off-by: Antonio Vicente <avd@google.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice, I like this way better.
High level thought - do we want to runtime guard this just because it's high risk? Mild concerns about the complexity here opening up a spin loop bug.
@@ -554,6 +558,7 @@ void ConnectionImpl::onReadReady() { | |||
return; | |||
} | |||
|
|||
want_read_ = false; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we also set this false for the early return onRead above? If not please comment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Commented.
MockTransportConnectionImplTest.ResumeWhileAndAfterReadDisable covers the early return case which requires us to clear want_read_ just before doRead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry how is want_read_ cleared before onRead above? Maybe update the comment above also as to why we don't need to clear or how it's already cleared?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added comment.
Signed-off-by: Antonio Vicente <avd@google.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice, I like this way better.
High level thought - do we want to runtime guard this just because it's high risk? Mild concerns about the complexity here opening up a spin loop bug.
Let's talk about the risk level of this PR. The case I'm adding resumption for is after a call to readDisable(false). That can happen when either:
- High watermark trigger cleared from read buffer as bytes are drained
- High watermark trigger cleared from write buffer as bytes are drained
- Finished processing of an H1 request from the client
All of these count as proofs of work. It doesn't seem likely that the connection will continuously wake up when there's no available work.
If for instance we were to not clear want_read_ when we should, we'ld get a spurious wakeup on readDisable(false), but only once unless the connection is able to successfully read additional bytes from the socket and push them to the filter pipeline.
I agree that this PR increases complexity as it requires that the connection track additional state about the state of the transport and increases coupling between the connection and transport implementation.
I think the risk level of this PR is relatively low. That said, I have no objections to runtime guarding this change as an extra precaution.
Thanks for the prompt review.
@@ -554,6 +558,7 @@ void ConnectionImpl::onReadReady() { | |||
return; | |||
} | |||
|
|||
want_read_ = false; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Commented.
MockTransportConnectionImplTest.ResumeWhileAndAfterReadDisable covers the early return case which requires us to clear want_read_ just before doRead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks this is much better agreed! I haven't looked at the tests but the change looks like the right shape. Happy to take another pass once the comments are fleshed out a bit.
/wait
setReadBufferReady(); | ||
file_event_->activate(Event::FileReadyType::Read); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For clarity I might revert this change, even though it redundantly sets want_read_, but nbd either way. Up to you.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reverted.
@@ -554,6 +558,7 @@ void ConnectionImpl::onReadReady() { | |||
return; | |||
} | |||
|
|||
want_read_ = false; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry how is want_read_ cleared before onRead above? Maybe update the comment above also as to why we don't need to clear or how it's already cleared?
Signed-off-by: Antonio Vicente <avd@google.com>
Signed-off-by: Antonio Vicente <avd@google.com>
Signed-off-by: Antonio Vicente <avd@google.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks this LGTM modulo potential comment on naming. I will defer to @alyssawilk for final review. Thank you for iteration this is much cleaner!
// be read disabled. To handle these cases, fake an event to make sure the buffered data gets | ||
// processed regardless and ensure that we dispatch it via onRead. | ||
if ((consumerWantsToRead() && read_buffer_.length() > 0) || | ||
(read_disable_count_ == 0 && want_read_)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry one other question from another look: is it possible to somehow fold all of this logic into consumerWantsToRead()
? It seems kind of odd that we have a function that is tracking the consumer wants to read and then a different wants to read boolean. If it's too hard to merge them maybe some better names for both the function and the boolean?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tried to improve on names and the if predicate.
Signed-off-by: Antonio Vicente <avd@google.com>
…ant_read Signed-off-by: Antonio Vicente <avd@google.com>
The change looks good and I think is safe. My paranoid self tells me the flag protecting it is a good idea though. I think we just need to flag protect setting the |
EXPECT_CALL(*file_event_, activate(Event::FileReadyType::Read)); | ||
connection_->readDisable(false); | ||
|
||
// Do a read to clear the want_read_ flag, verify that no read activation is scheduled. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
want_read_
-> transport_wants_read_
here and other places below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. Nice catch.
Signed-off-by: Antonio Vicente <avd@google.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks LGTM with one small question.
// Sanity check: resumption with read_disable_count_ > 0 should only happen if the read | ||
// buffer's high watermark has triggered. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Stupid question: why do we bother with this extra logic to read when the disable count is still 1 and high watermark triggered? Is the idea that we already have the data so we might as well flush it? If the high watermark triggered it's probably going to be stuck on the other side, so mostly just wondering if this extra logic is actually worth it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is part of the implementation of high-watermarks on the read buffer introduced in PR #11170
The parser is able to consume input from the read buffer, we just don't want to read additional bytes from the transport into the read buffer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So the idea here is that we allow reading data out in the hope that it will get us below the low watermark? OK thanks that makes sense.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes.
The common scenario is when the H1 parser stops consuming bytes from the read buffer and calls readDisable(true) once it detects the end of the current request message. When request processing is complete, the H1 parser re-enables read. When H1 pipelining is in use, a full second request message could be in the read buffer and the read buffer may have triggered the high-watermark.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Make sense, though note that we don't do pipelining.
/retest |
Retrying Azure Pipelines: |
/retest giving macos one last chance. |
Retrying Azure Pipelines: |
* master: (83 commits) tls: Typesafe tls slots (envoyproxy#13789) docs(example): Correct URL for caching example page (envoyproxy#13810) [fuzz] Made health check fuzz more efficient (envoyproxy#13747) rtds: properly scope rtds stats (envoyproxy#13764) http: fixing a bug with IPv6 hosts (envoyproxy#13798) connection: Remember transport socket read resumption requests and replay them when re-enabling read. (envoyproxy#13772) network: adding some accessors for ALPN work. (envoyproxy#13785) docs: added a step about how to handle platform specific extensions (envoyproxy#13759) Fix identation in ip transparency code snippet (envoyproxy#13743) wasm: enable WAVM's stack unwinding feature (envoyproxy#13792) log: set route name for direct response (envoyproxy#13683) Use nghttp2 as external dependsncy in protocol_constraints_lib (envoyproxy#13763) [Windows] Update windows dev docs (envoyproxy#13741) cel: patch thread safety issue (envoyproxy#13739) Windows: Fix ssl_socket_test (envoyproxy#13264) apple dns: add fake api test suite (envoyproxy#13780) overload: scale selected timers in response to load (envoyproxy#13475) examples: Add dynamic configuration (control plane) sandbox (envoyproxy#13746) Removed exception in getResponseStatus() (envoyproxy#13314) network: add timeout for transport connect (envoyproxy#13610) ... Signed-off-by: Michael Puncel <mpuncel@squareup.com>
/backport (carried from the issue #12304) |
@cpakulski anything with |
…sts and replay them when re-enabling read. (envoyproxy#13772) Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message Signed-off-by: Antonio Vicente <avd@google.com> Signed-off-by: antonio <avd@google.com> Signed-off-by: Christoph Pakulski <christoph@tetrate.io>
…on requests and replay them when re-enabling read. (#13772) (#14017) Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message Risk Level: low Testing: new unit and integration tests Docs Changes: n/a Release Notes: added Platform Specific Features: n/a Fixes #12304 Signed-off-by: Antonio Vicente <avd@google.com> Signed-off-by: Christoph Pakulski <christoph@tetrate.io>
…play them when re-enabling read. (envoyproxy#13772) Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message Signed-off-by: Antonio Vicente <avd@google.com> Signed-off-by: antonio <avd@google.com> Signed-off-by: Christoph Pakulski <christoph@tetrate.io>
…play them when re-enabling read. (envoyproxy#13772) Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message Signed-off-by: Antonio Vicente <avd@google.com>
…ion requests and replay them when re-enabling read. (#13772) (#14173) * connection: Remember transport socket read resumption requests and replay them when re-enabling read. (#13772) Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message Signed-off-by: Antonio Vicente <avd@google.com>
…ion requests and replay them when re-enabling read. (envoyproxy#13772) (envoyproxy#14173) Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message Signed-off-by: Antonio Vicente <avd@google.com>
…ion requests and replay them when re-enabling read. (envoyproxy#13772) (envoyproxy#14173) Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message Signed-off-by: Antonio Vicente <avd@google.com>
Backported to releases 1.13, 1.14, 1.15 and 1.16. Removing |
* docs: kick-off 1.15.1 release. (envoyproxy#12166) Signed-off-by: Piotr Sikora <piotrsikora@google.com> * tls: update BoringSSL-FIPS to 20190808. (envoyproxy#12170) Signed-off-by: Piotr Sikora <piotrsikora@google.com> * test: Exclude wasm_vm_test from CI by making it a "manual" test. (#207) The wee v8 build times out in CI under --config=asan because the machine the job is scheduled on is too small. Signed-off-by: Antonio Vicente <avd@google.com> * [v1.15] http: header map security fixes for duplicate headers (#197) (#200) Previously header matching did not match on all headers for non-inline headers. This patch changes the default behavior to always logically match on all headers. Multiple individual headers will be logically concatenated with ',' similar to what is done with inline headers. This makes the behavior effectively consistent. This behavior can be temporary reverted by setting the runtime value "envoy.reloadable_features.header_match_on_all_headers" to "false". Targeted fixes have been additionally performed on the following extensions which make them consider all duplicate headers by default as a comma concatenated list: 1) Any extension using CEL matching on headers. 2) The header to metadata filter. 3) The JWT filter. 4) The Lua filter. Like primary header matching used in routing, RBAC, etc. this behavior can be disabled by setting the runtime value "envoy.reloadable_features.header_match_on_all_headers" to false. Finally, the setCopy() header map API previously only set the first header in the case of duplicate non-inline headers. setCopy() now behaves similiarly to the other set*() APIs and replaces all found headers with a single value. This may have had security implications in the extauth filter which uses this API. This behavior can be disabled by setting the runtime value "envoy.reloadable_features.http_set_copy_replace_all_headers" to false. Fixes https://github.com/envoyproxy/envoy-setec/issues/188 Signed-off-by: Matt Klein <mklein@lyft.com> * backport to v1.15: Fix Kafka Repository Location (#223) Update mirror used to fetch kafka dependency to a valid, working mirror. Based on envoyproxy#13025 Resolves envoyproxy#13011 Signed-off-by: Antonio Vicente <avd@google.com> * release: cutting 1.15.1 (#217) Signed-off-by: Antonio Vicente <avd@google.com> * docs: Fix release notes for v1.15.1 release. (envoyproxy#13318) Signed-off-by: Antonio Vicente <avd@google.com> * Backport flaky test and tsan fixes to releases/v1.15 branch (envoyproxy#13337) * hds: fix integration test flakes (envoyproxy#12214) Part of envoyproxy#12184 Signed-off-by: Matt Klein <mklein@lyft.com> Signed-off-by: Antonio Vicente <avd@google.com> * Switch to a tsan-instrumented libc++ for tsan tests (envoyproxy#12134) This fixes envoyproxy#9784 and re-enables vhds_integration_test Risk Level: Low, but will most likely increase memory usage Signed-off-by: Dmitri Dolguikh <ddolguik@redhat.com> Signed-off-by: Antonio Vicente <avd@google.com> * test: shard hds_integration_test (envoyproxy#12482) This should avoid TSAN timeout flakes. Signed-off-by: Matt Klein <mklein@lyft.com> Signed-off-by: Antonio Vicente <avd@google.com> * test: shard http2_integration_test (envoyproxy#11939) This should mitigate TSAN timeout. Signed-off-by: Lizan Zhou <lizan@tetrate.io> Signed-off-by: Antonio Vicente <avd@google.com> * test: fix http2_integration_test flake (envoyproxy#12450) Fixes envoyproxy#12442 Signed-off-by: Matt Klein <mklein@lyft.com> Signed-off-by: Antonio Vicente <avd@google.com> * Kick CI Signed-off-by: Antonio Vicente <avd@google.com> Co-authored-by: Matt Klein <mklein@lyft.com> Co-authored-by: Dmitri Dolguikh <ddolguik@redhat.com> Co-authored-by: Lizan Zhou <lizan@tetrate.io> * docs: kick off v1.15.3-dev (envoyproxy#13695) Signed-off-by: Christoph Pakulski <christoph@tetrate.io> * 1.15: CI fixes backport (envoyproxy#13697) Backport following commits to 1.15: 748b2ab (mac ci: try ignoring update failure (envoyproxy#13658), 2020-10-20) f95f539 (ci: various improvements (envoyproxy#13660), 2020-10-20) 73d78f8 (ci: use multiple stage (envoyproxy#13557), 2020-10-15) b7a4756 (ci: use azp for api and go-control-plane sync (envoyproxy#13550), 2020-10-14) 876a6bb (ci use azp to sync filter example (envoyproxy#13501), 2020-10-12) a0f31ee (ci: use azp to generate docs (envoyproxy#13481), 2020-10-12) Signed-off-by: Lizan Zhou <lizan@tetrate.io> Co-authored-by: asraa <asraa@google.com> * 1.15: fix CI script (envoyproxy#13724) Signed-off-by: Lizan Zhou <lizan@tetrate.io> * Prevent SEGFAULT when disabling listener (envoyproxy#13515) (envoyproxy#13903) This prevents the stop_listening overload action from causing segmentation faults that can occur if the action is enabled after the listener has already shut down. Signed-off-by: Alex Konradi <akonradi@google.com> Signed-off-by: Christoph Pakulski <christoph@tetrate.io> * proxy protocol: set downstreamRemoteAddress on StreamInfo (envoyproxy#14131) (envoyproxy#14169) This fixes a regression which resulted in the downstreamRemoteAddress on the StreamInfo for a connection not having the address supplied by the proxy protocol filter, but instead having the address of the directly connected peer. This issue does not affect HTTP filters. Fixes envoyproxy#14087 Signed-off-by: Greg Greenway <ggreenway@apple.com> Signed-off-by: Christoph Pakulski <christoph@tetrate.io> * ci: temproray disable vhds_integration_test in TSAN (envoyproxy#12067) (envoyproxy#14217) Signed-off-by: Lizan Zhou <lizan@tetrate.io> * tcmalloc changed and the data coming out of tcmalloc::MallocExtension::GetNumericProperty("generic.current_allocated_bytes") (envoyproxy#14165) Commit Message: tcmalloc changed and the data coming out of tcmalloc::MallocExtension::GetNumericProperty("generic.current_allocated_bytes") no longer appears to be deterministic, even in unthreaded tests. So disable exact mem checks till we sort that out Additional Description: Risk Level: low Testing: just thread_local_store_test Docs Changes: n/a Release Notes: n/a no longer appears to be deterministic, even in unthreaded tests. So disable exact mem checks till we sort that out Signed-off-by: Joshua Marantz <jmarantz@google.com> Signed-off-by: Christoph Pakulski <christoph@tetrate.io> Co-authored-by: Joshua Marantz <jmarantz@google.com> * backport to v1.15: connection: Remember transport socket read resumption requests and replay them when re-enabling read. (envoyproxy#13772) (envoyproxy#14173) * connection: Remember transport socket read resumption requests and replay them when re-enabling read. (envoyproxy#13772) Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message Signed-off-by: Antonio Vicente <avd@google.com> * backport to 1.15: udp: properly handle truncated/dropped datagrams (envoyproxy#14122) (envoyproxy#14166) Signed-off-by: Matt Klein <mklein@lyft.com> Signed-off-by: Christoph Pakulski <christoph@tetrate.io> Co-authored-by: Matt Klein <mklein@lyft.com> Co-authored-by: Christoph Pakulski <christoph@tetrate.io> * backport to 1.15: vrp: allow supervisord to open its log file (envoyproxy#14066) (envoyproxy#14280) Commit Message: Allow supervisord to open its log file Additional Description: Change the default location of the log file and give supervisord permissions to write to it. Risk Level: low Testing: built image locally Docs Changes: n/a Release Notes: n/a Platform Specific Features: n/a Signed-off-by: Alex Konradi <akonradi@google.com> Signed-off-by: Christoph Pakulski <christoph@tetrate.io> * rel 1.15: close release 1.15.3 (envoyproxy#14303) Signed-off-by: Christoph Pakulski <christoph@tetrate.io> * Kick off rel 1.15.4. (envoyproxy#14323) Signed-off-by: Christoph Pakulski <christoph@tetrate.io> * backport to 1.15: http: fix datadog and squash handling of responses without body (envoyproxy#13328) (envoyproxy#14458) Commit Message: Fixing bugs in datadog and sqaush where unexpected bodyless responses would crash Envoy Risk Level: low Testing: new unit tests, updated certs Docs Changes: n/a Release Notes: inline Signed-off-by: Christoph Pakulski <christoph@tetrate.io> Co-authored-by: alyssawilk <alyssar@chromium.org> * backport 1.15: http: fixing a bug with IPv6 hosts (envoyproxy#14273) Fixing a bug where HTTP parser offsets for IPv6 hosts did not include [] and Envoy assumed it did. This results in mis-parsing addresses for IPv6 CONNECT requests and IPv6 hosts in fully URLs over HTTP/1.1 Risk Level: low Testing: new unit, integration tests Docs Changes: n/a Release Notes: inline Signed-off-by: Shikugawa <rei@tetrate.io> Co-authored-by: alyssawilk <alyssar@chromium.org> * backport to 1.15: tls: fix detection of the upstream connection close event. (envoyproxy#13858) (envoyproxy#14568) Fixes envoyproxy#13856. Signed-off-by: Piotr Sikora <piotrsikora@google.com> Signed-off-by: Christoph Pakulski <christoph@tetrate.io> Co-authored-by: Piotr Sikora <piotrsikora@google.com> Co-authored-by: antonio <avd@google.com> Co-authored-by: Matt Klein <mklein@lyft.com> Co-authored-by: Dmitri Dolguikh <ddolguik@redhat.com> Co-authored-by: Lizan Zhou <lizan@tetrate.io> Co-authored-by: Christoph Pakulski <christoph@tetrate.io> Co-authored-by: asraa <asraa@google.com> Co-authored-by: Joshua Marantz <jmarantz@google.com> Co-authored-by: Rei Shimizu <Shikugawa@gmail.com> Co-authored-by: alyssawilk <alyssar@chromium.org>
Commit Message: connection: Remember transport socket read resumption requests and replay them when re-enabling read.
Additional Description: Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message
Risk Level: low
Testing: new unit and integration tests
Docs Changes: n/a
Release Notes: added
Platform Specific Features: n/a
Fixes #12304