ingress: Add failfast to the forwarder #1035
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The ingress-mode proxy's forwarding stack--used when a request does not set the `l5d-dst-override` header--has no failfast implemetation. This means that when a connection can't be obtained for the endpoint, requests are buffered indefinitely. This change adds a failfast layer so that these requests are failed eagerly after 3s of unavailability, causing the serverside connection to be dropped (so that the application client may re-resolve the endpoint). This is really a temporary solution. We should probably avoid implementing reconnection at all in this case so that connection errors can be used in place of failfast errors. Related to linkerd/linkerd2#6184
hawkw
approved these changes
Jun 7, 2021
olix0r
added a commit
to linkerd/linkerd2
that referenced
this pull request
Jun 9, 2021
This release fixes a problem with the HTTP body buffering that was added to support gRPC retries. The proxy would buffer all request bodies, regardless of size or retry configurations. This has been fixed so that only requests with a retry configuration are buffered (and only when their bodies are less than 64KB). This release also fixes an issue with the outbound ingress-mode proxy where forwarded HTTP traffic could fail to detect when the target pod was deleted, retrying connections forever. This only impacted traffic forwarded directly to pod IPs (and not load balanced services). This has been fixed temporarily by adding a failfast layer that triggers 502 errors when the endpoint has disconected, which cause the connection to be torn down so that the ingress may reconnect. A more robust solution will replace this in the future. Furthermore, core dependencies have been updated including: futures, hyper, socket2, and tokio. --- * Fix MacOS conditional build in telemetry::process (linkerd/linkerd2-proxy#1023) * deps: update `futures` to 0.3.15 (linkerd/linkerd2-proxy#1022) * tracing: Split HTML-formatting into admin module (linkerd/linkerd2-proxy#1025) * tracing: Simplify initialization (linkerd/linkerd2-proxy#1026) * Repace linkerd-drain with drain from crates.io (linkerd/linkerd2-proxy#1027) * app: Move the admin server into a subcrate (linkerd/linkerd2-proxy#1028) * inbound: Simplify protocol-detection skipping (linkerd/linkerd2-proxy#1031) * proxy-api: Update proxy-api to use the main branch (linkerd/linkerd2-proxy#1029) * outbound: don't double-wrap replay bodies (linkerd/linkerd2-proxy#1036) * ingress: Add failfast to the forwarder (linkerd/linkerd2-proxy#1035) * Update tokio, hyper, and socket2 (linkerd/linkerd2-proxy#1037) * Implement reconnect as a NewService (linkerd/linkerd2-proxy#1032) * Introduce the tonic-watch crate (linkerd/linkerd2-proxy#1034) * service-profiles: Wrap receiver types (linkerd/linkerd2-proxy#1038) * retry: only wrap bodies when a request can be retried (linkerd/linkerd2-proxy#1039)
Pothulapati
pushed a commit
to linkerd/linkerd2
that referenced
this pull request
Jun 10, 2021
This release fixes a problem with the HTTP body buffering that was added to support gRPC retries. The proxy would buffer all request bodies, regardless of size or retry configurations. This has been fixed so that only requests with a retry configuration are buffered (and only when their bodies are less than 64KB). This release also fixes an issue with the outbound ingress-mode proxy where forwarded HTTP traffic could fail to detect when the target pod was deleted, retrying connections forever. This only impacted traffic forwarded directly to pod IPs (and not load balanced services). This has been fixed temporarily by adding a failfast layer that triggers 502 errors when the endpoint has disconected, which cause the connection to be torn down so that the ingress may reconnect. A more robust solution will replace this in the future. Furthermore, core dependencies have been updated including: futures, hyper, socket2, and tokio. --- * Fix MacOS conditional build in telemetry::process (linkerd/linkerd2-proxy#1023) * deps: update `futures` to 0.3.15 (linkerd/linkerd2-proxy#1022) * tracing: Split HTML-formatting into admin module (linkerd/linkerd2-proxy#1025) * tracing: Simplify initialization (linkerd/linkerd2-proxy#1026) * Repace linkerd-drain with drain from crates.io (linkerd/linkerd2-proxy#1027) * app: Move the admin server into a subcrate (linkerd/linkerd2-proxy#1028) * inbound: Simplify protocol-detection skipping (linkerd/linkerd2-proxy#1031) * proxy-api: Update proxy-api to use the main branch (linkerd/linkerd2-proxy#1029) * outbound: don't double-wrap replay bodies (linkerd/linkerd2-proxy#1036) * ingress: Add failfast to the forwarder (linkerd/linkerd2-proxy#1035) * Update tokio, hyper, and socket2 (linkerd/linkerd2-proxy#1037) * Implement reconnect as a NewService (linkerd/linkerd2-proxy#1032) * Introduce the tonic-watch crate (linkerd/linkerd2-proxy#1034) * service-profiles: Wrap receiver types (linkerd/linkerd2-proxy#1038) * retry: only wrap bodies when a request can be retried (linkerd/linkerd2-proxy#1039)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The ingress-mode proxy's forwarding stack--used when a request does not
set the
l5d-dst-overrideheader--has no failfast implemetation. Thismeans that when a connection can't be obtained for the endpoint,
requests are buffered indefinitely.
This change adds a failfast layer so that these requests are failed
eagerly after 3s of unavailability, causing the serverside connection to
be dropped (so that the application client may re-resolve the endpoint).
This is really a temporary solution. We should probably avoid
implementing reconnection at all in this case so that connection errors
can be used in place of failfast errors.
Related to linkerd/linkerd2#6184