Use a load-aware balancer #251
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently, the conduit proxy uses a simplistic Round-Robin load
balancing algorithm. This strategy degrades severely when individual
endpoints exhibit abnormally high latency.
This change improves this situation somewhat by making the load balancer
aware of the number of outstanding requests to each endpoint. When nodes
exhibit high latency, they should tend to have more pending requests
than faster nodes; and the Power-of-Two-Choices node selector can be
used to distribute requests to lesser-loaded instances.
From the finagle guide:
The algorithm randomly picks two nodes from the set of ready endpoints
and selects the least loaded of the two. By repeatedly using this
strategy, we can expect a manageable upper bound on the maximum load of
any server.
The maximum load variance between any two servers is bound by
ln(ln(n))` where `n` is the number of servers in the cluster.
Signed-off-by: Oliver Gould <ver@buoyant.io>
|
I know that conduit is supposed to be zero config. Does this mean that conduit users are always going to have to stick with the |
|
Great question, @deebo91! Power of two choices with the least loaded metrics (P2C+LL) is a very good general purpose load balancing algorithm that works pretty well for most kinds of traffic. So I think it's a great default. In the future, it would be really cool to see Conduit dynamically picking an LB algorithm depending on the nature of the traffic it sees. For example, EWMA+Latency for unary requests, P2C+LL for streaming requests, aperture when the load doesn't match the pool size, etc. But in general, it should be possible to intelligently make these determinations based on the live data instead of needing users to configure it. |
|
That sounds like a really good feature! Aligns with the "just works" philosophy. Thanks for clarifying @adleong! |
|
ran a little test: world is made of 9 "earth" pods with no added latency, and a single "mars" pod with 2s of added latency. p2c + pending requests (this branch) round robin (v0.2.0): |
|
Sorry for the noob question, but how I understand that results? |
|
@seanmonstar The big differentiator appears to be the improvement in p95 latency, from 2067ms to 84ms, on average. The latency columns in the output are Another useful metric to look at is |
|
For posterity, here's the base k8s configuration i used to test the load balancing behavior https://gist.github.com/olix0r/16006b1dd98fd43221820181d36293d9 |
seanmonstar
approved these changes
Feb 7, 2018
|
@seanmonstar i think the destination service should be extended to support weights; and once we've implemented that, we can introduce weighted balancing into the proxy. |
khappucino
pushed a commit
to Nordstrom/linkerd2
that referenced
this pull request
Mar 5, 2019
Currently, the conduit proxy uses a simplistic Round-Robin load
balancing algorithm. This strategy degrades severely when individual
endpoints exhibit abnormally high latency.
This change improves this situation somewhat by making the load balancer
aware of the number of outstanding requests to each endpoint. When nodes
exhibit high latency, they should tend to have more pending requests
than faster nodes; and the Power-of-Two-Choices node selector can be
used to distribute requests to lesser-loaded instances.
From the finagle guide:
The algorithm randomly picks two nodes from the set of ready endpoints
and selects the least loaded of the two. By repeatedly using this
strategy, we can expect a manageable upper bound on the maximum load of
any server.
The maximum load variance between any two servers is bound by
ln(ln(n))` where `n` is the number of servers in the cluster.
Signed-off-by: Oliver Gould <ver@buoyant.io>
olix0r
added a commit
that referenced
this pull request
May 16, 2019
commit b27dfb2d21aa8ca5466ea0edce17d27094ace7c1
Author: Takanori Ishibashi <takanori.1112@gmail.com>
Date: Wed May 15 05:58:42 2019 +0900
updaes->updates (#250)
Signed-off-by: Takanori Ishibashi <takanori.1112@gmail.com>
commit 16441c25a9d423a6ab12b689b830d9ae3798fa00
Author: Eliza Weisman <eliza@buoyant.io>
Date: Tue May 14 14:40:03 2019 -0700
Pass router::Config directly to router::Layer (#253)
Currently, router `layer`s are constructed with a single argument, a
type implementing `Recognize`. Then, the entire router stack is built
with a `router::Config`. However, in #248, it became necessary to
provide the config up front when constructing the `router::layer`, as
the layer is used in a fallback layer. Rather than providing a separate
type for a preconfigured layer, @olix0r suggested we simply change all
router layers to accept the `Config` when they're constructed (see
linkerd/linkerd2-proxy#248 (comment)).
This branch changes `router::Layer` to accept the config up front. The
`router::Stack` types `make` function now requires no arguments, and the
implementation of `Service` for `Stack` can be called with any `T` (as
the target is now ignored).
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
commit b70c68d4504a362eac6a7828039a2e5c7fcd308a
Author: Eliza Weisman <eliza@buoyant.io>
Date: Wed May 15 13:14:04 2019 -0700
Load balancers fall back to ORIG_DST when no endpoints exist (#248)
Currently, when no endpoints exist in the load balancer for a
destination, we fail the request. This is because we expect endpoints to
be discovered by both destination service queries _and_ DNS lookups, so
if there are no endpoints for a destination, it is assumed to not exist.
In #2661, we intend to remove the DNS lookup from the
proxy and instead fall back to routing requests for which no endpoints
exist in the destination service to their SO_ORIGINAL_DST IP address.
This means that the current approach of failing requests when the load
balancer has no endpoints will no longer work.
This branch introduces a generic `fallback` layer, which composes a
primary and secondary service builder into a new layer. The primary
service can fail requests with an error type that propages the original
request, allowing the fallback middleware to call the fallback service
with the same request. Other errors returned by the primary service are
still propagated upstream.
In contrast to the approach used in #240, this fallback middleware is
generic and not tied directly to a load balancer or a router, and can
be used for other purposes in the future. It relies on the router cache
eviction added in #247 to drain the router when it is not being used,
rather than proactively destroying the router when endpoints are
available for the lb, and re-creating it when they exist again.
A new trait, `HasEndpointStatus`, is added in order to allow the
discovery lookup to communicate the "no endpoints" state to the
balancer. In addition, we add a new `Update::NoEndpoints` variant to
`proxy::resolve::Update`, so that when the control plane sends a no
endpoints update, we switch from the balancer to the no endpoints state
_immediately_, rather than waiting for all the endpoints to be
individually removed. When the balancer has no endpoints, it fails all
requests with a fallback error, so that the fallback middleware
A subsequent PR (#248) will remove the DNS lookups from the discovery
module.
Closes #240.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
commit 6525b0638ad18e74510f3156269e0613f237e2f5
Author: Zahari Dichev <zaharidichev@gmail.com>
Date: Wed May 15 23:35:09 2019 +0300
Allow disabling tap by setting an env var (#252)
This PR fixes #2811. Now if
`LINKERD2_PROXY_TAP_DISABLED` is set, the tap is not served at all. The
approach taken is that the `ProxyParts` is changed so the
`control_listener` is now an `Option` that will be None if tap is
disabled as this control_listener seems to be exclusively used to serve
the tap. Feel free to suggest a better approach.
Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>
commit 91f32db2ea6d74470fd689c713ff87dc7586222d
Author: Zahari Dichev <zaharidichev@gmail.com>
Date: Thu May 16 00:45:23 2019 +0300
Assert that outbound TLS works before identity is certified (#251)
This commit introduces TLS capabilities to the support server as well as
tests to ensure that outbound TLS works even when there is no verified
certificate for the proxy yet.
Fixes #2599
Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>
commit 45aadc6b1b28e6daea0c40e694a86ae518887d85
Author: Sean McArthur <sean@buoyant.io>
Date: Wed May 15 14:25:39 2019 -0700
Update h2 to v0.1.19
Includes a couple HPACK fixes
Signed-off-by: Sean McArthur <sean@buoyant.io>
commit 3e0e00c6dfbf5a9155b887cfd594f611edfc135f
Author: Oliver Gould <ver@buoyant.io>
Date: Thu May 16 08:11:06 2019 -0700
Update mio to 0.6.17 (#257)
To pick up tokio-rs/mio#939
olix0r
added a commit
that referenced
this pull request
May 16, 2019
commit b27dfb2d21aa8ca5466ea0edce17d27094ace7c1
Author: Takanori Ishibashi <takanori.1112@gmail.com>
Date: Wed May 15 05:58:42 2019 +0900
updaes->updates (#250)
Signed-off-by: Takanori Ishibashi <takanori.1112@gmail.com>
commit 16441c25a9d423a6ab12b689b830d9ae3798fa00
Author: Eliza Weisman <eliza@buoyant.io>
Date: Tue May 14 14:40:03 2019 -0700
Pass router::Config directly to router::Layer (#253)
Currently, router `layer`s are constructed with a single argument, a
type implementing `Recognize`. Then, the entire router stack is built
with a `router::Config`. However, in #248, it became necessary to
provide the config up front when constructing the `router::layer`, as
the layer is used in a fallback layer. Rather than providing a separate
type for a preconfigured layer, @olix0r suggested we simply change all
router layers to accept the `Config` when they're constructed (see
linkerd/linkerd2-proxy#248 (comment)).
This branch changes `router::Layer` to accept the config up front. The
`router::Stack` types `make` function now requires no arguments, and the
implementation of `Service` for `Stack` can be called with any `T` (as
the target is now ignored).
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
commit b70c68d4504a362eac6a7828039a2e5c7fcd308a
Author: Eliza Weisman <eliza@buoyant.io>
Date: Wed May 15 13:14:04 2019 -0700
Load balancers fall back to ORIG_DST when no endpoints exist (#248)
Currently, when no endpoints exist in the load balancer for a
destination, we fail the request. This is because we expect endpoints to
be discovered by both destination service queries _and_ DNS lookups, so
if there are no endpoints for a destination, it is assumed to not exist.
In #2661, we intend to remove the DNS lookup from the
proxy and instead fall back to routing requests for which no endpoints
exist in the destination service to their SO_ORIGINAL_DST IP address.
This means that the current approach of failing requests when the load
balancer has no endpoints will no longer work.
This branch introduces a generic `fallback` layer, which composes a
primary and secondary service builder into a new layer. The primary
service can fail requests with an error type that propages the original
request, allowing the fallback middleware to call the fallback service
with the same request. Other errors returned by the primary service are
still propagated upstream.
In contrast to the approach used in #240, this fallback middleware is
generic and not tied directly to a load balancer or a router, and can
be used for other purposes in the future. It relies on the router cache
eviction added in #247 to drain the router when it is not being used,
rather than proactively destroying the router when endpoints are
available for the lb, and re-creating it when they exist again.
A new trait, `HasEndpointStatus`, is added in order to allow the
discovery lookup to communicate the "no endpoints" state to the
balancer. In addition, we add a new `Update::NoEndpoints` variant to
`proxy::resolve::Update`, so that when the control plane sends a no
endpoints update, we switch from the balancer to the no endpoints state
_immediately_, rather than waiting for all the endpoints to be
individually removed. When the balancer has no endpoints, it fails all
requests with a fallback error, so that the fallback middleware
A subsequent PR (#248) will remove the DNS lookups from the discovery
module.
Closes #240.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
commit 6525b0638ad18e74510f3156269e0613f237e2f5
Author: Zahari Dichev <zaharidichev@gmail.com>
Date: Wed May 15 23:35:09 2019 +0300
Allow disabling tap by setting an env var (#252)
This PR fixes #2811. Now if
`LINKERD2_PROXY_TAP_DISABLED` is set, the tap is not served at all. The
approach taken is that the `ProxyParts` is changed so the
`control_listener` is now an `Option` that will be None if tap is
disabled as this control_listener seems to be exclusively used to serve
the tap. Feel free to suggest a better approach.
Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>
commit 91f32db2ea6d74470fd689c713ff87dc7586222d
Author: Zahari Dichev <zaharidichev@gmail.com>
Date: Thu May 16 00:45:23 2019 +0300
Assert that outbound TLS works before identity is certified (#251)
This commit introduces TLS capabilities to the support server as well as
tests to ensure that outbound TLS works even when there is no verified
certificate for the proxy yet.
Fixes #2599
Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>
commit 45aadc6b1b28e6daea0c40e694a86ae518887d85
Author: Sean McArthur <sean@buoyant.io>
Date: Wed May 15 14:25:39 2019 -0700
Update h2 to v0.1.19
Includes a couple HPACK fixes
Signed-off-by: Sean McArthur <sean@buoyant.io>
commit 3e0e00c6dfbf5a9155b887cfd594f611edfc135f
Author: Oliver Gould <ver@buoyant.io>
Date: Thu May 16 08:11:06 2019 -0700
Update mio to 0.6.17 (#257)
To pick up tokio-rs/mio#939
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, the conduit proxy uses a simplistic Round-Robin load
balancing algorithm. This strategy degrades severely when individual
endpoints exhibit abnormally high latency.
This change improves this situation somewhat by making the load balancer
aware of the number of outstanding requests to each endpoint. When nodes
exhibit high latency, they should tend to have more pending requests
than faster nodes; and the Power-of-Two-Choices node selector can be
used to distribute requests to lesser-loaded instances.
From the finagle guide: