Health check by rpc call #694
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zyearn
commented
Mar 20, 2019
TousakaRin
reviewed
Mar 22, 2019
jamesge
requested changes
Mar 22, 2019
zyearn
force-pushed
the
health_check_by_rpc_call
branch
from
5281b2e
to
c407640
Compare
gydong
reviewed
Mar 24, 2019
gydong
reviewed
Mar 24, 2019
gydong
reviewed
Mar 24, 2019
gydong
reviewed
Mar 25, 2019
…el_health_checking & use Address in OnHealthCheckRPCDone
jamesge
requested changes
Mar 26, 2019
docs/cn/client.md
Outdated
| @@ -242,7 +242,7 @@ locality-aware，优先选择延时低的下游，直到其延时高于其他机 | |||
| | ------------------------- | ----- | ---------------------------------------- | ----------------------- | | |||
| | health_check_interval （R） | 3 | seconds between consecutive health-checkings | src/brpc/socket_map.cpp | | |||
|
|
|||
| 一旦server被连接上，它会恢复为可用状态。如果在隔离过程中，server从命名服务中删除了，brpc也会停止连接尝试。 | |||
| 在默认的配置下，一旦server被连接上，它会恢复为可用状态；brpc还提供了应用层健康检查的机制，协议是Http，只有当Server返回200时，这个server才算恢复，可以通过把-health\_check\_path设置成被检查的路径来打开这个功能（如果下游也是brpc，推荐设置成/health，服务健康的话会返回200），-health\_check\_timeout\_ms设置超时（默认500ms）。如果在隔离过程中，server从命名服务中删除了，brpc也会停止连接尝试。 | |||
There was a problem hiding this comment.
说-health_check_path对应的http service要求输入是什么，输出是什么
docs/cn/client.md
Outdated
| @@ -242,7 +242,7 @@ locality-aware，优先选择延时低的下游，直到其延时高于其他机 | |||
| | ------------------------- | ----- | ---------------------------------------- | ----------------------- | | |||
| | health_check_interval （R） | 3 | seconds between consecutive health-checkings | src/brpc/socket_map.cpp | | |||
|
|
|||
| 一旦server被连接上，它会恢复为可用状态。如果在隔离过程中，server从命名服务中删除了，brpc也会停止连接尝试。 | |||
| 在默认的配置下，一旦server被连接上，它会恢复为可用状态；brpc还提供了应用层健康检查的机制，协议是Http，只有当Server返回200时，这个server才算恢复，可以通过把-health\_check\_path设置成被检查的路径来打开这个功能（如果下游也是brpc，推荐设置成/health，服务健康的话会返回200），-health\_check\_timeout\_ms设置超时（默认500ms）。如果在隔离过程中，server从命名服务中删除了，brpc也会停止连接尝试。 | |||
There was a problem hiding this comment.
health_check_timeout_ms需要对Connect也生效么？
src/brpc/controller.h
Outdated
| @@ -118,6 +118,8 @@ friend int StreamCreate(StreamId*, Controller&, const StreamOptions*); | |||
| friend int StreamAccept(StreamId*, Controller&, const StreamOptions*); | |||
| friend void policy::ProcessMongoRequest(InputMessageBase*); | |||
| friend void policy::ProcessThriftRequest(InputMessageBase*); | |||
| friend class OnAppHealthCheckDone; | |||
| friend class HealthCheckManager; | |||
There was a problem hiding this comment.
用PrivateAccessor，不要加friend class
| @@ -122,7 +122,8 @@ int DynPartLoadBalancer::SelectServer(const SelectIn& in, SelectOut* out) { | |||
| for (size_t i = 0; i < n; ++i) { | |||
| const SocketId id = s->server_list[i].id; | |||
| if ((!exclusion || !ExcludedServers::IsExcluded(in.excluded, id)) | |||
| && Socket::Address(id, &ptrs[nptr].first) == 0) { | |||
| && Socket::Address(id, &ptrs[nptr].first) == 0 | |||
| && !(*out->ptr)->IsAppHealthCheck()) { | |||
jamesge
reviewed
Mar 26, 2019
src/brpc/socket.cpp
Outdated
| "By default health check succeeds if server can be connected. If this" | ||
| "flag is set, health check is completed not only when server can be" | ||
| "connected but also an additional http call succeeds indicated by this" | ||
| "flag and FLAGS_health_check_timeout_ms"); |
There was a problem hiding this comment.
if server can be connected -> if the server is connectable
not only...but also的意思是说hc不仅会在连接建立时完成，还会在http call成功时完成，也就是说会多次完成, 这个并不是正确的意思吧。应该说成，If this flag is set, health check is not completed until a http call to the path succeeds within -health_check_timeout_ms.
让我来写我还会加句提示" (to make sure the server functions well)"
另外一个细节是FLAGS_xxx是程序内部的说法，对于用户自然是-xxx而不是FLAGS_xxx
jamesge
requested changes
Mar 27, 2019
TousakaRin
reviewed
Mar 27, 2019
TousakaRin
reviewed
Mar 27, 2019
jamesge
reviewed
Mar 28, 2019
gydong
reviewed
Mar 28, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
By default health check succeeds if server can be connected. If this feature is used, health check is completed not only when server can be connected but also an additional http call succeeds indicated by FLAGS_health_check_path and FLAGS_health_check_timeout_ms
这个功能的改法是这样的，在健康检查时，默认如果connect成功server就恢复，而在health_check_path被设置时，socket在Revive()后会进入一个“用rpc进行hc”的状态，同时原来的HealthCheckTask结束，此时用户还是不能从lb中或者singleserver中选择这台机器。在这个状态时，如果rpc成功了，则恢复socket到正常状态；如果rpc失败了，只要socket不断，就会不停地间隔地发送rpc；如果rpc失败了且socket是Failed状态，则结束这个状态，之前的那一个SetFailed会触发下一次hc。
还需要区别是用户call还是hc call，前者无法选到socket，而后者可以正常选，从而发送rpc，这是通过在controller中加一个flag来实现