HDDS-5916. Add IP check when DN register to SCM #2802
Conversation
|
The old pipelines would get stuck in leader election, timeout and then get closed for some DN being candidate too long. |
Xushaohong
force-pushed
the
HDDS-5916
branch
2 times, most recently
from
e59fb4d
to
a77b09f
Compare
|
This fix leaves the defect that possibly there is still one or two pipelines not working in my K8S env, the raft peer info among pipeline raft group is latest according to the log. I have 156 pipelines in total, but only one is unstable, which is quite tricky, as most newly-created pipelines work fine. I've tried some times to redeploy the K8S env and this problem still happens. And sometimes this strange phenomenon could recover by itself. |
|
@Xushaohong re:
Does this mean that eventually after some timeout that the leader election is resolved? Or does it perpetuate for any previously created pipelines? |
The previously created pipelines would be closed after timeout, and new pipelines, most of them would work normally, except one or two still dangling. @techwhizbang |
|
I think this problem is more completely addressed in #3186. |
Thx for this information! I will close this PR. |
What changes were proposed in this pull request?
SCM doesn't upgrade DN NodeInfo if any DN restarts.
This causes the issue described in the JIRA, which in k8s env would lead to unavailable cluster conditions.
Thus, we should add this case to upgrade the LastknownIpAddress of Datanodes to NodeMap when they register again.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5916
How was this patch tested?
Manual Test on K8s