Inconsistent view of nodes across agents #11532
Labels
area/health
Relates to the cilium-health component
kind/community-report
This was reported by a user in the Cilium community, eg via Slack.
Comments
|
Seems a little bit similar to #11511 @seanpowell-f5 is the node with connectivity issues still alive in the cluster? |
aanm
added
area/health
|
This seems to be related to high node churn. In a cluster that just went through a large node (30 -> 700 -> 260) churn, and large pod churn (3000 -> 27000 -> 3000), many cilium agents report unreachable nodes that exceed the actual in-cluster node count: 503 unreachable nodes:
but only 258 nodes in the cluster: I went to get some more info about what Cilium agents think: Looking up instances listed in the above output: Instances found in ciliumnodes and What's really bizarre here is that
Also: |
|
The agent in the previous post has fixed itself, it appears: This reconciliation took ~3.5 hours. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
stale
bot
added
the
stale
|
Both this and the other linked issue are going to be/have been marked as stale. Is there any further insight we could get from this @aanm? |
stale
bot
removed
the
stale
|
@trynity It would be helpful to have steps to replicate the issue. |
|
@trynity What version of Cilium did you experience this? |
kkourt
added a commit
to kkourt/cilium
that referenced
this issue
Aug 27, 2020
NodeAdd and NodeUpdate update the node state for clients so that they can return the changes when client requests so. If a node was added and then updated, its old and new version would be on the added list and its old on the removed list. Instead, we can just update the node on the added list. Note that the setNodes() function on pkg/health/server/prober.go first deletes the removed nodes and then adds the new ones, which means that the old version of the node would be added and remain as stale on the health server. This was found during investigation of issues with inconsistent health reports when nodes are added/removed from the cluster (e.g., cilium#11532), and it seems to fix inconsistencies observed a small-scale test I did to reproduce the issue. Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com>
joestringer
pushed a commit
that referenced
this issue
Aug 28, 2020
NodeAdd and NodeUpdate update the node state for clients so that they can return the changes when client requests so. If a node was added and then updated, its old and new version would be on the added list and its old on the removed list. Instead, we can just update the node on the added list. Note that the setNodes() function on pkg/health/server/prober.go first deletes the removed nodes and then adds the new ones, which means that the old version of the node would be added and remain as stale on the health server. This was found during investigation of issues with inconsistent health reports when nodes are added/removed from the cluster (e.g., #11532), and it seems to fix inconsistencies observed a small-scale test I did to reproduce the issue. Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com>
kaworu
pushed a commit
that referenced
this issue
Aug 28, 2020
[ upstream commit 5550c0f ] NodeAdd and NodeUpdate update the node state for clients so that they can return the changes when client requests so. If a node was added and then updated, its old and new version would be on the added list and its old on the removed list. Instead, we can just update the node on the added list. Note that the setNodes() function on pkg/health/server/prober.go first deletes the removed nodes and then adds the new ones, which means that the old version of the node would be added and remain as stale on the health server. This was found during investigation of issues with inconsistent health reports when nodes are added/removed from the cluster (e.g., #11532), and it seems to fix inconsistencies observed a small-scale test I did to reproduce the issue. Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com> Signed-off-by: Alexandre Perrin <alex@kaworu.ch>
joestringer
pushed a commit
that referenced
this issue
Aug 28, 2020
[ upstream commit 5550c0f ] NodeAdd and NodeUpdate update the node state for clients so that they can return the changes when client requests so. If a node was added and then updated, its old and new version would be on the added list and its old on the removed list. Instead, we can just update the node on the added list. Note that the setNodes() function on pkg/health/server/prober.go first deletes the removed nodes and then adds the new ones, which means that the old version of the node would be added and remain as stale on the health server. This was found during investigation of issues with inconsistent health reports when nodes are added/removed from the cluster (e.g., #11532), and it seems to fix inconsistencies observed a small-scale test I did to reproduce the issue. Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com> Signed-off-by: Alexandre Perrin <alex@kaworu.ch>
christarazi
pushed a commit
that referenced
this issue
Sep 3, 2020
[ upstream commit 5550c0f ] NodeAdd and NodeUpdate update the node state for clients so that they can return the changes when client requests so. If a node was added and then updated, its old and new version would be on the added list and its old on the removed list. Instead, we can just update the node on the added list. Note that the setNodes() function on pkg/health/server/prober.go first deletes the removed nodes and then adds the new ones, which means that the old version of the node would be added and remain as stale on the health server. This was found during investigation of issues with inconsistent health reports when nodes are added/removed from the cluster (e.g., #11532), and it seems to fix inconsistencies observed a small-scale test I did to reproduce the issue. Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com> Signed-off-by: Chris Tarazi <chris@isovalent.com>
joestringer
pushed a commit
that referenced
this issue
Sep 3, 2020
[ upstream commit 5550c0f ] NodeAdd and NodeUpdate update the node state for clients so that they can return the changes when client requests so. If a node was added and then updated, its old and new version would be on the added list and its old on the removed list. Instead, we can just update the node on the added list. Note that the setNodes() function on pkg/health/server/prober.go first deletes the removed nodes and then adds the new ones, which means that the old version of the node would be added and remain as stale on the health server. This was found during investigation of issues with inconsistent health reports when nodes are added/removed from the cluster (e.g., #11532), and it seems to fix inconsistencies observed a small-scale test I did to reproduce the issue. Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com> Signed-off-by: Chris Tarazi <chris@isovalent.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug report
cilium version)uname -a)kubectl version, Mesos, ...)will be passed to a maintainer privately
How to reproduce the issue
Don't know the exact steps, but probably what lead to this was a constantly rolling cluster – i.e., nodes were continuously being replaced by
kops rolling-update(we were running a resiliency test) over the course of 7 days. After a couple of days I noticed that one agent was reportingcilium_unreachable_nodes=1. After investigating, I found that one agent was still listing a node that's been terminated over 9 hours ago incilium-health statusand in thenode list:The text was updated successfully, but these errors were encountered: