Question: Daemon “Node join event” logs in 17.06 #33962
Comments
|
Looks like something in libnetwork, which I don't see in master. Ping @sanimej |
|
@kinghuang This should have been a debug message. All the cluster nodes exchange a gossip message to indicate what networks are currently active on a node. This message is also synced over TCP every 30 seconds or if a peer queries for it. This doesn't indicate nor should cause any issue other than the chatty logs. I will push a change to fix it. |
|
Opened moby/libnetwork#1833 to change the logging level from |
|
@thaJeztah Will this also be fixed in 17.06.1? |
|
@Cas-pian For me I still see it in 17.06.1 |
|
I've upgraded to Docker CE 17.07 on one of my swarms, but I'm still getting the “Node join event …” log lines at info level. @thaJeztah Is the fix for sure in 17.07? Or, is this a new problem? |
|
@thaJeztah I've completed upgrading my swarms to 17.07.0-ce. They're all still outputing the “Node join event…” messages. |
|
I can confirm the logs are still noisy; I dived a bit in the code to understand what everything does, and think I see what's happening (I'm not really familiar with that code, so explaining what I saw below); First the node is retrieved from the database through After that, the node is retrieved from the database again, this time using Looking at After that, the code I added checks if the node's status was already known as "joined" (due to the explanation above, it won't be), and after that, adds the node to the list of joined nodes, and if it was a "new" join (again, probably "always"), logs the node join event. Possible solutions:
@fcrisciani @abhinandanpb any suggestions? |
|
@thaJeztah the proper solution is in splitting the lamport time between the node events and network events and after that correct the use of the witness. The witness clock always checks that the node lamport time is max(local, event) and if that is > than the local one it increments it by 1 and update the local one. Every 30s there is a TCP sync that exchange the full series of messages and the node related ones are getting the latest lamport time that is in the node, that got bumped by the previous logic that I was mentioning, creating the illusion to other nodes that the events received are actually new and so printing the log (and incrementing again the lamport time). |
|
Hi, I'm still getting this error with version 'Server Version: 17.09.0-ce'. @fcrisciani your solution has to do with the source code not with the server setup right? Because I don't know how to make sure that the lamport time remains consistent from the server/node side. Thanks |
|
@MarvAmBass yep correct, that will require a code change, at the moment not being a breaking issue that creates traffic disruption is not getting much priority. |
I'm still getting this on |
|
@agilob if you read beyond that you would have seen that this was reopened and my later comment: #33962 (comment) |
|
17.12 should fix that |
|
Let me close this ticket for now, as it looks like it went stale. |
Description
Since upgrading my swarms to Docker CE 17.06, the Docker daemons have been endlessly outputting info log lines about “Node join event for (some node here)”. Is this normal? They are annoying in that there's a huge number of log lines being generated, but there hasn't been any noticeable ill-effect.
Steps to reproduce the issue:
Describe the results you received:
The Docker daemon logs on all the swarm nodes are endlessly outputting log lines about node join events. Here's a brief sample.
Describe the results you expected:
Previous versions of Docker did not produce these log lines. They are adding very significant noise to the daemon logs. I'm not sure if this indicates a problem, or what node join events mean in this context.
Additional information you deem important (e.g. issue happens only occasionally):
The log line comes from libnetwork in networkdb/delegate.go for
NodeEventTypeJoin, and was merged in moby/libnetwork#1775.Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.):
The data here is from a 10-node Docker CE 17.06.0 swarm on RHEL 7.3. My other Docker CE 17.06.0 swarms also exhibit the same behaviour.
The text was updated successfully, but these errors were encountered: