Miss notification for IP does not trigger serf query message

[huikang]

I was looking at how the overlay driver resolves the IP and vtep by sending out serf query message.

On some machines, I can see the following serf query and response in the libnetwork log

time="2017-03-01T16:53:00Z" level=debug msg="Miss notification, l2 mac 02:42:0a:00:00:02"
time="2017-03-01T16:53:00Z" level=debug msg="Miss notification, l2 mac 02:42:0a:00:00:04"
time="2017-03-01T16:53:00Z" level=debug msg="Watch triggered with 3 nodes" discovery=consul
time="2017-03-01T16:53:00Z" level=debug msg="2017/03/01 16:53:00 [DEBUG] serf: messageQueryType: peerlookup\n"
time="2017-03-01T16:53:00Z" level=debug msg="Received query name:peerlookup, payload:5937a4c758da23582c7505d9fab6a031592ec46\
bdd32d63e736c5eb7d5064977 10.0.0.2\n"
time="2017-03-01T16:53:00Z" level=debug msg="Sending peer query resp mac 02:42:0a:00:00:02, mask 255.255.255.0, vtep 11.0.8.\
4"

However, on some machine, the miss notification for dest IP is not followed by any query message.

time="2017-03-02T15:34:49Z" level=debug msg="Parsed data = 571a22ba6111af873c4aeb4a72353eab1e3eed00bd38fb5a187b266232ca8bc7/\
ab2431febd11c8c477059e4d350764776c2036ca08bcef8be720969e85e3e121/11.0.8.4/10.0.0.3/255.255.255.0/02:42:0a:00:00:03\n"
time="2017-03-02T15:34:49Z" level=debug msg="checkEncryption(571a22b, 11.0.8.4, 256, false)"
time="2017-03-02T15:34:49Z" level=debug msg="miss notification for dest IP, 10.0.0.3"
time="2017-03-02T15:34:49Z" level=debug msg="Miss notification, l2 mac 02:42:0a:00:00:03"
time="2017-03-02T15:34:51Z" level=debug msg="Watch triggered with 3 nodes" discovery=consul

In both machine, I can ping the overlay IP. I know that an entry will be added to the peerDb on all the nodes on joining an endpoint and a sandbox (through serfuserevent). However, I do not understand why the query message appears on some hosts, but does not appear on some others (even all hosts have sandbox belong to the overlay network).

When I look through the libnetwork code, I found this (https://github.com/docker/libnetwork/blob/3118f18e6af826d7e064b11dc0a32ed0670068b4/drivers/overlay/ov_network.go#L662)

logrus.Debugf("miss notification for dest IP, %v", neigh.IP.String())

if neigh.State&(netlink.NUD_STALE|netlink.NUD_INCOMPLETE) == 0 {
				continue
}

The above line seems prevent the overlay driver sending out the serf query. Could anyone help me understand how the above works? Thanks.