Skip to content

Driver node events can be delivered out of order #2257

@talex5

Description

@talex5

I added some logging to the overlay driver to log NodeDiscovery events.
Then I killed dockerd (kill -9) on host 172.31.26.11 and restarted it. Here are the log events printed on another node, watching the restart:

Aug 21 07:47:21 ip-172-31-0-214.us-east-2.compute.internal dockerd[10382]: time="2018-08-21T07:47:21.292565622-07:00" level=warning msg="overlay: New/NodeDiscovery(discoverapi.NodeDiscoveryData{Address:\"172.31.26.11\", BindAddress:\"\", Self:false})"
Aug 21 07:47:34 ip-172-31-0-214.us-east-2.compute.internal dockerd[10382]: time="2018-08-21T07:47:34.175288657-07:00" level=warning msg="overlay: Delete/NodeDiscovery(discoverapi.NodeDiscoveryData{Address:\"172.31.26.11\", BindAddress:\"\", Self:false})"

Note that:

  1. It gets the notification of the node rejoining first, then the notification that it left second.
  2. The notification only contains the IP address.

The overlay driver might not care about this (DiscoverDelete just returns nil), but it might make sense to change this in some way. One option would be to notify drivers about changes in networkdb's view of membership rather than memberlist's view (networkdb removes the old node as soon as it hears about the old one - but see #2256). Another would be to include the node name in the message, so it's clear they are about different instances.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions