New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore previous serf user events to avoid wrong fdb programming #1639

Merged
merged 1 commit into from Feb 7, 2017

Conversation

Projects
None yet
3 participants
@sanimej
Contributor

sanimej commented Feb 7, 2017

When deploying classic swarm with overlay network and using on-node-failure reschedule policy randomly some containers might have connectivity issues with other containers in the same network.

To illustrate: One container C1 with three worker nodes W1, W2 and W3.

Container interface's mac is built from the IP address.
Let say C1 has 10.0.0.2 and mac 02:42:0a:00:00:02. If its initially scheduled on W1 the reachability info will be distributed to other nodes will be
<join 10.0.0.2, 02:42:0a:00:00:02, VTEP IP of W1>

If W1 goes through a failure and the container gets rescheduled to W2 the results events will be
<leave 10.0.0.2, 02:42:0a:00:00:02, VTEP IP of W1>
<join 10.0.0.2, 02:42:0a:00:00:02, VTEP IP of W2>

At this point if W3's daemon is restarted serf replays all the previous events. But they can be delivered out of order to the client. So libnetwork could end up getting the following..

<join 10.0.0.2, 02:42:0a:00:00:02, VTEP IP of W2>
<leave 10.0.0.2, 02:42:0a:00:00:02, VTEP IP of W1>
<join 10.0.0.2, 02:42:0a:00:00:02, VTEP IP of W1>

W3 ends up with incorrect forwarding info.

The change is to avoid getting the older events from serf when joining the gossip cluster. Hence for the containers that are already running in the network the miss notification and query mechanism will be used to program the neighbor entry/FDB.

Also added some debugs to help in debugging.

Signed-off-by: Santhosh Manohar santhosh@docker.com

Show outdated Hide outdated drivers/overlay/ov_serf.go Outdated
@sanimej

This comment has been minimized.

Show comment
Hide comment
@sanimej

sanimej Feb 7, 2017

Contributor

This change can be reverted (if we can avoid querying the cluster its better) if the serf ordering can be fixed.

hashicorp/serf#448

Contributor

sanimej commented Feb 7, 2017

This change can be reverted (if we can avoid querying the cluster its better) if the serf ordering can be fixed.

hashicorp/serf#448

Show outdated Hide outdated drivers/overlay/ov_serf.go Outdated
@aboch

This comment has been minimized.

Show comment
Hide comment
@aboch

aboch Feb 7, 2017

Contributor

LGTM besides few minor comments

Contributor

aboch commented Feb 7, 2017

LGTM besides few minor comments

Santhosh Manohar
Ignore previous serf user events to avoid wrong fdb programming
Signed-off-by: Santhosh Manohar <santhosh@docker.com>
@aboch

This comment has been minimized.

Show comment
Hide comment
@aboch

aboch Feb 7, 2017

Contributor

LGTM again

Contributor

aboch commented Feb 7, 2017

LGTM again

@aboch aboch merged commit 76ca671 into docker:master Feb 7, 2017

2 checks passed

ci/circleci Your tests passed on CircleCI!
Details
dco-signed All commits are signed

@sanimej sanimej deleted the sanimej:serf branch Mar 31, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment