Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: remove peers that have disappeared from kubernetes #3022

Closed
wants to merge 8 commits into from

Conversation

@bboreham
Copy link
Contributor

@bboreham bboreham commented Jun 19, 2017

Fixes #2797

Using Kubernetes annotations as a way to ensure only one peer at a time runs rmpeer

@bboreham bboreham self-assigned this Jun 19, 2017
// This should be sufficiently rare that we don't care.

// Question: Should we check against Weave Net IPAM?
// i.e. If peer X owns any address space and is marked unreachable, we want to rmpeer X

This comment has been minimized.

Loading

This comment has been minimized.

Loading

This comment has been minimized.

Loading

func reclaimRemovedPeers(apl *peerList, nodes []nodeInfo) error {
// TODO
// Outline of function:
// 1. Compare peers stored in the peerList against all peers reported by k8s now.

This comment has been minimized.

Loading

This comment has been minimized.

Loading

This comment has been minimized.

Loading

This comment has been minimized.

Loading

This comment has been minimized.

Loading

- apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: weave-net2

This comment has been minimized.

Loading

This comment has been minimized.

Loading

// 8. If step 5 failed due to optimistic lock conflict, stop: someone else is handling X
// 9. Else there is an existing annotation at step 3; call its owner peer Y.
// 10. If Y is not known to k8s and marked unreachable, recurse to remove Y at 3
// 11. If we succeed to claim the annotation for Y, remove its annotation for X too

This comment has been minimized.

Loading

This comment has been minimized.

Loading

@bboreham bboreham force-pushed the issues/2797-remove-dead-peers branch from 9e83787 to 040d7b7 Jul 20, 2017
@bboreham
Copy link
Contributor Author

@bboreham bboreham commented Jul 20, 2017

I have squashed some of the commits and written an implementation of the pseudo-code previously discussed.

Next steps: try it out, figure out how to test it properly.

Loading

@bboreham
Copy link
Contributor Author

@bboreham bboreham commented Jul 20, 2017

Also still to do, as noted earlier:

We need to distinguish between "stopped but coming back" and "stopped and never coming back".

Possibly we could extend the use of the peerList annotation to say "If I'm not currently listed as a peer then delete any old IPAM persistence data"?

Loading

@caarlos0
Copy link
Contributor

@caarlos0 caarlos0 commented Oct 19, 2017

We need to distinguish between "stopped but coming back" and "stopped and never coming back".

I'm not very familiar with weave, but, why? If a node disappears and we release its IPs and etc, and if it come back later, can't we just "re-setup" it (with new IPs and all)?

Loading

@bboreham
Copy link
Contributor Author

@bboreham bboreham commented Oct 19, 2017

Weave Net has no central point of control: its native data structure is a CRDT. The "release its IPs" breaks the rules of the CRDT so we need some extra mechanism to avoid going mad.

Loading

@caarlos0
Copy link
Contributor

@caarlos0 caarlos0 commented Oct 19, 2017

hmm, interesting. Thanks for explaining @bboreham :)

Loading

@bboreham
Copy link
Contributor Author

@bboreham bboreham commented Nov 9, 2017

Replaced by #3149

Loading

@bboreham bboreham closed this Nov 9, 2017
@bboreham bboreham added this to the n/a milestone Nov 13, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants