Add startup lock support (general framework + tested etcd implementation) #6

Gsantomaggio · 2017-03-27T07:44:14Z

This required completely reworking startup sequence in `autocluster.erl` without making the existing dozen step sequence into an incomprehensible mess. Now steps are more independent and are executed in order with function that provides something like State + Either monads. Boot process is now split into 3 stages: - First stage roughly corresponds to original implementation - Second step is responsible for registering node in the backend. Also TTL timers are now started there, instead of spreading `-rabbit_boot_steps` through backend modules. This makes `autocluster.erl` the single source of truth about what is actually happening during startup. - Third stage is responsible for releasing startup lock. We need it because of `ignore` failure mode - we want the lock to be released even if some prior steps failed. During this rework it became evident that explicit mnesia:reset/0 is not needed (more thorough explanation is in comments for `autocluster:maybe_cluster/1`). Startup locking support should work the same way for every backend, but for now the only working implementation is for `etcd`. It even has some property-based tests =) Other backend fall back to original random delay behaviour. Also it looks like `dialyzer` ignores type specs in comments - because some of types there were definitely wrong, and converting to them to `-spec` caused a lot of barfing from `dialyzer`.

…rabbitmq-autocluster_startup-locking-support

The plugin has to work even if the Backend is not configured. Add the unit tests for these cases

…r_startup-locking-support

Fix consul example

Gsantomaggio · 2017-04-13T12:22:22Z

examples/k8s_minikube/README.md

+6. Build a [Docker image](https://github.com/rabbitmq/rabbitmq-autocluster/blob/master/Dockerfile)
+```
+git clone https://github.com/rabbitmq/rabbitmq-autocluster.git rabbitmq-autocluster
+make dist


Here the users have to build the plugin from source.
We can remove:

git clone https://github.com/rabbitmq/rabbitmq-autocluster.git rabbitmq-autocluster make dist

when we will put the docker image on docker hub

michaelklishin · 2017-04-19T16:30:51Z

We will take a look after 0.7.0 GA is released.

michaelklishin · 2017-04-24T11:37:23Z

@binarin @Gsantomaggio I don't see where in the code is augment_nodelist actually used beyond the test suite. Am I missing something or can be it removed? Alternative node representation records a pretty controversial thing for us to adopt into RabbitMQ master.

michaelklishin · 2017-04-24T11:42:09Z

OK, so GitHub does not display the entire diff again. The "find best node to join" part can be a bit too opinionated for inclusion into RabbitMQ core.

dcorbacho · 2017-04-24T13:22:17Z

src/autocluster.erl

@@ -371,19 +382,34 @@ startup_delay(Max) ->

 -spec backend_register(#startup_state{}) -> ok | {error, iolist()}.
 backend_register(#startup_state{backend_module = Mod}) ->
-    Mod:register().
+    case erlang:function_exported(Mod, register, 0) of


Why not simply check if Mod is unconfigured ? All backends should support these callbacks, otherwise is a proper error.

if it is unconfigured it raises the error undef

dcorbacho · 2017-04-25T11:12:42Z

examples/k8s_minikube/README.md

+
+minikube service rabbitmq-deployment --namespace=test-rabbitmq 
+
+kubectl scale rabbitmq-deployment --replicas=4


Fails with:

$ kubectl scale rabbitmq-deployment --replicas=4 error: resource(s) were provided, but no name, label selector, or --all flag specified

This would work: kubectl scale --replicas=4 -f examples/k8s_minikube/rabbitmq.yaml

This error was in this commit a8987df#diff-c9aed055be592401ae2140d96ca90ad3R77

Already fixed here: https://github.com/rabbitmq/rabbitmq-autocluster/pull/6/files#diff-c9aed055be592401ae2140d96ca90ad3R94

kubectl scale deployment/rabbitmq-deployment --namespace=test-rabbitmq --replicas=6

Fix unit-tests by adding the autocluster:new_startup_state().

Add consul link example Fix consul link

Fixes #18. [#144234169]

Conflicts: src/autocluster_k8s.erl

michaelklishin · 2017-05-20T21:54:24Z

Available as of 0.8.0.M1.

binarin · 2017-05-22T15:25:58Z

Sadly This plugin will now pick a running reachable node that has the best uptime record from release notes is not true. While initially I though that this is a good idea, I wasn't able to come up with a solid (distributed) algorithm for this. So IIRC current implementation does the simplest thing that I can still reason about - it chooses first live node in alphabetical order.

michaelklishin · 2017-05-22T15:30:48Z

@binarin thanks, I will edit out that line now.

binarin and others added 6 commits October 11, 2016 11:38

Merge master into startup-locking-support

bc20fb6

Fix PropEr tests on travis

6d3b6c4

Update tests after merging master

be370d4

Don't ignore errors that happen in travis builds

67506b8

Merge remote-tracking branch 'upstream/startup-locking-support' into …

4909873

…rabbitmq-autocluster_startup-locking-support

Gsantomaggio force-pushed the rabbitmq-autocluster_startup-locking-support branch from 6093cd4 to 4909873 Compare March 27, 2017 10:06

Gabriele Santomaggio added 2 commits April 12, 2017 11:05

Fix error in case backend is not configured

b5e1bc4

The plugin has to work even if the Backend is not configured. Add the unit tests for these cases

Merge stable to rabbitmq-autocluster_startup-locking-support

014443a

Gsantomaggio changed the title ~~DO NOT MERGE Add startup lock support (general framework + tested etcd implementation)~~ Add startup lock support (general framework + tested etcd implementation) Apr 12, 2017

Gsantomaggio requested review from michaelklishin and dcorbacho April 12, 2017 10:27

Gabriele Santomaggio added 3 commits April 12, 2017 16:30

Merge remote-tracking branch 'origin/stable' into rabbitmq-autocluste…

0e0d020

…r_startup-locking-support

Add K8s example using minikue

a8987df

Fix consul example

Fix Readme errors

a9fad2c

Gsantomaggio commented Apr 13, 2017

View reviewed changes

Add CLEANUP_WARN_ONLY to the example

957fea8

dcorbacho reviewed Apr 24, 2017

View reviewed changes

dcorbacho reviewed Apr 25, 2017

View reviewed changes

ValFadeev mentioned this pull request Apr 25, 2017

Add startup lock support for Consul backend #20

Closed

Gabriele Santomaggio and others added 7 commits April 26, 2017 11:50

Remove function_exported.

00f74b2

Fix unit-tests by adding the autocluster:new_startup_state().

Update README.md

dbbf45b

Link to 0.7.0-RC1

47e342f

Typo

187634c

Link to general plugin doc guide

68acce1

Change the docker image name

e4b6482

Add consul link example Fix consul link

README updates

6bc8362

michaelklishin added 3 commits May 2, 2017 13:55

README updates

abc6445

Use a non-prefixed service ID in Consul backend's unregister/0

6a2e006

Fixes #18. [#144234169]

Editing WIP: wording, grammar, naming

c3bf3d4

michaelklishin added this to the 0.8.0 milestone May 20, 2017

michaelklishin mentioned this pull request May 20, 2017

0.8.0 Release Timeline #28

Closed

michaelklishin added 3 commits May 20, 2017 18:57

Wording, naming

740ab59

Naming

3fdb4be

Merge branch 'stable' into rabbitmq-autocluster_startup-locking-support

5763565

Conflicts: src/autocluster_k8s.erl

michaelklishin merged commit a8757de into stable May 20, 2017

michaelklishin mentioned this pull request May 20, 2017

Consul: added startup locking support #22

Merged

dcorbacho mentioned this pull request Jun 8, 2017

Implement locking support in peer discovery rabbitmq/rabbitmq-server#1257

Closed

dumbbell deleted the rabbitmq-autocluster_startup-locking-support branch January 2, 2018 17:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add startup lock support (general framework + tested etcd implementation) #6

Add startup lock support (general framework + tested etcd implementation) #6

Gsantomaggio commented Mar 27, 2017

Gsantomaggio Apr 13, 2017 •

edited

Loading

michaelklishin commented Apr 19, 2017

michaelklishin commented Apr 24, 2017

michaelklishin commented Apr 24, 2017 •

edited

Loading

dcorbacho Apr 24, 2017

Gsantomaggio Apr 24, 2017

dcorbacho Apr 25, 2017

Gsantomaggio Apr 26, 2017

Gsantomaggio Apr 26, 2017

michaelklishin commented May 20, 2017

binarin commented May 22, 2017

michaelklishin commented May 22, 2017


		minikube service rabbitmq-deployment --namespace=test-rabbitmq

		kubectl scale rabbitmq-deployment --replicas=4

Add startup lock support (general framework + tested etcd implementation) #6

Add startup lock support (general framework + tested etcd implementation) #6

Conversation

Gsantomaggio commented Mar 27, 2017

Gsantomaggio Apr 13, 2017 • edited Loading

Choose a reason for hiding this comment

michaelklishin commented Apr 19, 2017

michaelklishin commented Apr 24, 2017

michaelklishin commented Apr 24, 2017 • edited Loading

dcorbacho Apr 24, 2017

Choose a reason for hiding this comment

Gsantomaggio Apr 24, 2017

Choose a reason for hiding this comment

dcorbacho Apr 25, 2017

Choose a reason for hiding this comment

Gsantomaggio Apr 26, 2017

Choose a reason for hiding this comment

Gsantomaggio Apr 26, 2017

Choose a reason for hiding this comment

michaelklishin commented May 20, 2017

binarin commented May 22, 2017

michaelklishin commented May 22, 2017

Gsantomaggio Apr 13, 2017 •

edited

Loading

michaelklishin commented Apr 24, 2017 •

edited

Loading