[FIXED] Clustering: leadership acquired actions could get stuck

If a leadership changed occurred while leadership actions were
executed, before the raft.Barrier() call was made, the server
would be stuck in that call. This is because RAFT library
notifies the Streaming server code that a leadership changed
through a go channel that was just of size 1. Since the
streaming server read from the channel and then executes
the leadership acquired code, it could not read from the
notification channel that caused the RAFT library to block
on a go channel send, which then made the Barrier() call
block.

I believe the right approach is to have a bigger notification
go channel instead of making Barrier() time out. If it does
timeout, the server should then transfer leadership, which
I am afraid could cause a cascading effect if all servers
getting elected need longer that the chosen timeout to
apply all the preceding entries to the FSM.

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>

dependencies.md

@@ -6,26 +6,17 @@ This file lists the dependencies used in this repository.
 |-|-|
 | Go | BSD 3-Clause "New" or "Revised" License |
 | github.com/nats-io/nats-streaming-server | Apache License 2.0 |
-| github.com/armon/go-metrics v0.0.0-20190430140413-ec5e00d3c878 | MIT |
-| github.com/fatih/color v1.7.0 | MIT |
-| github.com/go-sql-driver/mysql v1.6.0 | Mozilla Public License 2.0 |
+| github.com/go-sql-driver/mysql v1.7.0 | Mozilla Public License 2.0 |
 | github.com/gogo/protobuf v1.3.2 | Go |
-| github.com/golang/protobuf v1.4.2 | BSD 3-Clause "New" or "Revised" License |
-| github.com/hashicorp/go-hclog v0.16.1 | MIT |
-| github.com/hashicorp/go-immutable-radix v1.0.0 | Mozilla Public License 2.0 |
-| github.com/hashicorp/go-msgpack v1.1.5 | BSD 3-Clause "New" or "Revised" License |
-| github.com/hashicorp/golang-lru v0.5.0 | Mozilla Public License 2.0 |
-| github.com/hashicorp/raft v1.3.1 | Mozilla Public License 2.0 |
-| github.com/lib/pq v1.10.2 | [MIT-Like](https://github.com/lib/pq/blob/master/LICENSE.md) |
-| github.com/mattn/go-colorable v0.1.4 | MIT |
-| github.com/mattn/go-isatty v0.0.10 | MIT |
-| github.com/nats-io/jwt/v2 v2.0.2 | Apache License 2.0 |
-| github.com/nats-io/nats-server/v2 v2.2.4 | Apache License 2.0 |
-| github.com/nats-io/nats.go v1.11.0 | Apache License 2.0 |
-| github.com/nats-io/nkeys v0.3.0 | Apache License 2.0 |
+| github.com/hashicorp/go-hclog v1.5.0 | MIT |
+| github.com/hashicorp/go-msgpack/v2 v2.1.0 | MIT |
+| github.com/hashicorp/raft v1.4.0 | Mozilla Public License 2.0 |
+| github.com/lib/pq v1.10.7 | [MIT-Like](https://github.com/lib/pq/blob/master/LICENSE.md) |
+| github.com/nats-io/nats-server/v2 v2.9.15 | Apache License 2.0 |
+| github.com/nats-io/nats.go v1.25.0 | Apache License 2.0 |
 | github.com/nats-io/nuid v1.0.1 | Apache License 2.0 |
-| github.com/nats-io/stan.go v0.8.3 | Apache License 2.0 |
-| github.com/prometheus/procfs v0.6.0 | Apache License 2.0 |
-| go.etcd.io/bbolt v1.3.5 | MIT |
-| golang.org/x/crypto v0.0.0-20210314154223-e6e6c4f2bb5b | BSD 3-Clause "New" or "Revised" License |
-| golang.org/x/sys v0.0.0-20210225134936-a50acf3fe073 | BSD 3-Clause "New" or "Revised" License |
+| github.com/nats-io/stan.go v0.10.4 | Apache License 2.0 |
+| github.com/prometheus/procfs v0.8.0 | Apache License 2.0 |
+| go.etcd.io/bbolt v1.3.6 | MIT |
+| golang.org/x/crypto v0.6.0 | BSD 3-Clause "New" or "Revised" License |
+| golang.org/x/sys v0.5.0 | BSD 3-Clause "New" or "Revised" License |

go.mod

@@ -5,16 +5,16 @@ go 1.14
 require (
 	github.com/go-sql-driver/mysql v1.7.0
 	github.com/gogo/protobuf v1.3.2
-	github.com/hashicorp/go-hclog v1.1.0
+	github.com/hashicorp/go-hclog v1.5.0
 	github.com/hashicorp/go-msgpack/v2 v2.1.0
-	github.com/hashicorp/raft v1.3.11
-	github.com/lib/pq v1.10.4
-	github.com/nats-io/nats-server/v2 v2.9.11
-	github.com/nats-io/nats.go v1.22.1
+	github.com/hashicorp/raft v1.4.0
+	github.com/lib/pq v1.10.7
+	github.com/nats-io/nats-server/v2 v2.9.15
+	github.com/nats-io/nats.go v1.25.0
 	github.com/nats-io/nuid v1.0.1
 	github.com/nats-io/stan.go v0.10.4
 	github.com/prometheus/procfs v0.8.0
 	go.etcd.io/bbolt v1.3.6
-	golang.org/x/crypto v0.5.0
-	golang.org/x/sys v0.4.1-0.20230105183443-b8be2fde2a9e
+	golang.org/x/crypto v0.6.0
+	golang.org/x/sys v0.5.0
 )