Usage

Construction

Construction of the Gossip instance requires a number of collaborators. The three non standard collaborators are the UdpCommunications instance, the SystemView instance and the HMAC instance.

Construction of the HMAC instance

The HMAC instance provides secure hash based message authentication within the system. This is based on a shared key. The HMAC uses SHA1 and allows the key to be dynamically changed while the system is operating. The default constructor of the HMAC sets up a default key (which should not be used if you care about security). The main constructor takes a secret key which is used to compute and validate the mac

HMAC(SecretKey key)

Construction of the UdpCommunications instance

The UdpCommunications instance is the underlying UDP communications system. The class has several convienence constructors for defaulting values, but the terminal constructor is:

 UdpCommunications(DatagramSocket socket, ExecutorService executor,
                   int receiveBufferMultiplier,
                   int sendBufferMultiplier, HMAC mac)

The socket is the UDP datagram socket that will be used for communications, the executor is the thread pool used to dispatch received messages to the Gossip instance. The receive and send buffer multipliers are used to configure these parameters on the datagram socket. Under load, the OS will discard datagrams when the socket already has a datagram ready to receive, or one that it has not sent. The default buffer multipliers are both 4 and this provides a good compromise to minimize the number of dropped datagrams due to high load. The HMAC of the UdpCommunications is not optional and must not be null.

Construction of the SystemView instance

The System view instance provides the management of the various endpoint sets that are required for the Gossip protocol. The SystemView has one constructor

SystemView(Random random, InetSocketAddress localAddress,
           Collection<InetSocketAddress> seedHosts,
           int quarantineDelay, int unreachableDelay)

The random source of entropy is central to the Gossip protocol, and if this is not a high quality source of entropy, the Gossip protocol will suffer greatly. Consequently, you should always provide a SecureRandom or equivalent to ensure quality entropy.

The localAddress of the system view must be the same as the endpoint for the UdpCommunications. The standard procedure is to construct the UdpCommunications first and then use the getLocalAddress() method on the communications instance to construct the system view.

The Gossip protocol requires a list of seed hosts in order to bootstrap the system. Not all the seed hosts have to be up and part of the gossip system, nor does this set of seed hosts required to be the same on all members of the gossip set. However, there must be some common thread that all members can eventually discover each other or you will end up with partitioned clusters that cannot communicate with each other.

The last two parameters define how long an endpoint has to be quarantined before the system will allow it to reenter the gossip set. This prevents some nasty thrashing and should not be set to too low of a value. The value is in milliseconds and should be greater than the gossip period. The unreachable delay determines how long an endpoint has to be unreachable before the protocol will give up trying to contact them.

The gossip protocol periodically retries dead members to reform after partions, so setting this value too low will promote partitioned groups that will no longer be able to communicate. This value is in milliseconds, and setting this value in the realm of hours to days are good ideas.

Construction of the Gossip instance

The Gossip instance has a number of convenience parameters that provide default settings, but they all ground out to this constructor

Gossip(NoArgGenerator idGenerator, GossipListener stateListener,
       GossipCommunications communicationsService, SystemView systemView,
       FailureDetectorFactory failureDetectorFactory, Random random,
       int gossipInterval, TimeUnit unit, int cleanupCycles)

The NoArgGenerator is the instance of the UUID generator. It is extremely important that UUIDs generated within the system be unique and no, UUID.randomUUID() is not sufficient to ensure this. The default id generator used is one based on the MAC address of the network and the system time. The GossipListener is the recipient of the notification events from the gossip protocol. This is an instance you supply. The communications instance, the system view instance and the failure detection factory are the instances you created previously as described earlier.

The source of entropy for the Gossip instance must also be of high quality and may be the same shared instance with the system view instance. The gossip interval parameters control how often the gossip protocol is run. Setting this too high will correspondingly create more CPU load and use more bandwidth. 3 seconds does is not unreasonable.

Finally, the Gossip protocol needs to know how many gossip cycles to wait before declaring a suspected endpoint as dead. A good number is 3, but your mileage may vary. Due to the epidemic nature of the gossip protocol, the system has a large amount of variation at the early phases of the system evolution, or when there is a large churn in membership. Consequently there is a tension between the random nature of the protocol and the failure detection mechanism. The protocol therefore uses the cycle time parameter to determine if the endpoint really is dead or if there is just a bit of churn in the system which confuses the failure detectors.