Provide failover/load balancing connection mechanism

[yasserf]

We need a way to allow us to provide multiple deepstream connection points to the client to increase resilience.

Given different nodes can be created and shutdown automatically when auto scaling this will preferably not be something that is hard coded in the client, but rather uses a discovery mechanism.

Note: Only api change is when connecting to deepstream initially
Note: Heartbeat on servers is configurable

The two requirements are:

1. Must work in a distributed fashion ( No central control )
2. Server lists should not be stored in cache and storage, instead just in memory ( This helps with distribution by avoiding concurrency issues )
3. Deepstreams that are no longer available should be deleted from lists in a reliable manner and time

Client

1: Provide multiple initial deepstream connection endpoints

    deepstream( [ 'localhost', 'deepstream1', 'deepstream2' ], options )

2: Request from a deepstream a list of all current deepstream nodes available ( Using a http get request )

    get( 'http://deepstream1/available-servers' ) // Returns array of all deepstream urls

3: Figure out which one has best score
A score is currently a combination of:
* Latency
* Resource usage on deepstream machine ( CPU, memory, logged in users, etc )

    calculateScore( 'http://deepstream1/calculate-score' )

4: Connect to best deepstream
5: On disconnect, repeat step 2 onwards

Server

1: On server being started, request all deepstreams to notify existence via event bus,

    messageBus.on( I_EXIST, function( serverConfig ) {
       // Store server config
    } );
    messageBus.emit( WHO_EXISTS )

2: When receiving existence request, respond with event,

    messageBus.on( WHO_EXISTS, function() {
        messageBus.emit( I_EXIST );
    } );

3: Send existence heartbeat every x minutes

    setInterval( function() {
        messageBus.emit( I_EXIST );
    }, x );

4: Delete servers from list if heartbeat is not received in y minutes ( where y > x )

    setInterval( function() {
          //Remove server config
    }, y );

[yasserf]

This client has now been updated with a different approach:

Case A:

1. Connect to a server
2. Receive ack
3. Use server

or

Case B:

1. Connect to a server
2. Receive a redirect
3. Connect to redirected server

or

Case C:

1. Connect to a server
2. Receive a challenge
3. Respond to challenge with valid url
4. Receive a redirect
5. Connect to redirected server

or

Case D:

1. Connect to a server
2. Receive a challenge
3. Respond to challenge with invalid url
4. Receive a REJECTION
5. Close socket