Adds StorePool

[BramGruneir]

StorePool will keep a list of alive/dead nodes as per the RPC from #2191
The allocator uses StorePool to ensure that the stores being picked are indeed alive.

Next up will be:

• get the scanner to sniff out replicas on failed stores

[mberhault]

This can go after Unmarshal.

[BramGruneir]

done

[BramGruneir]

After a chat with @tamird, I'm going to change this to wip until I get the allocator integration done.

[BramGruneir]

I think this is ready to go. @tamird PTAL

[mrtracy]

Is there anything that regularly re-gossips the storeDescriptors? That seems like a fairly important component to this.

[BramGruneir]

@mrtracy, yes, the stores all gossip their storeDescriptors every configGossipInterval, which is right now set to 1 min.

[tamird]

doesn't need to be public

[BramGruneir]

fixed

[BramGruneir]

Addressed all comments, PTAL

[bdarnell]

This loop needs a break after it reaches a live store (since every store after that will be live too). But I think this whole function could be simplified to remove the nested loop:

for {
    var nextTimeout time.Duration
    sp.mu.Lock()
    detail := sp.queue.peek()
    if detail == nil {
        nextTimeout = sp.timeUntilStoreDead
    } else if /*detail is dead */ {
        // mark as dead, dequeue. nextTimeout is 0 so we check the next one immediately.
    } else {
        // detail is alive. schedule next check
        nextTimeout = detail.lastUpdateTime.Add(sp.timeUntilStoreDead)
    }
    sp.mu.Unlock()
    <-time.After(nextTtimeout)  // in a select w/stopper
}

[BramGruneir]

Done. This is much simpler.

[mrtracy]

LGTM

[BramGruneir]

Thanks for the review @mrtracy
@tamird, want to give it one last pass before I merge?

[tamird]

possibly a dumb question: why does the allocator still need a mutex?

[BramGruneir]

Not a dumb question. I was going to try to remove it in a small follow up PR. But I just didn't want to pollute this one any further. I'll add a todo to consider its removal.