Better handing when SG drops TAP feed

[adamcfraser]

In https://issues.couchbase.com/browse/CBSE-1722, customer had a scenario where a SG node appeared to drop the TAP feed (in a two SG node cluster, one node stopped doing channel cache writes at a specific point in time).

Review whether it's possible to identify a dropped feed and restore in this scenario.

At minimum we should have better diagnostics/warnings/feedback in the logs when the server loses it's TAP feed.

[adamcfraser]

If not already covered by the go-couchbase libraries, this should also handle the case where the TAP feed drops a subset of vbuckets.

[nod]

As you develop a patch for this, I'd be very willing to test on our setup.

[nod]

refs #692

Adding ref here to loop in @nl

[adamcfraser]

Error log for dropped TAP feed:
https://gist.github.com/adamcfraser/4ed3858f7d42ce20cc49

[adamcfraser]

Sync Gateway will now take the bucket offline in response to a dropped TAP feed, and attempt to reconnect/bring the bucket online automatically.