Events dropped on client reconnect

[kristian-elder]

I have only seen this once in production, and so far been unable to reproduce, but I can confirm that for some reason (currently pointing to the ES client), I've missed events in a subscription as the client was reconnecting.

I am using the Reactive Streams API for subscriptions and had an active subscription to a category projection stream.

Eventstore client logs for the time period:

22:00:09.000
[default-akka.actor.default-dispatcher-9] - s.c.c.HttpHostConnectionSlot - Delivering 200 OK response for GET request to /gossip?format=json
22:00:09.000
[default-akka.actor.default-dispatcher-3] - eventstore.tcp.ConnectionActor - Connection lost to /10.4.0.4:1113: no heartbeat within 2 seconds
22:00:09.000
[default-akka.actor.default-dispatcher-5] - e.cluster.ClusterDiscovererActor - Cluster best node MemberInfo(77ed15a3-6a80-4777-84ee-5dbcbaba5c83,2016-08-25T21:00:09.110Z,Master,true,/10.4.0.4:1112,/10.4.0.4:1113,/10.4.0.4:0,/10.4.0.4:0,/10.4.0.4:2112,/10.4.0.4:2113,4236665466,4236665737,4236665737,3970346812,17,9173f38e-ccbc-4e21-95b0-89da82b01a89,0) failed, reported by Actor[akka://default/user/$a#1574497158]
22:00:10.000
[default-akka.actor.default-dispatcher-5] - akka.actor.LocalActorRef - Message [akka.io.Tcp$Write] from Actor[akka://default/user/$a/$h#1366248946] to Actor[akka://default/system/IO-TCP/selectors/$a/995#413901425] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
22:00:10.000
[default-akka.actor.default-dispatcher-5] - akka.actor.LocalActorRef - Message [akka.io.Tcp$Write] from Actor[akka://default/user/$a/$h#1366248946] to Actor[akka://default/system/IO-TCP/selectors/$a/995#413901425] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
22:00:10.000
[default-akka.actor.default-dispatcher-5] - s.c.c.HttpHostConnectionSlot - Attempting new connection to 10.4.2.4:2113
22:00:10.000
[default-akka.actor.default-dispatcher-5] - s.c.client.HttpClientConnection - Attempting connection to /10.4.2.4:2113
22:00:10.000
[default-akka.actor.default-dispatcher-5] - akka.io.TcpOutgoingConnection - Attempting connection to [/10.4.2.4:2113]
22:00:10.000
[default-akka.actor.default-dispatcher-5] - akka.io.TcpOutgoingConnection - Connection established to [/10.4.2.4:2113]
22:00:10.000
[default-akka.actor.default-dispatcher-5] - s.c.client.HttpClientConnection - Connected to /10.4.2.4:2113
22:00:10.000
[default-akka.actor.default-dispatcher-5] - s.c.c.HttpHostConnectionSlot - Connection to 10.4.2.4:2113 established, dispatching 1 pending requests
22:00:10.000
[default-akka.actor.default-dispatcher-5] - s.c.c.HttpHostConnectionSlot - Dispatching GET request to /gossip?format=json across connection Actor[akka://default/user/IO-HTTP/group-0/1826#-746410576]
22:00:10.000
[default-akka.actor.default-dispatcher-5] - s.c.c.HttpHostConnectionSlot - Delivering 200 OK response for GET request to /gossip?format=json
22:00:10.000
[default-akka.actor.default-dispatcher-5] - s.c.c.HttpHostConnectionSlot - Attempting new connection to 10.4.1.3:2113
22:00:10.000
[default-akka.actor.default-dispatcher-5] - s.c.client.HttpClientConnection - Attempting connection to /10.4.1.3:2113
22:00:10.000
[default-akka.actor.default-dispatcher-5] - e.cluster.ClusterDiscovererActor - Discovering cluster: attempt 1/10 successful: best candidate is MemberInfo(77ed15a3-6a80-4777-84ee-5dbcbaba5c83,2016-08-25T21:00:09.236Z,Master,true,/10.4.0.4:1112,/10.4.0.4:1113,/10.4.0.4:0,/10.4.0.4:0,/10.4.0.4:2112,/10.4.0.4:2113,4236670793,4236671430,4236671430,3970346812,17,9173f38e-ccbc-4e21-95b0-89da82b01a89,0)
22:00:10.000
[default-akka.actor.default-dispatcher-5] - akka.io.TcpOutgoingConnection - Attempting connection to [/10.4.1.3:2113]
22:00:10.000
[default-akka.actor.default-dispatcher-5] - akka.io.TcpOutgoingConnection - Attempting connection to [/10.4.0.4:1113]
22:00:10.000
[default-akka.actor.default-dispatcher-5] - akka.io.TcpOutgoingConnection - Connection established to [/10.4.0.4:1113]
22:00:10.000
[default-akka.actor.default-dispatcher-5] - eventstore.tcp.ConnectionActor - Connected to /10.4.0.4:1113
22:00:10.000
[default-akka.actor.default-dispatcher-5] - akka.io.TcpOutgoingConnection - Connection established to [/10.4.1.3:2113]
22:00:10.000
[default-akka.actor.default-dispatcher-5] - s.c.client.HttpClientConnection - Connected to /10.4.1.3:2113
22:00:10.000
[default-akka.actor.default-dispatcher-5] - s.c.c.HttpHostConnectionSlot - Connection to 10.4.1.3:2113 established, dispatching 1 pending requests
22:00:10.000
[default-akka.actor.default-dispatcher-5] - s.c.c.HttpHostConnectionSlot - Dispatching GET request to /gossip?format=json across connection Actor[akka://default/user/IO-HTTP/group-0/1827#1976135446]
22:00:10.000
[default-akka.actor.default-dispatcher-5] - s.c.c.HttpHostConnectionSlot - Delivering 200 OK response for GET request to /gossip?format=json