Random testThaliPullReplicationFromNotificationCoordinated test failure

[chapko]

Logs from devices:

• Nexus_6P ENU7N15B26000497.log
• Nexus_6P ENU7N16425000183.log
• Nexus_6P ENU7N16425000206.log

Branch: iOS
Commit: 4b97950

Reproducible on desktop

[chapko]

Turns out cancelling replication does not always emits 'complete' event. We should also listen to 'cancel' event. I'm still trying to figure out how to reliably reproduce this issue.

[yaronyg]

Have we tried to update our release of PouchDB to see if perhaps this has been addressed? If you do get a repro then it would be good to file this as a bug with PouchDB even as we work around it.

[chapko]

We already have the latest version in the test app. It really looks like a bug in PouchDB, but I can't get reliable repro. It is some kind of race condition, but when I was adding more and more logs at some point it stopped reproducing at all. I doubt it would be easier to reproduce this issue with just PouchDB and node (if possible at all) but I will try it on Monday. On Feb 3, 2017 7:51 PM, "Yaron Y Goland" <notifications@github.com> wrote: Have we tried to update our release of PouchDB to see if perhaps this has been addressed? If you do get a repro then it would be good to file this as a bug with PouchDB even as we work around it. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#1744 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAauvujDsbgFDp5U8kpD9bE8T3VEonWqks5rY1sWgaJpZM4LqCPW> .

[chapko]

Actually this is not a PouchDB bug. Something is wrong with ForverAgent. I tried to run test using default https agent and forever-agent:

• https.Agent 0/100 failures
• ForeverAgent.SSL 23/100 failures.

[chapko]

The bug itself is that when we are trying to make request our callback is never called. It is reproducible with our fork of forever-agent and with original forever-agent.

[yaronyg]

Then it's time to file a bug with forever-agent. I wonder if anyone will notice. :) In the meantime we should obviously fix it in our fork.

[chapko]

I've finally found the problem and this is a bug in forever-agent.

Imagine the scenario when we are making 3 serial requests to the same http server using the same ForeverAgent instance:

1. We create http.ClientRequest object via http.request(...). It asks agent for socket connection. Agents creates new net.Socket object and provides it to the http.ClientRequest object.
2. Request writes body into the provided socket. Once we receive reply via this socket, parser creates http.IncomingMessage stream. As soon as this stream ends, the underlying socket emits 'free' event marking this socket as "free".
3. ForeverAgent listens to "free" event and stores these sockets to reuse them later.
4. Following requests to the same server would result in agent providing used sockets.

So basically if we have "free" socket it can be "borrowed" by some request. As soon as the corresponding response (IncomingMessage) emits 'end' event, our sockets is "free" again.

And everything works fine until we destroy socket.

Destroyed socket emits 'close' event. Agent listens to this event to remove closed sockets. And it works as intended, destroyed socket is removed from the agent, but the problem is that agent still listens to the 'free' event on removed socket. And if we destroy socket in the middle of request and if there is a corresponding response object, response will be ended emitting 'end' event marking the socket as "free" again (1 2 3).

The next part was to figure out if it is agent's job to filter out destroyed sockets or agent should not listen to 'free' event after removing socket or it is node's job to not emit 'free' event on destroyed socket. Nodejs after 0.10 (not sure what exact) version implemented ForeverAgent functionality natively and it just checks if the socket is destroyed before providing it to the request objects. And the latest node has the same problem with ForeverAgent (that means it still emits 'free' events after destroying and default agent still listens to the 'free' event after socket has been removed). So the obvious solution is to check if the socket is destroyed when agent receives 'free' event.

I don't think ForeverAgent is still maintained, nodejs 0.10 and 0.12 are abandoned since the December 2016, and node 4+ supports this functionality natively. Anyway I will provide a pull request to the original repo.

[yaronyg]

@chapko Thanks! This is a painful but necessary catch and yet another reason why we have to figure out how to get off JXcore.