Make httpc connection closing asynchronous (#47)

httpc pool's terminate/2 can block up to 5 seconds in case when ssl sockets
are used. Previously this was not an issue as replication jobs would run until
completion and only exit once. With the scheduler, replication jobs are stopped
regularly during rescheduling. So terminate/2 could be called `max_churn` number
of times in a row during each reschedule cycle. This could lead to scheduler
gen_server being blocked for up to 100 seconds at a time.

To fix, unlink the connection and call stop in a separate process. This way,
socket will get a chance to be closed properly, but if it doesn't, ibrowse
client will kill it (this ensures connection processes won't leak).

src/couch_replicator_httpc_pool.erl

@@ -147,9 +147,15 @@ code_change(_OldVsn, #state{}=State, _Extra) ->
     {ok, State}.
 
 
-terminate(_Reason, State) ->
-    lists:foreach(fun ibrowse_http_client:stop/1, State#state.free),
-    lists:foreach(fun ibrowse_http_client:stop/1, State#state.busy).
+terminate(_Reason, #state{free=Free, busy=Busy}) ->
+    % ibrowse_http_client stop can take up to 5 seconds to close an ssl socket
+    % (if it times out after five, it brutally kills process)
+    [begin
+        unlink(Con),
+        spawn(ibrowse_http_client, stop, [Con])
+    end || Con <- Free ++ Busy],
+    ok.
+
 
 monitor_client(Callers, Worker, {ClientPid, _}) ->
     [{Worker, erlang:monitor(process, ClientPid)} | Callers].