Commits on Jan 25, 2014

1. Switch to BasicHttpClientConnectionManager instead of …

   PoolingHttpClientConnectionManager, and make the connection manager a member
   variable of HttpRequest. We had been seeing a strange bug where once in
   every few thousand urls or so, the connection manager would get gc'd
   while the http request was being sent, and RecordingOutputStream.flush()
   would get called, as in the stack trace below.  The ToeThread was using
   it at the same time, so if conditions were right, the recorded stream
   would get mangled. This commit makes the connection manager a member of
   FetchHTTPRequest so it doesn't get gc'd prematurely. Also switches to
   BasicHttpClientConnectionManager, since we're not doing any pooling.
   
     it.unimi.dsi.fastutil.io.FastBufferedOutputStream.dumpBuffer(boolean) line: 133
     it.unimi.dsi.fastutil.io.FastBufferedOutputStream.flush() line: 166
     org.archive.io.RecordingOutputStream.flush() line: 469
     org.apache.http.impl.io.SessionOutputBufferImpl.flushStream() line: 128
     org.apache.http.impl.io.SessionOutputBufferImpl.flush() line: 143
     org.archive.modules.fetcher.FetchHTTPRequest$RecordingHttpClientConnection(org.apache.http.impl.BHttpConnectionBase).close() line: 314
     org.archive.modules.fetcher.FetchHTTPRequest$RecordingHttpClientConnection.close() line: 520
     org.apache.http.impl.conn.CPoolEntry.closeConnection() line: 69
     org.apache.http.impl.conn.CPoolEntry.close() line: 95
     org.apache.http.impl.conn.CPool(org.apache.http.pool.AbstractConnPool<T,C,E>).shutdown() line: 130
     org.apache.http.impl.conn.PoolingHttpClientConnectionManager.shutdown() line: 347
     org.apache.http.impl.conn.PoolingHttpClientConnectionManager.finalize() line: 168
     java.lang.ref.Finalizer.invokeFinalizeMethod(java.lang.Object) line: not available [native method]

   

   nlevitt committed Jan 25, 2014
   Configuration menu

   • View commit details
   • Copy full SHA for 6b72ac9
   • Browse repository at this point

   Copy the full SHA

   6b72ac9 View commit details

   Browse the repository at this point in the history