GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
Already on GitHub? Sign in to your account
Full description available at: https://github.com/blitzcode/http-conduit-bug-reproducer
Originally reported on Reddit: http://www.reddit.com/r/haskell/comments/1s58dh/ann_wai_20_and_httpclient/cdu4ahl
In the Reddit comment I mentioned this:
After my programs goes about its business for a minute or two, all connection requests fail for a while with this one:
interspersed with this one:
HttpException: InternalIOException send: invalid argument (Socket is not connected)
this still seems to be the case even with the older http-conduit 220.127.116.11. If I modify the bug reproducer like this:
21 catch ( runResourceT $ do
22 req <- parseUrl url
23 res <- httpLbs req manager
24 liftIO $ do BL.writeFile "/dev/null" $ responseBody res
25 tid <- show <$> myThreadId
26 printf "%s: %s\n" tid url
28 ( \(e :: SomeException) -> do tid <- show <$> myThreadId
29 printf "%s: Exception: %s\n" tid $ show e
and build with -threaded and run with +RTS -N I also get this behavior:
ThreadId 9: Exception: FailedConnectionException "pbs.twimg.com" 80
ThreadId 9: Exception: InternalIOException send: invalid argument (Socket is not connected)
The program will run fine for a minute or two and then completely jam up like this, stall, freeze etc.
I also noticed that the runtime performance of my actual program has decreased. Unfortunately, I don't know the exact version of all the dependencies I had installed originally. But it seems like the last few versions of http-conduit + dependencies have made some performance regressions plus this complete jamming up under load. Then finally >18.104.22.168 appears to just give seemingly random errors.
I'll see if I can track down which dependency / version caused this.
Alright, I've identified the bug: connections are being both put back into the manager and closed, leading to garbage. The easy solution is to add an extra mutable reference to ensure only one of these actions happens. This is what the previous version of http-conduit did implicitly by using resourcet. I want to see if there's a simple way to stop the double-free calls.
Ensure no double-close of connections #9
After looking at the code, I've decided that the mutable reference protection approach is the right thing to do, since we want to have the behavior that calling responseClose multiple times is a safe operation. I've released this as a new version of http-client to Hackage (0.2.0.2); can you confirm that upgrading solves the problem?
I can confirm that this solves the issue in the bug reproducer and my actual program. Thank you!
Maybe it would make sense to bump the http-client dependency in http-conduit up to 0.2.0.2?
Good idea, new version released. Thanks again for reporting this bug so quickly, I'm glad you caught it so early.
Regarding the performance degradation: if you get more information, can you open it as a separate issue?
I had some time to investigate the performance drop, it seems this is entirely due to my code. The dependencies alter the timing a bit, and my code was rather sensitive to that. After some tweaking and optimizing it's all equally fast with the new http-conduit. So, false alarm ;-)
Awesome, I'm glad to hear it, thanks for reporting back.