Buggy behavior in multithreaded code #9

Closed
snoyberg opened this Issue Dec 5, 2013 · 7 comments

Comments

Projects
None yet
2 participants

In the Reddit comment I mentioned this:

After my programs goes about its business for a minute or two, all connection requests fail for a while with this one:
FailedConnectionException
interspersed with this one:
HttpException: InternalIOException send: invalid argument (Socket is not connected)

this still seems to be the case even with the older http-conduit 1.9.4.2. If I modify the bug reproducer like this:

 21                 catch ( runResourceT $ do
 22                               req <- parseUrl url
 23                               res <- httpLbs req manager
 24                               liftIO $ do BL.writeFile "/dev/null" $ responseBody res
 25                                           tid <- show <$> myThreadId
 26                                           printf "%s: %s\n" tid url
 27                       )
 28                       ( \(e :: SomeException) -> do tid <- show <$> myThreadId
 29                                                     printf "%s: Exception: %s\n" tid $ show e
 30                       )

and build with -threaded and run with +RTS -N I also get this behavior:

...
ThreadId 9: Exception: FailedConnectionException "pbs.twimg.com" 80
ThreadId 9: Exception: InternalIOException send: invalid argument (Socket is not connected)
...

The program will run fine for a minute or two and then completely jam up like this, stall, freeze etc.

I also noticed that the runtime performance of my actual program has decreased. Unfortunately, I don't know the exact version of all the dependencies I had installed originally. But it seems like the last few versions of http-conduit + dependencies have made some performance regressions plus this complete jamming up under load. Then finally >1.9.4.2 appears to just give seemingly random errors.

I'll see if I can track down which dependency / version caused this.

Owner

snoyberg commented Dec 9, 2013

Alright, I've identified the bug: connections are being both put back into the manager and closed, leading to garbage. The easy solution is to add an extra mutable reference to ensure only one of these actions happens. This is what the previous version of http-conduit did implicitly by using resourcet. I want to see if there's a simple way to stop the double-free calls.

Owner

snoyberg commented Dec 9, 2013

After looking at the code, I've decided that the mutable reference protection approach is the right thing to do, since we want to have the behavior that calling responseClose multiple times is a safe operation. I've released this as a new version of http-client to Hackage (0.2.0.2); can you confirm that upgrading solves the problem?

I can confirm that this solves the issue in the bug reproducer and my actual program. Thank you!

Maybe it would make sense to bump the http-client dependency in http-conduit up to 0.2.0.2?

Owner

snoyberg commented Dec 9, 2013

Good idea, new version released. Thanks again for reporting this bug so quickly, I'm glad you caught it so early.

Regarding the performance degradation: if you get more information, can you open it as a separate issue?

@snoyberg snoyberg closed this Dec 9, 2013

I had some time to investigate the performance drop, it seems this is entirely due to my code. The dependencies alter the timing a bit, and my code was rather sensitive to that. After some tweaking and optimizing it's all equally fast with the new http-conduit. So, false alarm ;-)

Owner

snoyberg commented Dec 11, 2013

Awesome, I'm glad to hear it, thanks for reporting back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment