Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Buggy behavior in multithreaded code #9

Closed
snoyberg opened this Issue · 7 comments

2 participants

Michael Snoyman Tim C. Schröder
Tim C. Schröder

In the Reddit comment I mentioned this:

After my programs goes about its business for a minute or two, all connection requests fail for a while with this one:
FailedConnectionException
interspersed with this one:
HttpException: InternalIOException send: invalid argument (Socket is not connected)

this still seems to be the case even with the older http-conduit 1.9.4.2. If I modify the bug reproducer like this:

 21                 catch ( runResourceT $ do
 22                               req <- parseUrl url
 23                               res <- httpLbs req manager
 24                               liftIO $ do BL.writeFile "/dev/null" $ responseBody res
 25                                           tid <- show <$> myThreadId
 26                                           printf "%s: %s\n" tid url
 27                       )
 28                       ( \(e :: SomeException) -> do tid <- show <$> myThreadId
 29                                                     printf "%s: Exception: %s\n" tid $ show e
 30                       )

and build with -threaded and run with +RTS -N I also get this behavior:

...
ThreadId 9: Exception: FailedConnectionException "pbs.twimg.com" 80
ThreadId 9: Exception: InternalIOException send: invalid argument (Socket is not connected)
...

The program will run fine for a minute or two and then completely jam up like this, stall, freeze etc.

I also noticed that the runtime performance of my actual program has decreased. Unfortunately, I don't know the exact version of all the dependencies I had installed originally. But it seems like the last few versions of http-conduit + dependencies have made some performance regressions plus this complete jamming up under load. Then finally >1.9.4.2 appears to just give seemingly random errors.

I'll see if I can track down which dependency / version caused this.

Michael Snoyman
Owner

Alright, I've identified the bug: connections are being both put back into the manager and closed, leading to garbage. The easy solution is to add an extra mutable reference to ensure only one of these actions happens. This is what the previous version of http-conduit did implicitly by using resourcet. I want to see if there's a simple way to stop the double-free calls.

Michael Snoyman
Owner

After looking at the code, I've decided that the mutable reference protection approach is the right thing to do, since we want to have the behavior that calling responseClose multiple times is a safe operation. I've released this as a new version of http-client to Hackage (0.2.0.2); can you confirm that upgrading solves the problem?

Tim C. Schröder

I can confirm that this solves the issue in the bug reproducer and my actual program. Thank you!

Maybe it would make sense to bump the http-client dependency in http-conduit up to 0.2.0.2?

Michael Snoyman
Owner

Good idea, new version released. Thanks again for reporting this bug so quickly, I'm glad you caught it so early.

Regarding the performance degradation: if you get more information, can you open it as a separate issue?

Michael Snoyman snoyberg closed this
Tim C. Schröder

I had some time to investigate the performance drop, it seems this is entirely due to my code. The dependencies alter the timing a bit, and my code was rather sensitive to that. After some tweaking and optimizing it's all equally fast with the new http-conduit. So, false alarm ;-)

Michael Snoyman
Owner

Awesome, I'm glad to hear it, thanks for reporting back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.