Connection-0.2.6 breaks Amazonka (stackage lts-6.12 on up)

[dysinger]

I tracked this down over the last full day of pulling my hair. It was not unlike the bug seen in hs-tls 1.3.4 where you would get bad mac every once in a while.

With connection-0.2.5 I can pound away on amazonka. With connection-0.2.6 I get random bad mac errors.

user@computer:~/src/github.com/fpco/ops% time stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624                               
.....................ops: TransportError (TlsExceptionHostPort (Terminated True "received fatal error: BadRecordMac" (Error_Protocol ("remote side fatal error",True,BadRecordMac))) "s3.amazonaws.com" 443)
stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624  3.27s user 0.99s system 20% cpu 21.052 total
user@computer:~/src/github.com/fpco/ops% time stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624
........................................................ops: TransportError (TlsExceptionHostPort (Terminated True "received fatal error: BadRecordMac" (Error_Protocol ("remote side fatal error",True,BadRecordMac))) "s3.amazonaws.com" 443)
stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624  5.08s user 1.31s system 11% cpu 56.224 total
user@computer:~/src/github.com/fpco/ops% time stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624
........................ops: TransportError (TlsExceptionHostPort (Terminated True "received fatal error: BadRecordMac" (Error_Protocol ("remote side fatal error",True,BadRecordMac))) "s3.amazonaws.com" 443)
stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624  3.28s user 0.95s system 17% cpu 24.207 total
user@computer:~/src/github.com/fpco/ops% time stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624
............................ops: TransportError (TlsExceptionHostPort (Terminated True "received fatal error: BadRecordMac" (Error_Protocol ("remote side fatal error",True,BadRecordMac))) "s3.amazonaws.com" 443)
stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624  3.15s user 0.87s system 14% cpu 27.544 total
user@computer:~/src/github.com/fpco/ops% time stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624
.........................ops: TransportError (TlsExceptionHostPort (Terminated True "received fatal error: BadRecordMac" (Error_Protocol ("remote side fatal error",True,BadRecordMac))) "s3.amazonaws.com" 443)
stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624  3.27s user 0.75s system 15% cpu 25.356 total
user@computer:~/src/github.com/fpco/ops% time stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624
.................................ops: TransportError (TlsExceptionHostPort (Terminated True "received fatal error: BadRecordMac" (Error_Protocol ("remote side fatal error",True,BadRecordMac))) "s3.amazonaws.com" 443)
stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624  3.18s user 1.00s system 12% cpu 33.218 total
user@computer:~/src/github.com/fpco/ops% time stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624
..................................ops: TransportError (TlsExceptionHostPort (Terminated True "received fatal error: BadRecordMac" (Error_Protocol ("remote side fatal error",True,BadRecordMac))) "s3.amazonaws.com" 443)
stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624  2.50s user 1.00s system 10% cpu 34.065 total
user@computer:~/src/github.com/fpco/ops% time stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624
........................ops: TransportError (TlsExceptionHostPort (Terminated True "received fatal error: BadRecordMac" (Error_Protocol ("remote side fatal error",True,BadRecordMac))) "s3.amazonaws.com" 443)
stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624  2.58s user 0.75s system 13% cpu 24.302 total
user@computer:~/src/github.com/fpco/ops% time stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624
........................ops: TransportError (TlsExceptionHostPort (Terminated True "received fatal error: BadRecordMac" (Error_Protocol ("remote side fatal error",True,BadRecordMac))) "s3.amazonaws.com" 443)
stack exec -- ops aws s3 purge 222b3ebc-70a0-11e6-80be-0f425a3fa624  2.22s user 0.84s system 12% cpu 24.030 total

[dysinger]

lts-6.11 completes without error for my project

% diff <(stack --resolver=lts-6.11 list-dependencies) <(stack --resolver=lts-6.12 list-dependencies) 
43c43
< connection 0.2.5
---
> connection 0.2.6

[vincenthz]

Sorry for your hair !
could you try with the latest cryptonite commit 8908af32168f6676b8f98b509ab5376706ab5660 to see if that works for you ?

[dysinger]

My hair is ok :) Here's a script that will reproduce

#!/usr/bin/env stack
-- stack --resolver lts-6.12 --install-ghc runghc --package conduit --package lens --package amazonka --package amazonka-s3

{-# LANGUAGE ViewPatterns #-}
{-# LANGUAGE OverloadedStrings #-}

import Control.Lens
import Control.Monad.IO.Class
import Data.Conduit
import qualified Data.Conduit.List as CL
import Data.Monoid
import Data.String
import Data.Foldable
import Data.Text (Text)
import qualified Data.Text.IO as TI
import Network.AWS
import Network.AWS.Data
import Network.AWS.S3

main = do
    env <- newEnv NorthVirginia Discover
    let say = liftIO . TI.putStrLn
        val
            :: ToText a
            => Maybe a -> Text
        val = maybe "Nothing" toText
        lat v = maybe mempty (mappend " - " . toText) (v ^. ovIsLatest)
        key v = val (v ^. ovKey) <> ": " <> val (v ^. ovVersionId) <> lat v
    runResourceT . runAWS env $
        do say "Listing Buckets .."
           bs <- view lbrsBuckets <$> send listBuckets
           say $ "Found " <> toText (length bs) <> " Buckets."
           forM_ bs $
               \(view bName -> b) -> do
                   say $ "Listing Object Versions in: " <> toText b
                   paginate (listObjectVersions b) =$=
                       CL.concatMap (view lovrsVersions) $$
                       CL.mapM_ (say . mappend " -> " . key)

Run it against any AWS account with some buckets & objects. After about 30 seconds or so it'll crash.

[dysinger]

change the script to lts-6.11 and it'll work fine

[erikd]

I wonder if this is related to erikd-ambiata/test-warp-wai#1 (comment) ?

[dysinger]

Looks similar @erikd

[dysinger]

snoyberg/http-client#225

[vincenthz]

dysinger: I'm not sure how to test this script with a non-released version of cryptonite, any idea ?

[erikd]

I've tried recreating my issue just using the connection library talking to HTTP and HTTPS servers. Was not able to find anything, suggesting that this was actually a problem with http-client.

[vincenthz]

@erikd: I don't think that's the case, definitely a low level crypto issue. did you use the git version of cryptonite ?

[erikd]

Nope, used the 0.19 release version.

[vincenthz]

@erikd I don't think you have the same issue than @dysinger here, this is the BadRecordMac issue

[erikd]

Ah ok, may be a different issue.

[vincenthz]

is this still a bug ? (both connection and tls has new revisions since)

[erikd]

Its still a HANDLE leak as far as I can tell. I tried it with connection 2.7 about a month ago.

I can test again (tomorrow) if you let me know what versions I should be testing.

[vincenthz]

that sounds like a different bug from the BadRecordMac

[erikd]

Opened #29 .