handleToFd leaks memory

[ghost]

On unix-2.6.0.1 with GHC 7.6.3 the following program leaks memory very fast:

import Control.Monad
import System.IO
import System.IO.Error
import System.Posix.IO
import System.Posix.Types

test :: IO ()
test = do
  let path = "/tmp/leak-test"
  handle <- openFile path WriteMode
  putStrLn $ "Opened file " ++ path
  fd <- handleToFd handle
  putStrLn $ "Converted to handle " ++ path
  _ <- tryIOError $ closeFd fd
  putStrLn $ "Closed file " ++ path

main = forever test

This is only due to handleToFd. Opening a file directly using openFd doesn't leak.

[rwbarton]

I can't reproduce this. I see variable but bounded virtual memory size for this program, never exceeding about 60M, with resident size around half that.

[ghost]

What version of unix are you using and what version of GHC? I can
reproduce it quite reliably with the included program, on some setups it
consumes several GBs of virtual memory in a few seconds.

On Tue, Jul 29, 2014 at 9:24 PM, rwbarton notifications@github.com wrote:

I can't reproduce this. I see variable but bounded virtual memory size for
this program, never exceeding about 60M, with resident size around half
that.

—
Reply to this email directly or view it on GitHub
#4 (comment).

[snoyberg]

@pudlak How are you compiling the program? Are you passing in any RTS options?

[ghost]

@snoyberg I don't pass any optimization or RTS options, but anyway it seems it doesn't really matter. Here is a cabal setup that produces such an executable:

name:                leak
version:             0.1.0.0
license:             PublicDomain
author:              Petr Pudlak
build-type:          Simple
cabal-version:       >=1.10

executable leak
  build-depends:       base >=4.6 && <4.7, unix ==2.7.0.1
  default-language:    Haskell2010
  main-is:             Leak.hs

I always compile and run it inside a schroot, IDK if this could somehow contribute to the problem. I tested it in both Wheezy (GHC 7.4.1, unix-2.5.0.1) and Squeeze schroot (GHC 7.6.3 and unix-2.[6,7].0.1), the problem manifests in all these setups.

[snoyberg]

I'm still unable to repro. Can you try without schroot and see if it still occurs?

[gregorycollins]

Can you reproduce the problem with a Dockerfile or vagrant script? Given the trouble others are having reproducing this, if we can get a repeatable build setup that illustrates the bug via VM/container that would be helpful here.

[ghost]

@gregorycollins I'll try to do that. @snoyberg So far I'm unable to reproduce it outside schroot as well.

[ppetr]

It can be reproduced in a docker image my coworker (@nh2) created: nh2docker/ganeti-ubuntu1404-code-syshs
The image already contains GHC, just compile the sample code from above and run. It eats around 100MB RAM per second.

[nomeata]

I can reproduce this using the docker image.

[hvr]

@nomeata any idea on how to fix this?

[rwbarton]

This seems to be some issue with broken locales related to GHC Trac #7695 and #10298. I can reproduce it on an ordinary system after export LC_ALL=badlc. I'd guess the allocations are happening in iconv.

[argiopetech]

Nice find. It doesn't leak very quickly, but this definitely triggers the issue on my machine.

[rwbarton]

OK, the cause is rather simple. Compare the eventual implementation of hClose from base

hClose_handle_ :: Handle__ -> IO (Handle__, Maybe SomeException)
hClose_handle_ h_@Handle__{..} = do

    -- close the file descriptor, but not when this is the read
    -- side of a duplex handle.
    -- If an exception is raised by the close(), we want to continue
    -- to close the handle and release the lock if it has one, then
    -- we return the exception to the caller of hClose_help which can
    -- raise it if necessary.
    maybe_exception <-
      case haOtherSide of
        Nothing -> trymaybe $ IODevice.close haDevice
        Just _  -> return Nothing

    -- free the spare buffers
    writeIORef haBuffers BufferListNil
    writeIORef haCharBuffer noCharBuffer
    writeIORef haByteBuffer noByteBuffer

    -- release our encoder/decoder
    closeTextCodecs h_

    -- we must set the fd to -1, because the finalizer is going
    -- to run eventually and try to close/unlock it.
    -- ToDo: necessary?  the handle will be marked ClosedHandle
    -- XXX GHC won't let us use record update here, hence wildcards
    return (Handle__{ haType = ClosedHandle, .. }, maybe_exception)

to that of handleToFd from unix

handleToFd' :: Handle -> Handle__ -> IO (Handle__, Fd)
handleToFd' h h_@Handle__{haType=_,..} = do
  case cast haDevice of
    Nothing -> ioError (ioeSetErrorString (mkIOError IllegalOperation
                                           "handleToFd" (Just h) Nothing)
                        "handle is not a file descriptor")
    Just fd -> do
     -- converting a Handle into an Fd effectively means
     -- letting go of the Handle; it is put into a closed
     -- state as a result.
     flushWriteBuffer h_
     FD.release fd
     return (Handle__{haType=ClosedHandle,..}, Fd (FD.fdFD fd))

flushWriteBuffer is just

flushWriteBuffer h_@Handle__{..} = do
  buf <- readIORef haByteBuffer
  when (isWriteBuffer buf) $ flushByteWriteBuffer h_

so clearly handleToFd' is rather delinquent in freeing resources, particularly the text codecs.

The reason the (large) leak does not normally appear is just that GHC implements UTF-8 encoding itself, without using iconv. Any locale for which GHC uses iconv produces the same behavior, not just non-existent locales.

It looks like this should be fixable by just copying the relevant parts of hClose_handle_ into handleToFd', though the resulting cross-package code duplication is unfortunate.

[hasufell]

Tried to reproduce it in 2022... the only GHC I could reproduce it with is 8.0.2. Next unix release won't support 8.0.2 or earlier, so it appears this issue is resolved.