Memory leak?

[samoht]

The com.docker.db process memory usage is growing up regularly, even when there is no activity -- this can grow very large over time.

Need to be fixed ASAP

Reported by @aluzzardi and @stevvooe on https://docker.slack.com/archives/pinata-dev/p1459565205001093

[talex5]

I added this to the main file:

Store.master (fun () -> Irmin.Task.empty) repo >|= fun master ->
Store.watch_head (master ()) (fun _ -> print_endline "notify"; Lwt.return ()) |> ignore;

That causes the leak without having anything connect over 9p, running stand-alone.

(reducing the polling time makes the problem appear faster)

[talex5]

After some investigation with @samoht, the problem seems to be that listing a directory with Lwt_unix leaks. e.g. this program rapidly increases its memory consumption:

open Lwt

let rec loop () =
  Lwt_unix.opendir "/" >>= fun handle ->
  Lwt_unix.readdir_n handle 1024 >>= fun entries ->
  Lwt_unix.closedir handle >>= fun () ->
  ignore entries;
  loop ()

let () =
  Lwt_main.run (loop ())

[talex5]

Upstream bug: ocsigen/lwt#229

[talex5]

Running with:

valgrind --leak-check=full --show-leak-kinds=definite ./main.native --git ~/go/src/github.com/docker/pinata

reports

==68436== HEAP SUMMARY:
==68436==     in use at exit: 120,642,439 bytes in 4,955 blocks
==68436==   total heap usage: 32,646 allocs, 27,691 frees, 284,346,383 bytes allocated
==68436== 
==68436== 148 (80 direct, 68 indirect) bytes in 1 blocks are definitely lost in loss record 221 of 333
==68436==    at 0x1008FFEBB: malloc (in /usr/local/Cellar/valgrind/3.11.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==68436==    by 0x100C6E8D6: __Balloc_D2A (in /usr/lib/system/libsystem_c.dylib)
==68436==    by 0x100C6F21F: __d2b_D2A (in /usr/lib/system/libsystem_c.dylib)
==68436==    by 0x100C6B877: __dtoa (in /usr/lib/system/libsystem_c.dylib)
==68436==    by 0x100C943E6: __vfprintf (in /usr/lib/system/libsystem_c.dylib)
==68436==    by 0x100CBD6C8: __v2printf (in /usr/lib/system/libsystem_c.dylib)
==68436==    by 0x100CA1914: _vsnprintf (in /usr/lib/system/libsystem_c.dylib)
==68436==    by 0x100CA19C9: vsnprintf (in /usr/lib/system/libsystem_c.dylib)
==68436==    by 0x10029881D: caml_alloc_sprintf (in ./main.native)
==68436==    by 0x100258426: .L3074 (in ./main.native)
==68436==    by 0x2: ???
==68436==    by 0x101538337: ???
==68436== 
==68436== 680 (176 direct, 504 indirect) bytes in 1 blocks are definitely lost in loss record 271 of 333
==68436==    at 0x1008FFEBB: malloc (in /usr/local/Cellar/valgrind/3.11.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==68436==    by 0x10095BA03: CRYPTO_malloc (in /usr/lib/libcrypto.0.9.8.dylib)
==68436==    by 0x100957FC5: lh_new (in /usr/lib/libcrypto.0.9.8.dylib)
==68436==    by 0x10096DDC2: def_get_class (in /usr/lib/libcrypto.0.9.8.dylib)
==68436==    by 0x10096D6A6: int_get_new_index (in /usr/lib/libcrypto.0.9.8.dylib)
==68436==    by 0x100A8E454: SSL_get_ex_data_X509_STORE_CTX_idx (in /usr/lib/libssl.0.9.8.dylib)
==68436==    by 0x100A931C7: SSL_CTX_new (in /usr/lib/libssl.0.9.8.dylib)
==68436==    by 0x10027E36E: ocaml_ssl_create_context (in ./main.native)
==68436==    by 0x10017BF43: .L257 (in ./main.native)
==68436==    by 0x100001098: .L204 (in ./main.native)
==68436==    by 0x1002A3B67: caml_start_program (in ./main.native)
==68436==    by 0xFFFFFFFF: ???
==68436== 
==68436== 2,928 (176 direct, 2,752 indirect) bytes in 1 blocks are definitely lost in loss record 303 of 333
==68436==    at 0x1008FFEBB: malloc (in /usr/local/Cellar/valgrind/3.11.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==68436==    by 0x10095BA03: CRYPTO_malloc (in /usr/lib/libcrypto.0.9.8.dylib)
==68436==    by 0x100957FC5: lh_new (in /usr/lib/libcrypto.0.9.8.dylib)
==68436==    by 0x1009C5889: OBJ_NAME_add (in /usr/lib/libcrypto.0.9.8.dylib)
==68436==    by 0x1009A523F: EVP_add_cipher (in /usr/lib/libcrypto.0.9.8.dylib)
==68436==    by 0x100A8C762: SSL_library_init (in /usr/lib/libssl.0.9.8.dylib)
==68436==    by 0x10027E020: ocaml_ssl_init (in ./main.native)
==68436==    by 0x100188054: .L176 (in ./main.native)
==68436==    by 0x1050F9B0D: ???
==68436==    by 0x1050F9B65: ???
==68436==    by 0x1050F9B7A: ???
==68436==    by 0x1050F9B8A: ???
==68436== 
==68436== 57,088 (176 direct, 56,912 indirect) bytes in 1 blocks are definitely lost in loss record 318 of 333
==68436==    at 0x1008FFEBB: malloc (in /usr/local/Cellar/valgrind/3.11.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==68436==    by 0x10095BA03: CRYPTO_malloc (in /usr/lib/libcrypto.0.9.8.dylib)
==68436==    by 0x100957FC5: lh_new (in /usr/lib/libcrypto.0.9.8.dylib)
==68436==    by 0x1009B33D0: int_err_get (in /usr/lib/libcrypto.0.9.8.dylib)
==68436==    by 0x1009B35B5: int_err_set_item (in /usr/lib/libcrypto.0.9.8.dylib)
==68436==    by 0x1009B2B9C: ERR_load_strings (in /usr/lib/libcrypto.0.9.8.dylib)
==68436==    by 0x100987680: ERR_load_ERR_strings (in /usr/lib/libcrypto.0.9.8.dylib)
==68436==    by 0x1009A72B8: ERR_load_crypto_strings (in /usr/lib/libcrypto.0.9.8.dylib)
==68436==    by 0x100A90E98: SSL_load_error_strings (in /usr/lib/libssl.0.9.8.dylib)
==68436==    by 0x10027E028: ocaml_ssl_init (in ./main.native)
==68436==    by 0x100188054: .L176 (in ./main.native)
==68436==    by 0x1050F9B0D: ???
==68436== 
==68436== 4,126,320 (2,342,416 direct, 1,783,904 indirect) bytes in 281 blocks are definitely lost in loss record 329 of 333
==68436==    at 0x1008FFEBB: malloc (in /usr/local/Cellar/valgrind/3.11.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==68436==    by 0x1002843AF: lwt_unix_readdir_n_job (lwt_unix_stubs.c:101)
==68436==    by 0x1001939BD: .L937 (in ./main.native)
==68436==    by 0x1001AA5AF: .L782 (in ./main.native)
==68436==    by 0x10018F351: .L179 (in ./main.native)
==68436==    by 0x1001A6F71: .L119 (in ./main.native)
==68436==    by 0x1001A8BC6: .L451 (in ./main.native)
==68436==    by 0x1001A8EDB: .L489 (in ./main.native)
==68436==    by 0x10023718B: .L383 (in ./main.native)

The others are single blocks, so probably just there because I hit Ctrl-C, or maybe some initialisation stuff.

[samoht]

See #57

[stevvooe]

@samoht @talex5 Thanks for investigating. Let me know when this hits dev channel and we'll see if this addressed it.

[samoht]

should have landed in pinata's dev channel now. com.docker.db --version should return v0.3.0.

[stevvooe]

@samoht Seems stable. I'll let you know if there are issues.