flatpak build-bundle causes glib memory error when operating on v. large bundles #483

Open
rmacqueen opened this Issue Jan 11, 2017 · 3 comments

Projects

None yet

3 participants

@rmacqueen

Running flatpak bundle-build on a bundle that is very large ( > 2GB) generates the following error:

(/usr/bin/flatpak build-bundle:28630): GLib-ERROR **: /usr/src/packages/BUILD/glib2.0-2.50.2+dev22.0cb2266/./glib/gmem.c:165: failed to allocate 4294967296 bytes

Running the same command under gdb produces this backtrace:

#0  0x00007ffff6d243c1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#1  0x00007ffff6d253ed in g_log_default_handler ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007ffff6d256d1 in g_logv () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff6d2591f in g_log () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007ffff6d24046 in g_realloc ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007ffff6d40247 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#6  0x00007ffff6d4048f in g_string_set_size ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#7  0x00007ffff78f7436 in ?? () from /lib/x86_64-linux-gnu/libostree-1.so.1
#8  0x00007ffff78f7a35 in ?? () from /lib/x86_64-linux-gnu/libostree-1.so.1
#9  0x00007ffff78f9e91 in ostree_repo_static_delta_generate ()
   from /lib/x86_64-linux-gnu/libostree-1.so.1
#10 0x000055555557848f in build_bundle (repo=0x1, repo@entry=0x555555805000, 
    file=0x1, file@entry=0x6, 
    name=0x1 <error: Cannot access memory at address 0x1>, 
    name@entry=0x555555809f20 "\220\071\200UUU", 
    full_branch=0x5555557fef50 "PQ", cancellable=0x7fffe0000ae0, 
    cancellable@entry=0x0, error=0x0)
    at app/flatpak-builtins-build-bundle.c:245
#11 0x00005555555799a2 in flatpak_builtin_build_bundle (argc=5, 
    argv=0x7fffffffdf98, cancellable=0x0, error=0x7fffffffde28)
    at app/flatpak-builtins-build-bundle.c:948
#12 0x000055555556a772 in flatpak_run (argc=5, argv=0x7fffffffdf98, 
    res_error=0x7fffffffde78) at app/flatpak-main.c:366
#13 0x0000555555569e92 in main (argc=6, argv=0x7fffffffdf98)
    at app/flatpak-main.c:454
@alexlarsson
Member

Its unclear why you can't allocate a 4 gigabyte chunk, how much memory do you have?
However, its clearly a problem that we're building up the entire bundle in memory before saving it.
@cgwalters: Did you ever look at doing streaming builds of static deltas?

@rmacqueen
rmacqueen commented Jan 12, 2017 edited

I was running on a VM, to which I had assigned 4GB of memory. I just upped that to 8GB and tried again, and got the same error (with same backtrace), though, curiously, this time it fails trying to allocate 2GB.

(/usr/bin/flatpak build-bundle:1573): GLib-ERROR **: /usr/src/packages/BUILD/glib2.0-2.50.2+dev22.0cb2266/./glib/gmem.c:165: failed to allocate 2147483648 bytes

@cgwalters
Contributor

Potentially, it's a bit tricky as to stream we'd have to move away from GVariant, or at least the standard GVariant APIs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment