Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

large pull sometimes segfaults in g_mutex_lock(<some invalid pointer>) #601

Closed
smcv opened this issue Nov 29, 2016 · 9 comments
Closed

large pull sometimes segfaults in g_mutex_lock(<some invalid pointer>) #601

smcv opened this issue Nov 29, 2016 · 9 comments
Labels
bug

Comments

@smcv
Copy link
Contributor

smcv commented Nov 29, 2016

This is on Debian stable with backports: currently GLib 2.42, ostree 2016.13, flatpak 0.6.13, libsoup 2.48.

A large pull (downloading new Flatpak runtimes) sometimes segfaults. I was able to get this backtrace:

(gdb) thread apply all bt

Thread 11 (Thread 0x7fffe9ffb700 (LWP 11080)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007ffff6d9d8fc in g_mutex_unlock_slowpath (mutex=<optimized out>, 
    prev=<optimized out>)
    at /build/glib2.0-y6934K/glib2.0-2.42.1/./glib/gthread-posix.c:1330
#2  0x00007ffff6d9e30e in g_mutex_unlock (mutex=<optimized out>)
    at /build/glib2.0-y6934K/glib2.0-2.42.1/./glib/gthread-posix.c:1350
#3  0x00007ffff6d578a5 in g_source_attach (source=source@entry=0x7fffc8001950, 
    context=0x5555557e5390)
    at /build/glib2.0-y6934K/glib2.0-2.42.1/./glib/gmain.c:1146
#4  0x00007ffff72e47d0 in run_in_thread (job=<optimized out>, 
    c=<optimized out>, _data=0x55555580a4a0)
    at /build/glib2.0-y6934K/glib2.0-2.42.1/./gio/gsimpleasyncresult.c:867
#5  0x00007ffff72d15a6 in io_job_thread (task=<optimized out>, 
    source_object=<optimized out>, task_data=0x7fffec185880, 
    cancellable=<optimized out>)
    at /build/glib2.0-y6934K/glib2.0-2.42.1/./gio/gioscheduler.c:85
#6  0x00007ffff72f4c65 in g_task_thread_pool_thread (
    thread_data=0x7fffec0e2350, pool_data=<optimized out>)
    at /build/glib2.0-y6934K/glib2.0-2.42.1/./gio/gtask.c:1215
#7  0x00007ffff6d811d8 in g_thread_pool_thread_proxy (data=<optimized out>)
    at /build/glib2.0-y6934K/glib2.0-2.42.1/./glib/gthreadpool.c:307
---Type <return> to continue, or q <return> to quit---
#8  0x00007ffff6d80845 in g_thread_proxy (data=0x7fffec185d90)
    at /build/glib2.0-y6934K/glib2.0-2.42.1/./glib/gthread.c:764
#9  0x00007ffff66e00a4 in start_thread (arg=0x7fffe9ffb700)
    at pthread_create.c:309
#10 0x00007ffff641562d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 5 (Thread 0x7ffff2a9a700 (LWP 11074)):
#0  0x00007ffff6d9e2e5 in g_mutex_lock (
    mutex=mutex@entry=0x7ffff72d49c0 <g_memory_input_stream_skip_async>)
    at /build/glib2.0-y6934K/glib2.0-2.42.1/./glib/gthread-posix.c:1337
#1  0x00007ffff6d571b8 in g_source_destroy_internal (source=0x555555833d20, 
    context=0x7ffff72d49c0 <g_memory_input_stream_skip_async>, have_lock=0)
    at /build/glib2.0-y6934K/glib2.0-2.42.1/./glib/gmain.c:1159
#2  0x00007ffff765bca8 in ?? () from /usr/lib/x86_64-linux-gnu/libsoup-2.4.so.1
#3  0x00007ffff70341f5 in g_object_unref (_object=0x7fffec008260)
    at /build/glib2.0-y6934K/glib2.0-2.42.1/./gobject/gobject.c:3133
#4  0x00007ffff790ba62 in ?? () from /usr/lib/x86_64-linux-gnu/libostree-1.so.1
#5  0x00007ffff6d80845 in g_thread_proxy (data=0x5555557dbed0)
    at /build/glib2.0-y6934K/glib2.0-2.42.1/./glib/gthread.c:764
#6  0x00007ffff66e00a4 in start_thread (arg=0x7ffff2a9a700)
    at pthread_create.c:309
#7  0x00007ffff641562d in clone ()
---Type <return> to continue, or q <return> to quit---
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7ffff7fde8c0 (LWP 11049)):
#0  0x00007ffff66e6a7d in write () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffff6d9cef2 in g_wakeup_signal (wakeup=0x7fffec11b470)
    at /build/glib2.0-y6934K/glib2.0-2.42.1/./glib/gwakeup.c:239
#2  0x00007ffff790caf5 in ?? () from /usr/lib/x86_64-linux-gnu/libostree-1.so.1
#3  0x00007ffff703426a in g_object_unref (_object=0x555555833240)
    at /build/glib2.0-y6934K/glib2.0-2.42.1/./gobject/gobject.c:3170
#4  0x00007ffff78e8013 in ostree_repo_pull_with_options ()
   from /usr/lib/x86_64-linux-gnu/libostree-1.so.1
#5  0x0000555555588a8f in ?? ()
#6  0x00007fffffffe170 in ?? ()
#7  0x00005555557e1000 in ?? ()
#8  0x0000000000000000 in ?? ()

(Sorry, currently no debug symbols on the test system for libsoup or ostree - I'll try a rebuild with debug symbols when I have more time to investigate.)

Thread 5 is the one segfaulting. The mutex being a pointer to totally the wrong thing might be optimization noise, or it might indicate a use-after-free?
@cgwalters
Copy link
Member

Might be related to #383
and https://bugzilla.gnome.org/show_bug.cgi?id=768567

Though I haven't re-verified whether the libsoup patch there is still necessary.

@cgwalters cgwalters added the bug label Nov 29, 2016
@smcv smcv changed the title large pull sometimes segfaults on Debian stable large pull sometimes segfaults Nov 30, 2016
@smcv
Copy link
Contributor Author

smcv commented Nov 30, 2016

Here's a better backtrace, from Debian unstable (GLib 2.50.2 and libsoup 2.56.0):

(gdb) thread apply all bt

Thread 4 (Thread 0x7fcf30ba1700 (LWP 8869)):
#0  0x00007fcf39c7e119 in syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007fcf3a49826a in g_cond_wait_until (cond=cond@entry=0x7fcf3cef29a8, mutex=mutex@entry=0x7fcf3cef29a0, end_time=end_time@entry=3591488307905) at ././glib/gthread-posix.c:1442
#2  0x00007fcf3a426e89 in g_async_queue_pop_intern_unlocked (queue=0x7fcf3cef29a0, wait=wait@entry=1, end_time=3591488307905) at ././glib/gasyncqueue.c:422
#3  0x00007fcf3a4274e8 in g_async_queue_timeout_pop_unlocked (queue=<optimized out>, timeout=timeout@entry=500000) at ././glib/gasyncqueue.c:570
#4  0x00007fcf3a47ad16 in g_thread_pool_thread_proxy (pool=<optimized out>) at ././glib/gthreadpool.c:262
#5  0x00007fcf3a47ad16 in g_thread_pool_thread_proxy (data=<optimized out>) at ././glib/gthreadpool.c:296
#6  0x00007fcf3a47a345 in g_thread_proxy (data=0x7fcf3cefbca0) at ././glib/gthread.c:784
#7  0x00007fcf39f3f464 in start_thread (arg=0x7fcf30ba1700) at pthread_create.c:333
#8  0x00007fcf39c829df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105

Thread 3 (Thread 0x7fcf32487700 (LWP 8868)):
#0  0x00007fcf39c7956d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007fcf3a4529f6 in g_main_context_iterate (priority=<optimized out>, n_fds=1, fds=0x7fcf3cef0fe0, timeout=<optimized out>, context=0x7fcf3cef2ae0) at ././glib/gmain.c:4228
#2  0x00007fcf3a4529f6 in g_main_context_iterate (context=context@entry=0x7fcf3cef2ae0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ././glib/gmain.c:3924
#3  0x00007fcf3a452b0c in g_main_context_iteration (context=0x7fcf3cef2ae0, may_block=may_block@entry=1) at ././glib/gmain.c:3990
#4  0x00007fcf3a452b51 in glib_worker_main (data=<optimized out>) at ././glib/gmain.c:5783
#5  0x00007fcf3a47a345 in g_thread_proxy (data=0x7fcf3cee98f0) at ././glib/gthread.c:784
#6  0x00007fcf39f3f464 in start_thread (arg=0x7fcf32487700) at pthread_create.c:333
#7  0x00007fcf39c829df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105

Thread 2 (Thread 0x7fcf3b820940 (LWP 8866)):
#0  0x00007fcf39f406bd in pthread_join (threadid=140527886960384, thread_return=thread_return@entry=0x0) at pthread_join.c:90
#1  0x00007fcf3a498047 in g_system_thread_wait (thread=0x7fcf3cee98a0) at ././glib/gthread-posix.c:1212
#2  0x00007fcf3a47a7aa in g_thread_join (thread=0x7fcf3cee98a0) at ././glib/gthread.c:952
#3  0x00007fcf3b0496f6 in _ostree_fetcher_finalize (object=0x7fcf3ceefd20 [OstreeFetcher]) at src/libostree/ostree-fetcher.c:650
#4  0x00007fcf3a730c7a in g_object_unref (_object=0x7fcf3ceefd20) at ././gobject/gobject.c:3185
#5  0x00007fcf3b0212ba in reinitialize_fetcher (pull_data=pull_data@entry=0x7ffeb5aed4d0, remote_name=remote_name@entry=0x7fcf3ceedc80 "origin", error=error@entry=0x7ffeb5aed850) at src/libostree/ostree-repo-pull.c:2294
#6  0x00007fcf3b0268d9 in ostree_repo_pull_with_options (self=<optimized out>, remote_name_or_baseurl=0x7fcf3ceedc80 "origin", options=options@entry=0x7fcf3ceefe10, progress=progress@entry=0x0, cancellable=cancellable@entry=0x0, error=error@entry=0x7ffeb5aed850) at src/libostree/ostree-repo-pull.c:2929
#7  0x00007fcf3b6ba734 in ostree_builtin_pull (argc=<optimized out>, argv=<optimized out>, cancellable=0x0, error=0x7ffeb5aed850) at src/ostree/ot-builtin-pull.c:296
#8  0x00007fcf3b6b18c5 in ostree_run (argc=<optimized out>, argv=<optimized out>, commands=0x7fcf3b8ce020 <commands>, res_error=0x7ffeb5aed8a0) at src/ostree/ot-main.c:204
#9  0x00007fcf3b6a9010 in main (argc=4, argv=0x7ffeb5aed9a8) at src/ostree/main.c:78

Thread 1 (Thread 0x7fcf32c88700 (LWP 8867)):
#0  0x00007fcf3a497fe5 in g_mutex_lock (mutex=mutex@entry=0xe2e2e2e2e2e2e2e2) at ././glib/gthread-posix.c:1336
#1  0x00007fcf3a44fb50 in g_source_destroy_internal (source=0x7fcf3cef0d20, context=0xe2e2e2e2e2e2e2e2, have_lock=0) at ././glib/gmain.c:1217
#2  0x00007fcf3a450365 in g_source_destroy (source=<optimized out>) at ././glib/gmain.c:1285
#3  0x00007fcf3ad832f9 in soup_session_dispose (object=0x7fcf3cefa140 [SoupSessionAsync]) at soup-session.c:317
#4  0x00007fcf3a730c05 in g_object_unref (_object=0x7fcf3cefa140) at ././gobject/gobject.c:3148
#5  0x00007fcf3b0481bf in ostree_fetcher_session_thread (data=0x7fcf3cef1830) at src/libostree/ostree-fetcher.c:588
#6  0x00007fcf3a47a345 in g_thread_proxy (data=0x7fcf3cee98a0) at ././glib/gthread.c:784
#7  0x00007fcf39f3f464 in start_thread (arg=0x7fcf32c88700) at pthread_create.c:333
#8  0x00007fcf39c829df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105

I had hoped that I was running with a locally-built GLib and libsoup with -Og -g, but apparently not :-(

I'll try to reproduce this with locally-built libraries so I can try the patches from https://bugzilla.gnome.org/show_bug.cgi?id=768567.

@smcv
Copy link
Contributor Author

smcv commented Dec 1, 2016 •
edited
Loading

It seems I can reliably reproduce this with a locally-built GLib, libsoup and ostree master by running several ostree pull tests in parallel:

for x in `seq 1 100`; do echo "==$x=="; systemd-run --user --scope env LD_LIBRARY_PATH=$prefix/lib make -j -C $builddir check-TESTS TESTS="tests/test-pull-repeated.sh tests/test-pull-archive-z.sh tests/test-pull-commit-only.sh tests/test-pull-contenturl.sh tests/test-pull-corruption.sh tests/test-pull-depth.sh tests/test-pull-metalink.sh tests/test-pull-large-metadata.sh tests/test-pull-mirrorlist.sh tests/test-pull-mirror-summary.sh tests/test-pull-override-url.sh tests/test-pull-resume.sh tests/test-pull-subpath.sh tests/test-pull-summary-sigs.sh tests/test-pull-untrusted.sh" || { echo "Error in iteration $x: $?"; break }; done

(On my laptop it has failed in or before iteration 3 every time so far, with either this, #605 or #583.)

@smcv smcv changed the title large pull sometimes segfaults large pull sometimes segfaults in g_mutex_lock(<some invalid pointer>) Dec 1, 2016
This was referenced Dec 1, 2016
Open
Closed
@cgwalters
Copy link
Member

Can you reproduce this with one of the sanitizers, particularly -fsanitize=address? I've been trying that without much luck.

@smcv
Copy link
Contributor Author

smcv commented Dec 1, 2016

Can you reproduce this with one of the sanitizers, particularly -fsanitize=address?

Sadly no, I just get it to hang (#605 or similar).

@smcv
Copy link
Contributor Author

smcv commented Dec 1, 2016

With asan, and without the patches from libsoup, I can repro this very quickly (no need to use -j and spam all the pull tests as fast as possible):

==11202==ERROR: AddressSanitizer: heap-use-after-free on address 0x6080000091c0 at pc 0x7fa906d74d8e bp 0x7fa8fc4e7b20 sp 0x7fa8fc4e7b18
READ of size 8 at 0x6080000091c0 thread T1
    #0 0x7fa906d74d8d in g_source_destroy /home/smcv/src/glib/glib/gmain.c:1282
    #1 0x7fa9089ef488 in soup_session_dispose /home/smcv/src/libsoup/libsoup/soup-session.c:317
    #2 0x7fa9074fdaa6 in g_object_unref /home/smcv/src/glib/gobject/gobject.c:3148
    #3 0x7fa9090c63c7 in ostree_fetcher_session_thread /home/smcv/src/ostree/src/libostree/ostree-fetcher.c:588
    #4 0x7fa906df8b30 in g_thread_proxy /home/smcv/src/glib/glib/gthread.c:784
    #5 0x7fa90587e463 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7463)
    #6 0x7fa9055c19de in __clone (/lib/x86_64-linux-gnu/libc.so.6+0xe89de)

0x6080000091c0 is located 32 bytes inside of 96-byte region [0x6080000091a0,0x608000009200)
freed by thread T1 here:
    #0 0x7fa909b6fa10 in free (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1a10)
    #1 0x7fa906d90968 in g_free /home/smcv/src/glib/glib/gmem.c:189

previously allocated by thread T0 here:
    #0 0x7fa909b6fed0 in calloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1ed0)
    #1 0x7fa906d908b7 in g_malloc0 /home/smcv/src/glib/glib/gmem.c:124

Thread T1 created by T0 here:
    #0 0x7fa909adef59 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x30f59)
    #1 0x7fa906e5c334 in g_system_thread_new /home/smcv/src/glib/glib/gthread-posix.c:1170

SUMMARY: AddressSanitizer: heap-use-after-free /home/smcv/src/glib/glib/gmain.c:1282 in g_source_destroy
Shadow bytes around the buggy address:
  0x0c107fff91e0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c107fff91f0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c107fff9200: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c107fff9210: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c107fff9220: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c107fff9230: fa fa fa fa fd fd fd fd[fd]fd fd fd fd fd fd fd
  0x0c107fff9240: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c107fff9250: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c107fff9260: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c107fff9270: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c107fff9280: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==11202==ABORTING

==11951==ERROR: AddressSanitizer: heap-use-after-free on address 0x6080000093c0 at pc 0x7f4f6165ad8e bp 0x7f4f56dcdb20 sp 0x7f4f56dcdb18
READ of size 8 at 0x6080000093c0 thread T1
    #0 0x7f4f6165ad8d in g_source_destroy /home/smcv/src/glib/glib/gmain.c:1282
    #1 0x7f4f632d5488 in soup_session_dispose /home/smcv/src/libsoup/libsoup/soup-session.c:317
    #2 0x7f4f61de3aa6 in g_object_unref /home/smcv/src/glib/gobject/gobject.c:3148
    #3 0x7f4f639ac3c7 in ostree_fetcher_session_thread /home/smcv/src/ostree/src/libostree/ostree-fetcher.c:588
    #4 0x7f4f616deb30 in g_thread_proxy /home/smcv/src/glib/glib/gthread.c:784
    #5 0x7f4f60164463 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7463)
    #6 0x7f4f5fea79de in __clone (/lib/x86_64-linux-gnu/libc.so.6+0xe89de)

0x6080000093c0 is located 32 bytes inside of 96-byte region [0x6080000093a0,0x608000009400)
freed by thread T1 here:
    #0 0x7f4f64455a10 in free (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1a10)
    #1 0x7f4f61676968 in g_free /home/smcv/src/glib/glib/gmem.c:189

previously allocated by thread T0 here:
    #0 0x7f4f64455ed0 in calloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1ed0)
    #1 0x7f4f616768b7 in g_malloc0 /home/smcv/src/glib/glib/gmem.c:124

Thread T1 created by T0 here:
    #0 0x7f4f643c4f59 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x30f59)
    #1 0x7f4f61742334 in g_system_thread_new /home/smcv/src/glib/glib/gthread-posix.c:1170

SUMMARY: AddressSanitizer: heap-use-after-free /home/smcv/src/glib/glib/gmain.c:1282 in g_source_destroy
Shadow bytes around the buggy address:
  0x0c107fff9220: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c107fff9230: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c107fff9240: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c107fff9250: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c107fff9260: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c107fff9270: fa fa fa fa fd fd fd fd[fd]fd fd fd fd fd fd fd
  0x0c107fff9280: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c107fff9290: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c107fff92a0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
  0x0c107fff92b0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c107fff92c0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==11951==ABORTING

# ==12208==ERROR: AddressSanitizer: heap-use-after-free on address 0x608000009340 at pc 0x7f9abd4c2d8e bp 0x7f9ab2c35b20 sp 0x7f9ab2c35b18
# READ of size 8 at 0x608000009340 thread T1
#     #0 0x7f9abd4c2d8d in g_source_destroy /home/smcv/src/glib/glib/gmain.c:1282
#     #1 0x7f9abf13d488 in soup_session_dispose /home/smcv/src/libsoup/libsoup/soup-session.c:317
#     #2 0x7f9abdc4baa6 in g_object_unref /home/smcv/src/glib/gobject/gobject.c:3148
#     #3 0x7f9abf8143c7 in ostree_fetcher_session_thread /home/smcv/src/ostree/src/libostree/ostree-fetcher.c:588
#     #4 0x7f9abd546b30 in g_thread_proxy /home/smcv/src/glib/glib/gthread.c:784
#     #5 0x7f9abbfcc463 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7463)
#     #6 0x7f9abbd0f9de in __clone (/lib/x86_64-linux-gnu/libc.so.6+0xe89de)
# 
# 0x608000009340 is located 32 bytes inside of 96-byte region [0x608000009320,0x608000009380)
# freed by thread T1 here:
#     #0 0x7f9ac02bda10 in free (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1a10)
#     #1 0x7f9abd4de968 in g_free /home/smcv/src/glib/glib/gmem.c:189
# 
# previously allocated by thread T0 here:
#     #0 0x7f9ac02bded0 in calloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1ed0)
#     #1 0x7f9abd4de8b7 in g_malloc0 /home/smcv/src/glib/glib/gmem.c:124
# 
# Thread T1 created by T0 here:
#     #0 0x7f9ac022cf59 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x30f59)
#     #1 0x7f9abd5aa334 in g_system_thread_new /home/smcv/src/glib/glib/gthread-posix.c:1170
# 
# SUMMARY: AddressSanitizer: heap-use-after-free /home/smcv/src/glib/glib/gmain.c:1282 in g_source_destroy
# Shadow bytes around the buggy address:
#   0x0c107fff9210: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
#   0x0c107fff9220: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
#   0x0c107fff9230: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
#   0x0c107fff9240: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
#   0x0c107fff9250: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
# =>0x0c107fff9260: fa fa fa fa fd fd fd fd[fd]fd fd fd fd fd fd fd
#   0x0c107fff9270: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
#   0x0c107fff9280: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
#   0x0c107fff9290: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
#   0x0c107fff92a0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
#   0x0c107fff92b0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
# Shadow byte legend (one shadow byte represents 8 application bytes):
#   Addressable:           00
#   Partially addressable: 01 02 03 04 05 06 07 
#   Heap left redzone:       fa
#   Heap right redzone:      fb
#   Freed heap region:       fd
#   Stack left redzone:      f1
#   Stack mid redzone:       f2
#   Stack right redzone:     f3
#   Stack partial redzone:   f4
#   Stack after return:      f5
#   Stack use after scope:   f8
#   Global redzone:          f9
#   Global init order:       f6
#   Poisoned by user:        f7
#   Container overflow:      fc
#   Array cookie:            ac
#   Intra object redzone:    bb
#   ASan internal:           fe
#   Left alloca redzone:     ca
#   Right alloca redzone:    cb
# ==12208==ABORTING

@smcv
Copy link
Contributor Author

smcv commented Dec 1, 2016

So I think we can safely say that the libsoup patches are an improvement.

This makes me wonder whether the times I thought I reproduced this with the libsoup patches, but without asan, were PEBKAC.

@cgwalters
Copy link
Member

IME when using asan I had to rebuild dependent libraries (mostly glib) with -Og -fno-omit-frame-pointer to get useful backtraces, since the sanitizers AFAIK don't use DWARF unwinding.

@smcv
Copy link
Contributor Author

smcv commented Jan 14, 2017

This has not recurred on ci.debian.net since I landed the libsoup patches, so I think we can consider it closed NOTOURBUG.

@smcv smcv closed this as completed Jan 14, 2017
cgwalters added a commit to cgwalters/ostree that referenced this issue Jan 18, 2017
@cgwalters
tests: Add a big (many objects) pull
1729f0b 
This would be more likely to tickle things like
ostreedev#601
reliably.

Also, while working on the curl backend, I hit on the fact that curl doesn't
queue (by default, you can enable) and will happily create 20000+ concurrent TCP
connections if you try. Having this test would have made that more likely to
fail.
@cgwalters cgwalters mentioned this issue Jan 18, 2017
Closed
rh-atomic-bot pushed a commit that referenced this issue Jan 23, 2017
@cgwalters @rh-atomic-bot
tests: Add a big (many objects) pull
d1f4d94 
This would be more likely to tickle things like
#601
reliably.

Also, while working on the curl backend, I hit on the fact that curl doesn't
queue (by default, you can enable) and will happily create 20000+ concurrent TCP
connections if you try. Having this test would have made that more likely to
fail.

Closes: #650
Approved by: giuseppe
@cgwalters cgwalters mentioned this issue Feb 4, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cgwalters @smcv