Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[2.0.1] Repeated SIGABRT before finish syncing #3844

Closed
NicolasGoeddel opened this issue Sep 16, 2015 · 25 comments

Comments

@NicolasGoeddel
Copy link

commented Sep 16, 2015

Hi there,

since a few days I have a problem with my Owncloud-Client 2.0.1. It starts, begins to sync and then it gives me a SIGABRT before finish syncing.

Client: Version 2.0.1, Revision 58ea49
OS: Lubuntu 14.04.3 (3.16.0-46-generic x86_64)

Server: Ubuntu 14.04.3
Apache: Version 2.4.7
SSL: Yes

This is the backtrace from gdb. The issue seems to occur in the main Thread.

bundestag@vpn-server:~/bin$ gdb /usr/bin/owncloud
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/owncloud...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/bin/owncloud 
Traceback (most recent call last):
  File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", line 63, in <module>
    from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe5cc8700 (LWP 10892)]
[New Thread 0x7fffe541d700 (LWP 10897)]
[New Thread 0x7fffe4c1c700 (LWP 10898)]
[New Thread 0x7fffd7bc2700 (LWP 10899)]
[New Thread 0x7fffd73c1700 (LWP 10900)]
[New Thread 0x7fffd6104700 (LWP 10901)]
[New Thread 0x7fffd5903700 (LWP 10903)]
[Thread 0x7fffe541d700 (LWP 10897) exited]
[Thread 0x7fffd73c1700 (LWP 10900) exited]

Program received signal SIGABRT, Aborted.
0x00007ffff29d1cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56  ../nptl/sysdeps/unix/sysv/linux/raise.c: Datei oder Verzeichnis nicht gefunden.
(gdb) thread apply all bt

Thread 8 (Thread 0x7fffd5903700 (LWP 10903)):
#0  0x00007ffff2a8812d in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffff082afe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007ffff082b0ec in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff44967a1 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#4  0x00007ffff44680af in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#5  0x00007ffff44683a5 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#6  0x00007ffff4364c5f in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#7  0x00007ffff436732f in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#8  0x00007fffefb49182 in start_thread (arg=0x7fffd5903700) at pthread_create.c:312
#9  0x00007ffff2a9547d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 7 (Thread 0x7fffd6104700 (LWP 10901)):
#0  0x00007ffff2a8812d in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffff082afe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007ffff082b0ec in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff44967a1 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#4  0x00007ffff44680af in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#5  0x00007ffff44683a5 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#6  0x00007ffff4364c5f in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#7  0x00007ffff436732f in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#8  0x00007fffefb49182 in start_thread (arg=0x7fffd6104700) at pthread_create.c:312
#9  0x00007ffff2a9547d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 5 (Thread 0x7fffd7bc2700 (LWP 10899)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff4367816 in QWaitCondition::wait(QMutex*, unsigned long) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#2  0x00007ffff436395b in QSemaphore::acquire(int) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#3  0x00007ffff447dcb2 in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#4  0x00007ffff48c40bc in ?? () from /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4
#5  0x00007ffff482881d in ?? () from /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4
#6  0x00007ffff4829067 in ?? () from /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4
#7  0x00007ffff4829637 in ?? () from /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4
#8  0x00007ffff4829e23 in ?? () from /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4
#9  0x00007ffff482ab12 in ?? () from /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4
#10 0x00007ffff447d87a in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#11 0x00007ffff48ba091 in ?? () from /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4
#12 0x00007ffff48b3eb9 in ?? () from /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4
#13 0x00007ffff447d87a in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#14 0x00007ffff4894ccd in ?? () from /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4
#15 0x00007ffff489dcfd in ?? () from /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4
#16 0x00007ffff4f5de2c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
#17 0x00007ffff4f644a0 in QApplication::notify(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
#18 0x00007ffff44694dd in QCoreApplication::notifyInternal(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#19 0x00007ffff44974a8 in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#20 0x00007ffff082ae04 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#21 0x00007ffff082b048 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#22 0x00007ffff082b0ec in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#23 0x00007ffff44967be in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#24 0x00007ffff44680af in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#25 0x00007ffff44683a5 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#26 0x00007ffff4364c5f in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#27 0x00007ffff436732f in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#28 0x00007fffefb49182 in start_thread (arg=0x7fffd7bc2700) at pthread_create.c:312
#29 0x00007ffff2a9547d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7fffe4c1c700 (LWP 10898)):
#0  0x00007ffff2a8812d in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffff082afe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007ffff082b0ec in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007ffff44967a1 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#4  0x00007ffff44680af in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#5  0x00007ffff44683a5 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#6  0x00007ffff4364c5f in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#7  0x00007ffff436732f in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#8  0x00007fffefb49182 in start_thread (arg=0x7fffe4c1c700) at pthread_create.c:312
#9  0x00007ffff2a9547d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7fffe5cc8700 (LWP 10892)):
#0  0x00007ffff2a8cda3 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffff4446171 in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#2  0x00007ffff436732f in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#3  0x00007fffefb49182 in start_thread (arg=0x7fffe5cc8700) at pthread_create.c:312
#4  0x00007ffff2a9547d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7ffff7fc0840 (LWP 10888)):
#0  0x00007ffff29d1cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff29d50d8 in __GI_abort () at abort.c:89
#2  0x00007ffff435cc92 in qt_message_output(QtMsgType, char const*) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#3  0x00007ffff435cff9 in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#4  0x00007ffff435d804 in qFatal(char const*, ...) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#5  0x00007ffff404766d in OCC::PropagateDirectory::slotSubJobFinished(OCC::SyncFileItem::Status) () from /usr/lib/x86_64-linux-gnu/libowncloudsync.so.0
#6  0x00007ffff4481c1e in QObject::event(QEvent*) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#7  0x00007ffff4f5de2c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
#8  0x00007ffff4f644a0 in QApplication::notify(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
#9  0x00007ffff44694dd in QCoreApplication::notifyInternal(QObject*, QEvent*) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#10 0x00007ffff446cb3d in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#11 0x00007ffff4496f83 in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#12 0x00007ffff082ae04 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#13 0x00007ffff082b048 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#14 0x00007ffff082b0ec in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#15 0x00007ffff44967a1 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#16 0x00007ffff4fffbe6 in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
#17 0x00007ffff44680af in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#18 0x00007ffff44683a5 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#19 0x00007ffff446db79 in QCoreApplication::exec() () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#20 0x00000000004511c3 in main ()

At the moment I solved the problem presumably by exporting OWNCLOUD_MAX_PARALLEL=1 before starting owncloud.
I also have a log file but it has a few hundred MB. I don't know why it grows so fast.

@NicolasGoeddel NicolasGoeddel changed the title Repeated SIGABRT before finish syncing [2.0.1] Repeated SIGABRT before finish syncing Sep 16, 2015

@guruz

This comment has been minimized.

Copy link
Collaborator

commented Sep 16, 2015

FYI @ogoffart

The only assert here I see is

    // We finished to processing all the jobs
    // check if we finished
    if (_current >= total) {
        Q_ASSERT(!_runningNow); // how can we finished if there are still jobs running now
        finalize();
    } else {
        emit ready();
    }
@guruz

This comment has been minimized.

Copy link
Collaborator

commented Sep 16, 2015

This is also the same assert that @rmoesbergen is seeing in #3816 (comment)

@guruz guruz added the bug label Sep 16, 2015

@guruz guruz added this to the 2.0.2-current milestone Sep 16, 2015

@ckamm ckamm added the sev1-critical label Sep 17, 2015

@ckamm

This comment has been minimized.

Copy link
Member

commented Sep 17, 2015

I've spent some time looking at this and unfortunately don't see what could cause this error. The assert seems correct, and the counting was correct (pushed a small cleanup). Everything should be happening in the same thread.

Maybe this slot is called twice for some reason. But then MAX_PARALLEL=1 shouldn't help.

@ogoffart Any ideas?

@NicolasGoeddel

This comment has been minimized.

Copy link
Author

commented Sep 17, 2015

Are _current and total volatile? Did you use a barrier to wait for the other threads to finish?

@ckamm

This comment has been minimized.

Copy link
Member

commented Sep 17, 2015

@NicolasGoeddel The slot is always executed by the same thread. Can you compile the client yourself? If so, I'd love to give you some debugging code.

@NicolasGoeddel

This comment has been minimized.

Copy link
Author

commented Sep 17, 2015

Maybe I can try this at weekend because the client runs in a productive environment. Compiling should be no problem. It seems there are some Ubuntu packages missing which were not mentioned in the building instructions at https://doc.owncloud.org/desktop/2.0/building.htm
When I have figured out what is missing I will comment again here.

@NicolasGoeddel

This comment has been minimized.

Copy link
Author

commented Sep 17, 2015

After installing the packages "build-essential qt4-make qt4-dev-tools pkg-config libneon27-dev qtkeychain-dev libsqlite3-dev" I was able to compile the whole thing. But I do not understand why I should create the directory client-build and the run cmake ../client in it. After compiling the directory client-build remains empty and all binaries were put into client/bin . I would recommend to rework the compiling instructions.

Can I start the newly compiled owncloud-client as a second instance or do I need to quit the running instance first?

@ckamm

This comment has been minimized.

Copy link
Member

commented Sep 17, 2015

@NicolasGoeddel Thanks for trying this! You can run your selfmade client with --confdir path (create the directory first) to ensure it uses a different configuration from your production client. You have to quit your other client before starting this one though.

@NicolasGoeddel

This comment has been minimized.

Copy link
Author

commented Sep 17, 2015

@ckamm Ok then. If I have to quit the other client before starting the compiled one, then we have to wait until the day after tomorrow. It is important that the client runs flawlessly on this specific machine. It would not help to test your debugging code on an other machine.
What debugging code do you want to give me?

ckamm added a commit to ckamm/owncloud-client that referenced this issue Sep 17, 2015

@ckamm

This comment has been minimized.

Copy link
Member

commented Sep 17, 2015

@NicolasGoeddel Okay then. The code is uploaded to my repo as e3b65c5. If you can compile it and run with --logfile my.log and then show all lines that contain "BUG!", that should help us figure out what's going on.

@NicolasGoeddel

This comment has been minimized.

Copy link
Author

commented Sep 17, 2015

@ckamm Please can you give me advice how to checkout this code? Or have I simply to do a 'git pull' and then recompile it with 'cmake -DCMAKE_BUILD_TYPE="Debug" .'?

@ckamm

This comment has been minimized.

Copy link
Member

commented Sep 17, 2015

@NicolasGoeddel Something like this should do it:

git remote add ckamm https://github.com/ckamm/mirall.git
git fetch ckamm
git checkout -b debug ckamm/debug

You do not need to compile in debug mode to get the log output.

@ckamm ckamm added the Needs info label Sep 18, 2015

@NicolasGoeddel

This comment has been minimized.

Copy link
Author

commented Sep 21, 2015

I am now trying to crash owncloud for half an hour and it don't want to crash. The log has already a size of 1.4 GB. I hope it won't grow too fast.

@guruz

This comment has been minimized.

Copy link
Collaborator

commented Sep 25, 2015

Any news here?

@ckamm

This comment has been minimized.

Copy link
Member

commented Sep 30, 2015

@NicolasGoeddel So it crashed easily with "2.0.1, Revision 58ea49" but you could not yet make "ckamm/debug" crash?

@NicolasGoeddel

This comment has been minimized.

Copy link
Author

commented Sep 30, 2015

I was able to crash it after a week again. But this time it's an other error and my log has a size of 44 GB.
This is the new error:

user@oc-server:~/Downloads/client/bin$ ./owncloud --logfile ~/owncloud/filelog.log
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

I can try to grep all lines of the log with a "BUG!" in it into a new file if that would help. But I don't think so.
It would be nice to be able to create smaller logs because a 44 GB log after running the client for 12.5 hours is not good.

@ckamm

This comment has been minimized.

Copy link
Member

commented Sep 30, 2015

@NicolasGoeddel and me talked on IRC. The "bad_alloc" crash was most likely due to logging being enabled. He'll try to reproduce the problem again with --logdir.

@cmonteroluque cmonteroluque modified the milestones: 2.1-next, 2.0.2-current Oct 5, 2015

@guruz

This comment has been minimized.

Copy link
Collaborator

commented Oct 15, 2015

@NicolasGoeddel Did this clear up?

@NicolasGoeddel

This comment has been minimized.

Copy link
Author

commented Oct 15, 2015

The owncloud client is running since Oct 2 without problems. The latest log file is "owncloud.log.27398". I am still using the version "ckamm/debug". So I guess the problem is gone. I don't know why. I could check the old version again, if you think that would help.

@guruz guruz removed the sev1-critical label Oct 16, 2015

@ckamm

This comment has been minimized.

Copy link
Member

commented Oct 16, 2015

@NicolasGoeddel The debug patch didn't change any logic, just add debug output. Maybe you could try with a regular build again?

@NicolasGoeddel

This comment has been minimized.

Copy link
Author

commented Oct 22, 2015

Yeah. I've just got a log with some BUG!s in it. The main problem seems to be this line:

10-20 13:03:45:886 0x178e460 BUG! Finished job running -1 done 2 

I would like to send you the tar.gziped logfile per mail because I don't want to make the whole directory structure available here.

@ckamm

This comment has been minimized.

Copy link
Member

commented Oct 27, 2015

I got the log and will take a deeper look tomorrow. A first glance shows that PropagateDirectory::slotSubJobFinishedseems to be called twice from the same sender, which is a OCC::PropagateUploadFileQNAM.

We should fix the bug in that job and ideally make this robust against bugs in jobs.

@ckamm

This comment has been minimized.

Copy link
Member

commented Oct 28, 2015

@NicolasGoeddel Fixed in master, thanks for the log file! :)

@ckamm ckamm added ReadyToTest and removed Needs info labels Oct 28, 2015

@NicolasGoeddel

This comment has been minimized.

Copy link
Author

commented Oct 28, 2015

@ckamm Nice! I will check out the fix the next days and test it.

@guruz

This comment has been minimized.

Copy link
Collaborator

commented Nov 3, 2015

Closing for now :-) You know when/how to re-open..

@guruz guruz closed this Nov 3, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.