unity-panel-service crashes because of owncloud

[nazar-pc]

Expected behaviour

ownCloud icon displayed constantly

Actual behaviour

ownCloud icon causes all icons to blinks few times per minute (like slow re-rendering of all icons) and eventually unity-panel-service crashes multiple times per day (once it didn't crash for a long time and its memory consumption grown to ridiculous 2.4 GiB). As a side effect, when I click on icon and wait few seconds, menu will be closed on menu re-rendering, so it is not always easy to even open settings.

Steps to reproduce

1. Install ownCloud client nightly 2.1.1+
2. Open it
3. Observe tray icons

80 seconds video shows how icons blink, how memory consumption of unity-panel-service grows and it crashes eventually: https://owncloud.mokrynskyi.com/index.php/s/GWDpN3lexMgSOUG

Client configuration

Client version: 2.2.1~nightly2016617-1.1

Operating system: Ubuntu 16.04 x64 (Unity)

OS language: Ukrainian

Logs

Output of owncloud --logfile log.txt during blinking:

06-20 04:44:43:051 0xad87b0 csync_reconcile: Reconciliation for remote replica took 1,03 seconds visiting 329 files.
06-20 04:44:43:052 0xad87b0 csync_statedb_close: sqlite3_close=0
06-20 04:44:43:053 9675 OCC::SyncEngine::slotDiscoveryJobFinished: <<#### Reconcile end ####################################################  7070
06-20 04:44:43:054 9675 OCC::SyncEngine::slotDiscoveryJobFinished: Permissions of the root folder:  "RDNVCK"
06-20 04:44:43:055 9675 OCC::SyncJournalDb::commitInternal: void OCC::SyncJournalDb::commitInternal(const QString&, bool) Transaction commit  "post treewalk" and starting new transaction
06-20 04:44:43:057 9675 OCC::SyncJournalDb::commitInternal: void OCC::SyncJournalDb::commitInternal(const QString&, bool) Transaction commit  "post stale entry removal" and starting new transaction
06-20 04:44:43:058 9675 OCC::OwncloudPropagator::start: Using QNAM/HTTP parallel code path
06-20 04:44:43:059 9675 OCC::SyncEngine::slotDiscoveryJobFinished: <<#### Post-Reconcile end ####################################################  7076
06-20 04:44:43:069 9675 OCC::SyncEngine::slotItemCompleted: void OCC::SyncEngine::slotItemCompleted(const OCC::SyncFileItem&, const OCC::PropagatorJob&) "" INSTRUCTION_NONE 4 ""
06-20 04:44:43:073 9675 OCC::SyncJournalDb::walCheckpoint: void OCC::SyncJournalDb::walCheckpoint() took 0 msec
06-20 04:44:43:074 9675 OCC::SyncJournalDb::commitInternal: void OCC::SyncJournalDb::commitInternal(const QString&, bool) Transaction commit  "All Finished." 
06-20 04:44:43:076 9675 OCC::SyncJournalDb::close: void OCC::SyncJournalDb::close() "/home/nazar-pc/***/.csync_journal.db"
06-20 04:44:43:076 9675 OCC::SyncJournalDb::commitTransaction: No database Transaction to commit
06-20 04:44:43:078 9675 OCC::SyncEngine::finalize: CSync run took  7095
06-20 04:44:43:078 9675 OCC::BandwidthManager::~BandwidthManager: virtual OCC::BandwidthManager::~BandwidthManager()
06-20 04:44:43:080 9675 OCC::Folder::slotSyncFinished:  - client version 2.2.1nightly2016617 (build 1697)  Qt 5.5.1  SSL  OpenSSL 1.0.2g-fips  1 Mar 2016
06-20 04:44:43:085 9675 OCC::Folder::slotSyncFinished: -> SyncEngine finished without problem.
06-20 04:44:43:086 9675 OCC::Folder::bubbleUpSyncResult: Processing result list and logging took  0  Milliseconds.
06-20 04:44:43:087 9675 OCC::Folder::bubbleUpSyncResult: OO folder slotSyncFinished: result:  2
06-20 04:44:43:089 9675 OCC::SyncJournalDb::checkConnect: sqlite3 version "3.13.0"
06-20 04:44:43:750 9675 OCC::SyncJournalDb::checkConnect: sqlite3 journal_mode= "wal"
06-20 04:44:43:751 9675 OCC::SyncJournalDb::commitInternal: void OCC::SyncJournalDb::commitInternal(const QString&, bool) Transaction commit  "checkConnect" and starting new transaction
06-20 04:44:43:752 9675 OCC::SyncJournalDb::tableColumns: Columns in the current journal:  ("phash", "pathlen", "path", "inode", "uid", "gid", "mode", "modtime", "type", "md5", "fileid", "remotePerm", "filesize", "ignoredChildrenRemote", "contentChecksum", "contentChecksumTypeId")
06-20 04:44:43:754 9675 OCC::SyncJournalDb::commitInternal: void OCC::SyncJournalDb::commitInternal(const QString&, bool) Transaction commit  "update database structure: add inode index" and starting new transaction
06-20 04:44:43:755 9675 OCC::SyncJournalDb::commitInternal: void OCC::SyncJournalDb::commitInternal(const QString&, bool) Transaction commit  "update database structure: add path index" and starting new transaction
06-20 04:44:43:756 9675 OCC::SyncJournalDb::tableColumns: Columns in the current journal:  ("path", "lastTryEtag", "lastTryModtime", "retrycount", "errorstring", "lastTryTime", "ignoreDuration", "renameTarget")
06-20 04:44:43:757 9675 OCC::SyncJournalDb::commitInternal: void OCC::SyncJournalDb::commitInternal(const QString&, bool) Transaction commit  "checkConnect End" 

[guruz]

But this is a bug in Ubuntu then? @sil2100

@jnweiger @ckamm @ogoffart This might be when the tray menu is re-created?

[ckamm]

@nazar-pc Do you see the "Enabled QDBusPlatformMenu workaround" string in your log output? In that case we hide and re-show the tray menu regularly, as a workaround for some other bugs.

[nazar-pc]

Yes, I see following in log file:

06-20 16:38:32:000 28541 OCC::ownCloudGui::setupContextMenu: Enabled QDBusPlatformMenu workaround

[ckamm]

@nazar-pc Could you check if running the client like this

XDG_CURRENT_DESKTOP=xfce /path/to/owncloud

is a workaround?

[nazar-pc]

No, there is still the same message in log file and the same behavior.

[ckamm]

@nazar-pc What happens when you pause all syncs or log out of the account? In that case the tray menu shouldn't be updated as frequently.

[nazar-pc]

I've paused all synchronization for half an our and haven't seen any blinking during this period of time. Also, memory consumption of unity-panel-service didn't grow during this time (before I paused it's memory was 1.7 GiB).

[ckamm]

@nazar-pc My current guess is that the problem appears with setupContextMenu calls. When you unpause one of the folders and then just wait for a bit without triggering synchronization, do you observe abnormal CPU use and memory growth?

[nazar-pc]

CPU use was always low.
Now I unpaused just one directory and opened log window.
As soon as first lines appeared, memory consumption of unity-panel-service grown a bit, and with every new synchronization it grows further.
Here are first lines that appeared and when memory growth happened:

06-23 11:11:29:835 15044 OCC::Folder::slotRunEtagJob: * Trying to check "https://owncloud.*/remote.php/webdav/xyz" for changes via ETag check. (time since last sync: 93 s)
06-23 11:11:29:835 15044 OCC::FolderMan::slotRunOneEtagJob: Scheduling "https://owncloud.*/remote.php/webdav/xyz" to check remote ETag
06-23 11:11:29:836 15044 OCC::AbstractNetworkJob::start: !!! OCC::RequestEtagJob created for "https://owncloud.*" + "/xyz" "OCC::Folder"

2 minutes - memory consumption grown by ~3 MiB.

[ckamm]

@nazar-pc It looks like whenever the ownCloud tray menu is adjusted, unity-panel-service leaks some memory. It's probably a bug there, user programs shouldn't be able to break it in this way.

I've added a minimal tray menu option (enabled by setting the OWNCLOUD_MINIMAL_TRAY_MENU environment variable) for testing purposes. I guess the symptoms will go away when it is set. You'll either need to compile the 2.2 branch yourself or wait for a new nightly https://build.opensuse.org/package/show/isv:ownCloud:community:nightly/owncloud-client

[nazar-pc]

I'm on nightly channel already, will test it as soon as new build is out

[nazar-pc]

Got fresh nightly from 3a77074. Still blinking as before and still causes unity-panel-service to crash from time to time.

[ckamm]

@nazar-pc Was that with or without OWNCLOUD_MINIMAL_TRAY_MENU being set?

[nazar-pc]

Yeah, setting OWNCLOUD_MINIMAL_TRAY_MENU really prevents this issue.
Even when icon changes - nothing blinks and drop-down menu is not closing on icon update.

[ckamm]

@nazar-pc Thanks for testing. This comment applies here too: #4990 (comment)

I'll see whether I can reproduce the problem in a VM and get some hint about the culprit - right now it seems unlikely that it'd be something the client can fix.

[danimo]

@ckamm I am running owncloud routinely here without any problems. Probably related to specific sync conditions.

[ckamm]

Thanks @danimo! I've also not seen in in my VM yet. unity-panel-service takes a surprising 30M - but I haven't watched it long-term yet.

@nazar-pc Can you tell us more about your setup? Do you have several accounts, several sync folders or anything else that might make this happen for you and not for us?

[nazar-pc]

I have 2 ownCloud servers/accounts connected.
One have 5 directories configured for synchronization, another have 6 directories.

[ckamm]

@nazar-pc Okay, I've added some more sync folder configs to my test.
Btw, @nazar-pc @danimo, do the account submenus and the "Recent Changes" submenu show for you in the tray context menu? They don't show up for me (stock Qt 5.5.1 that Ubuntu 16.04 comes with).

[nazar-pc]

No, only "close" item there. I'm on 16.10, but it is also with Qt 5.5.1, so no substantial difference here.

[ckamm]

@nazar-pc I meant without OWNCLOUD_MINIMAL_TRAY_MENU?

[nazar-pc]

Without they are present, 2 accounts and recent changes

[ckamm]

@nazar-pc Sorry, one more claification: do you see sub-menus on these three menus? (like "open in browser", "sign in/out", etc. for accounts)

[ckamm]

Possible workaround: #5072

Note that it does not fix the underlying issue, but should make it less apparent by updating the context menu less often.

[goghard]

I experienced similar problems with OC client 2.2.2-19.1 on Ubuntu 16.04 but I found a workaround: setting up the repository to 14.04. First, I removed the repository from the system list and then added the 14.04 one:
sudo sh -c "echo 'deb http://download.opensuse.org/repositories/isv:/ownCloud:/desktop/Ubuntu_14.04/ /' >> /etc/apt/sources.list.d/owncloud-client.list"
Finnally, I used a full-upgrade and restarted the client.

[ckamm]

Even though this is an upstream bug, the problem should be significantly reduced with the latest master (2.3.x) builds.

[nazar-pc]

I only receive 2.2.3 nightly builds, is there any other repository for Ubuntu than http://download.opensuse.org/repositories/isv:/ownCloud:/community:/nightly/xUbuntu_16.04/ ?

[guruz]

@danimo @nazar-pc I'll change the nightlies to build master/2.3 branch.

[guruz]

Did you try to confirm this with a 2.3 nightly?
https://owncloud.org/install/#testing-development

[nazar-pc]

Both nightly and http://download.opensuse.org/repositories/isv:/ownCloud:/community:/testing/xUbuntu_16.04/ contain only 2.2.3 builds

[guruz]

Sorry, should be working fine tomorrow, please try then.

[nazar-pc]

They already are since yesterday I guess.
I'm not using Unity right now, so I'll install it and then reply if it helped.

[nazar-pc]

Looks good, memory doesn't leak, but I can't use menu anymore. When I click on icon it feels like I've clicked twice - menu closes in a fraction of second.
However, as I click to open menu I see that memory consumption of unity-panel-service is constantly growing by 0.7-1 MiB

[guruz]

Closing in favour of #5164 (comment)

[guruz]

@nazar-pc Do you also want to try the PR?
#5179

[nazar-pc]

Will it be in Nighly? I'm not using Unity anymore, so I need to install it and reboot to try.
Compile ownCloud client from source in addition to that is quite a lot of an effort:)

[kichooo]

Same problem. Running that for some time, and I get 1GB consumed.

[syzer]

Is RAM issue resolved?

[guruz]

1GB consumed where? This is the ownCloud client bugtracker, if the memory is consumed in other software please try there.

[kichooo]

Hello guruz - it is the ubuntu-panel-service. I didn't have a problem before. It happens only with owncloud client. I will try to report it on ubuntu tracker as well.

Also, I got redirected here from Ubuntu: https://bugs.launchpad.net/ubuntu/+source/unity/+bug/1199877

[IvarSnaaijer]

The problem is currently closed on both sides. The problem seems to only happen with the owncloud app. After a day (screen is locked most of the time) unity has racked up 10Gb of data in the unity-panel-service. This does not happen when the client is paused. Blaming it on the other guy will not solve the problem.

[mitya57]

@IvarSnaaijer The fix for Ubuntu package is in progress, tracked at LP: #1636677.

[guruz]

The problem seems to only happen with the owncloud app

2.3 nightly might improve this, please try.

[mabkenar]

According to this report, this problem is fixed in 2.3.0git.

[kichooo]

I have a clean install with apt-get versions of both programs and it doesn't reproduce, whereas I had it happening on every single one of the deployments I did before, irregardless of hardware. Looks like this bug is killed.