Migration from 2.8.1 to 2.9.0 fails silently #9019

Vogtinator · 2021-09-14T09:17:52Z

After upgrading from 2.8.1 to 2.9.0, it shows a message that "multiple accounts are sharing the same folder" and started to sync paths which were explicitly excluded before. The sync folder contains both .sync_journal.db and .sync_286ccd77d0ee.db.

Neither issue happens if I rename .sync_286ccd77d0ee.db to the new .sync_journal.db name before starting 2.9.0.

Server configuration

Not relevant (I think)

Client configuration

Client version: From 2.8.1 to 2.9.0

Operating system: openSUSE Tumbleweed

OS language: German

Qt version used by client package (Linux only, see also Settings dialog): 5.15.2 (before and after update)

Client package (From ownCloud or distro) (Linux only): Using the package from the distro (2.8.1-2.1 to 2.9.0-1.1), which has no patches.

Logs

Nothing really relevant unfortunately. The log shows that it fills a new database:

09-14 11:11:01:133 [ debug sync.database.sql ]  [ OCC::SqlQuery::bindValue ]:   SQL bind 1 2
09-14 11:11:01:133 [ debug sync.database.sql ]  [ OCC::SqlQuery::bindValue ]:   SQL bind 2 9
09-14 11:11:01:133 [ debug sync.database.sql ]  [ OCC::SqlQuery::bindValue ]:   SQL bind 3 0
09-14 11:11:01:133 [ debug sync.database.sql ]  [ OCC::SqlQuery::bindValue ]:   SQL bind 4 0
09-14 11:11:01:133 [ debug sync.database.sql ]  [ OCC::SqlQuery::exec ]:        SQL exec "INSERT INTO version VALUES (?1, ?2, ?3, ?4);"
09-14 11:11:01:133 [ debug sync.database.sql ]  [ OCC::SqlQuery::exec ]:        Last exec affected 1 rows.

I should mention that for reproducing this to get logs, I just "undid" the migration by doing mv .sync_journal.db .sync_286ccd77d0ee.db.

The text was updated successfully, but these errors were encountered:

TheOneRing · 2021-09-14T09:49:49Z

I assume you also got to see the setup wizard and you where asked to setup your account again.
In 2.9 we migrate old client configurations to a new config folder, the unbranded ownCloud version however should not have been affected.

Vogtinator · 2021-09-14T09:53:44Z

I assume you also got to see the setup wizard and you where asked to setup your account again.

I did not see any setup wizard. I only noticed the issue because of unusual high network activity, which made me open the client window.

TheOneRing · 2021-09-14T09:57:27Z

Well then I'm a bit lost on what might have happened :(

Vogtinator · 2021-09-14T09:58:52Z

Can you link to the code which is responsible for either migrating the database to the new location or selecting the old database name? Maybe I can spot something.

TheOneRing · 2021-09-14T10:03:31Z

The migration does not do anything to your database it only migrates the location of the config file containing the name of your db.
4314447#diff-157772a2ade24319c7637b70253590dddfb55f6d4902b814582aca1a2ce23bc8

Vogtinator · 2021-09-14T12:04:44Z

The migration does not do anything to your database it only migrates the location of the config file containing the name of your db.
4314447#diff-157772a2ade24319c7637b70253590dddfb55f6d4902b814582aca1a2ce23bc8

Ok. That approach probably failed here because the configuration here does not include journalPath keys:

[Accounts]
0\Folders\ownCloud\ignoreHiddenFiles=true
0\Folders\ownCloud\localPath=/home/fvogt/ownCloud/
0\Folders\ownCloud\paused=false
0\Folders\ownCloud\targetPath=/
0\authType=http
...

I guess the previous version(s) never wrote that into the file?

TheOneRing · 2021-09-15T07:56:57Z

But how https://github.com/owncloud/client/blob/master/src/gui/folder.cpp#L1256

TheOneRing · 2021-09-15T07:57:51Z

The settings folder should contain backups of your config

Vogtinator · 2021-09-15T08:01:51Z

It looks like this:

-rw-r--r--  1 fvogt    50   125 15. Sep 09:02 cookies0.db
-rw-r--r--  1 fvogt    50   377 14. Mär 2017  cookies.db
drwxr-xr-x  2 fvogt    50  4096  8. Dez 2015  folders
-rw-r--r--  1 fvogt users  3174 15. Sep 09:03 owncloud.cfg
-rw-r--r--  1 fvogt users  1565 15. Okt 2018  owncloud.cfg.backup_20181015_153124
-rw-r--r--  1 fvogt users  1597 21. Nov 2018  owncloud.cfg.backup_20181121_135453_2.5.0
-rw-r--r--  1 fvogt users  1615 28. Feb 2019  owncloud.cfg.backup_20190228_132204_2.5.1
-rw-r--r--  1 fvogt users  1664  2. Mai 2019  owncloud.cfg.backup_20190502_100026_2.5.3
-rw-r--r--  1 fvogt users  1667 24. Feb 2020  owncloud.cfg.backup_20200224_095230_2.5.4
-rw-r--r--  1 fvogt users  1672 15. Jul 13:28 owncloud.cfg.backup_20210715_132817_2.6.1
-rw-r--r--  1 fvogt users  1672 13. Sep 16:11 owncloud.cfg.backup_20210913_161144_2.8.1

None of them contain journalPath.

Vogtinator · 2021-09-15T08:06:57Z

I guess Folder::saveToSettings is only called when settings actually changed, so that might not have happened for a long time.

I set this up sometime 2015 and haven't changed any folder settings since then.

TheOneRing · 2021-09-15T08:07:36Z

Awesome the old migration code sets a journalPath but never saves it 😢 https://github.com/owncloud/client/blob/master/src/gui/folderman.cpp#L293

Fixes: #9019

TheOneRing · 2021-09-15T13:38:41Z

@gabi18 Please consider the following testing scenarios:

2.4 -> 2.8 -> 2.9.1
2.4 -> 2.9.0 -> 2.9.1
2.9.0 -> 2.9.1

What happens when I emulate the old behaviour and remove the journalPath from the config.

Vogtinator · 2021-09-15T14:49:22Z

I built the 2.9 branch locally and gave it a try.

2.4 -> 2.8 -> 2.9.1

That did unfortunately not work. Before 2.9.1, only .sync_286ccd77d0ee.db is present, but 2.9.1 then ignores that and created a new ._sync_286ccd77d0ee.db instead (???)

2.4 -> 2.9.0 -> 2.9.1

That worked, it continued to use the .sync_journal.db created by 2.9.0. However, the old database is still there, so the warning about multiple accounts will stay.

TheOneRing · 2021-09-15T15:00:41Z

I built the 2.9 branch locally and gave it a try.

2.4 -> 2.8 -> 2.9.1

That did unfortunately not work. Before 2.9.1, only .sync_286ccd77d0ee.db is present, but 2.9.1 then ignores that and created a new ._sync_286ccd77d0ee.db instead (???)

._sync_286ccd77d0ee.db Looks like a 2.4 db where is sync_286ccd77d0ee.db coming from?

2.4 -> 2.9.0 -> 2.9.1

That worked, it continued to use the .sync_journal.db created by 2.9.0. However, the old database is still there, so the warning about multiple accounts will stay.

That is expected deleting stuff is always dangerous and will still require manual intervention.

Vogtinator · 2021-09-15T15:06:44Z

._sync_286ccd77d0ee.db Looks like a 2.4 db where is sync_286ccd77d0ee.db coming from?

Sorry, that was supposed to be .sync_92839cef8e49.db. For some reason after the 2.9.1 migration the hash changed.

Using GDB, I see the difference in the key used to generate the hash:

user@https://host:/ (2.8.1 config) vs. user@https://host/:/ (2.9.1 migrated config)

So the config migration added a trailing / to the URL, which results in it no longer finding the database...

TheOneRing · 2021-09-15T15:09:46Z

Regarding 2.4 -> 2.8 -> 2.9.1

Yes thats broken since 5211d9a

So we probably need a 2.4 -> 2.5 -> 2.6 .......

I need another round to fix the migration from ._sync to .sync.....

silvain · 2021-09-16T19:02:20Z

I have 6 owncloud accounts, and 2 of them got the error message «Multiple accounts are sharing the folder /home/silvain/ownCloud/...».
Reading the explanation above, I tried one thing which worked:
Deleting the old synchronization db files (hidden files) from the data folders of the 2 accounts affected.
._sync_5ff816b09f82.db
._sync_5ff816b09f82.db-shm
._sync_5ff816b09f82.db-wal
(only one was present in one case, and I actually moved them somewhere else to be safe).
This was enough for the error message to disappear.

Why did this happen?

I suspect this happened only on two of my accounts... because these are the older ones I have, and the previous upgrad(s) kept these old versions of the db files, which did not have any effect until this recent upgrade.
These deleted db files are all dated May 21, 2019.

Vogtinator · 2021-09-17T08:45:37Z

I think this got closed by accident, it's not fixed yet

TheOneRing · 2021-09-17T08:50:43Z

Yes thx for mentioning, I merged a commit with a Fixes: to master

TheOneRing · 2021-09-17T11:17:18Z

@gabi18 we are ready for another round of tests:

2.4 -> 2.5 -> 2.6 -> 2.7-> 2.8 -> 2.9.1
2.4 -> 2.8 -> 2.9.1
2.4 -> 2.9.0 -> 2.9.1
2.9.0 -> 2.9.1

The idea is to keep the old database but if a previous migration already created a duplicated db we only take the newest.

Vogtinator · 2021-09-17T12:21:01Z

With #9046 the issue with the hash generation remains. Instead of using the existing sync journal, it creates a new empty one again. This time it's properly called .sync_journal.db though, instead of using the wrong hash.

TheOneRing · 2021-09-17T12:28:30Z

#9019 (comment) I didn't get that the hash changed...
Ill have another look...

Vogtinator · 2021-09-21T08:52:35Z

I tried 2.9 with #9054 merged and it appears to be working, thanks!

While testing 2.8.1 -> 2.9 again I only had to reenter the password, which is apparently not migrated from the old entry format to the new one.

TheOneRing · 2021-09-21T09:03:23Z

Thank you very much ❤️
Credentials should be migrated but I can imagine that also your tests could have messed up stuff.
Anyhow the pain of reentering a pw is much less than what we experienced here.

Many thanks for your reporting and the ongoing testing.

Vogtinator · 2021-09-21T09:07:17Z

Credentials should be migrated but I can imagine that also your tests could have messed up stuff.

I actually started from a clean keychain (kwallet) to avoid that. Where can I find the migration code?

TheOneRing · 2021-09-21T09:13:35Z

Here we read the old credentials https://github.com/owncloud/client/blob/master/src/libsync/creds/httpcredentials_p.h#L81 they are then saved in https://github.com/owncloud/client/blob/master/src/libsync/creds/httpcredentials.cpp#L354 .

Vogtinator · 2021-09-21T09:54:57Z

I had a quick look:

2.8.1 creates a password entry in the format of username:https://host/:0 with the password, nothing else.

The 2.9 branch reads this, and then tries to delete a nonexistant username:https://host/ entry. For some reason (removeEntry seems to be dead code), QtKeychain instead calls writePassword with empty data, so it creates an empty entry instead of doing nothing...

On top of that QtKeychain bug there is another weirdness which I can't explain so far: The account is "offline" immediately after migration, and I need to restart the client to trigger a connection. Could be some missing signal emission or something like that.

TheOneRing · 2021-09-21T10:27:41Z

Lets continue in #9064

gabi18 · 2021-09-27T11:07:12Z

Testing on Windows 10 VFS OFF:

2.4.3 -> 2.5.4 -> ~~2.6~~ -> 2.7.6-> 2.8.2 -> 2.9.1daily20210927.5255

Result: for all versions the name of the db remains unchanged, journalPath is written to owncloud.cfg.

owncloud.cfg is migrated correctly from 2.4.3 location in \Users\user\AppData\Local\owncloud to %APPDATA%\owncloud:
owncloud.cfg.gz
owncloud.cfg.backup_20210927_144210_2.8.2 (build 4246).gz
owncloud.cfg.backup_20210927_142315_2.7.6 (build 3261).gz
owncloud.cfg.backup_20210927_131238_2.5.4 (build 11415).gz
owncloud.cfg.backup_20210927_130048.gz

2.4.3 -> 2.5.4

2.5.4 -> 2.7.6

2.7.6 -> 2.8.2

2.8.2 -> 2.9.1daily20210927.5255

gabi18 · 2021-09-28T06:42:58Z

Testing on Windows 10 VFS OFF/ON

2.4.3 -> 2.8.2 -> 2.9.1daily20210927.5255
2.4.3 -> 2.7.6 -> 2.9.1daily20210927.5255

2.4.3 -> 2.8.2 and also 2.4.3 -> 2.7.6
The account isn't configured any longer -> known issue, see #9019 (comment)

\Users\gabi\AppData\Roaming\owncloud\owncloud.cfg

[General]
clientVersion=2.8.2 (build 4246)
logHttp=false
optionalDesktopNotifications=true
showInExplorerNavigationPane=true

\Users\gabi\AppData\Roaming\owncloud\owncloud.cfg.backup_20210928_083432

[General]
logHttp=false

Recreating the account (VFS OFF and ON) and reusing previous syn folder results in having 2 database files:

2.8.2 -> 2.9.1daily20210927.5255
VFS OFF or ON -> same result

gabi18 · 2021-09-28T07:04:38Z

Test on Windows 10 VFS OFF

2.4.3 -> 2.9.0 -> 2.9.1

Result: works correctly old database remains (no new db file)

2.4.3 -> 2.9.0

2.9.0 -> 2.9.1daily20210927.5255
"Enable virtual file support" works correctly

Test on Windows 10 VFS ON

2.9.0 -> 2.9.1daily20210927.5255

Result: works correctly, database file: .sync_journal.db

TheOneRing · 2021-09-28T09:43:33Z

#9019 (comment) is unrelated and covered by https://github.com/owncloud/enterprise/issues/4597 closed in 2.9.0

gabi18 · 2021-09-28T12:13:27Z

To complete update scenarios (Windows 10 VFS ON)

2.8.2 -> 2.9.1daily20210927.5255

Result: works correctly, database file: .sync_xxxx.db

gabi18 · 2021-10-11T12:39:41Z

Retesting with owncloud-client 2.9.1rc2

2.4.3 to 2.9.1rc2 VFS OFF -> okay (database file remains, cfg-file migrated, account is configured, 'Enable virtual file support' works)

Cfg-files
2_4_3-update.zip

2.9.0 to 2.9.1rc2 VFS ON -> okay (database file .sync_journal.db, account is configured / VFS ON)

Cfg-files
2_9_0-update.zip

gabi18 · 2021-10-12T10:21:22Z

Test scenario: Update testpilotcloud 2.8.2 to 2.9.1rc2 VFS ON -> okay

Result: account is configured, database file, folder and (dehydrated) files there

But the update is NOT suggested when switching the Update channel to 'Beta, there is an info that update status was not checked

10-12 11:48:17:570 [ info sync.accessmanager ]: 2 "" "https://updates.owncloud.com/client/?version=2.9.0.5148&platform=win32&oem=testpilotcloud&buildArch=x86_64&currentArch=x86_64&versionsuffix=&channel=beta&msi=true" has X-Request-ID "6d4364e6-6970-4385-88c9-49b60ff067ed"
10-12 11:48:17:570 [ debug sync.cookiejar ]     [ OCC::CookieJar::cookiesForUrl ]:      QUrl("https://updates.owncloud.com/client/?version=2.9.0.5148&platform=win32&oem=testpilotcloud&buildArch=x86_64&currentArch=x86_64&versionsuffix=&channel=beta&msi=true") requests: ()
10-12 11:48:17:570 [ info sync.httplogger ]:    "6d4364e6-6970-4385-88c9-49b60ff067ed: Request: GET https://updates.owncloud.com/client/?version=2.9.0.5148&platform=win32&oem=testpilotcloud&buildArch=x86_64&currentArch=x86_64&versionsuffix=&channel=beta&msi=true Header: { User-Agent: Mozilla/5.0 (Windows) mirall/2.9.0 (build 5148) (testpilotcloud, windows-10.0.19042 ClientArchitecture: x86_64 OsArchitecture: x86_64), Accept: */*, X-Request-ID: 6d4364e6-6970-4385-88c9-49b60ff067ed, Original-Request-ID: 6d4364e6-6970-4385-88c9-49b60ff067ed, } Data: []"
10-12 11:48:17:628 [ info sync.httplogger ]:    "6d4364e6-6970-4385-88c9-49b60ff067ed: Response: GET 0 https://updates.owncloud.com/client/?version=2.9.0.5148&platform=win32&oem=testpilotcloud&buildArch=x86_64&currentArch=x86_64&versionsuffix=&channel=beta&msi=true Header: { } Data: []"
10-12 11:48:17:628 [ warning gui.updater ]:     Failed to reach version check url:  "SSL handshake failed"

gabi18 · 2021-10-12T14:11:29Z

Problem was because of outdated certificates. After a system update of the Win 10 20H2 VM the update check is successful:

Vogtinator mentioned this issue Sep 14, 2021

Multiple accounts are sharing the folder error #9002

Closed

TheOneRing added this to the 2.9.1 milestone Sep 15, 2021

michaelstingl added the p2-high Escalation, on top of current planning, release blocker label Sep 15, 2021

TheOneRing added a commit that referenced this issue Sep 15, 2021

Migrate existing db files and save their journal path

2487529

Fixes: #9019

TheOneRing mentioned this issue Sep 15, 2021

Migrate existing db files and save their journal path #9028

Merged

TheOneRing linked a pull request Sep 15, 2021 that will close this issue

Migrate existing db files and save their journal path #9028

Merged

michaelstingl mentioned this issue Sep 16, 2021

Error message about multiple accounts #9039

Closed

TheOneRing closed this as completed in 9c4058a Sep 17, 2021

TheOneRing reopened this Sep 17, 2021

TheOneRing linked a pull request Sep 17, 2021 that will close this issue

Also migrate pre 2.6 db files #9046

Merged

TheOneRing linked a pull request Sep 17, 2021 that will close this issue

Fix migration on minor url format changes like… #9054

Merged

TheOneRing closed this as completed Sep 21, 2021

TheOneRing mentioned this issue Sep 21, 2021

QtKeyhcain: DeletePasswordJob with KWallet does not delete, but creates an empty entry instead #9064

Closed

gabi18 mentioned this issue Sep 27, 2021

Release Desktop 2.9.1 #9079

Closed

46 tasks

gabi18 reopened this Sep 28, 2021

TheOneRing closed this as completed Sep 28, 2021

Migration from 2.8.1 to 2.9.0 fails silently #9019

Migration from 2.8.1 to 2.9.0 fails silently #9019

Comments

Vogtinator commented Sep 14, 2021

Server configuration

Client configuration

Logs

TheOneRing commented Sep 14, 2021

Vogtinator commented Sep 14, 2021

TheOneRing commented Sep 14, 2021

Vogtinator commented Sep 14, 2021

TheOneRing commented Sep 14, 2021

Vogtinator commented Sep 14, 2021

TheOneRing commented Sep 15, 2021

TheOneRing commented Sep 15, 2021

Vogtinator commented Sep 15, 2021

Vogtinator commented Sep 15, 2021

TheOneRing commented Sep 15, 2021

TheOneRing commented Sep 15, 2021

Vogtinator commented Sep 15, 2021

2.4 -> 2.8 -> 2.9.1

2.4 -> 2.9.0 -> 2.9.1

TheOneRing commented Sep 15, 2021

2.4 -> 2.8 -> 2.9.1

2.4 -> 2.9.0 -> 2.9.1

Vogtinator commented Sep 15, 2021 • edited Loading

TheOneRing commented Sep 15, 2021

silvain commented Sep 16, 2021

Why did this happen?

Vogtinator commented Sep 17, 2021

TheOneRing commented Sep 17, 2021

TheOneRing commented Sep 17, 2021

Vogtinator commented Sep 17, 2021

TheOneRing commented Sep 17, 2021

Vogtinator commented Sep 21, 2021

TheOneRing commented Sep 21, 2021

Vogtinator commented Sep 21, 2021

TheOneRing commented Sep 21, 2021

Vogtinator commented Sep 21, 2021

TheOneRing commented Sep 21, 2021

gabi18 commented Sep 27, 2021 • edited Loading

gabi18 commented Sep 28, 2021 • edited Loading

gabi18 commented Sep 28, 2021 • edited Loading

TheOneRing commented Sep 28, 2021

gabi18 commented Sep 28, 2021

gabi18 commented Oct 11, 2021 • edited Loading

gabi18 commented Oct 12, 2021 • edited Loading

gabi18 commented Oct 12, 2021

Vogtinator commented Sep 15, 2021 •

edited

Loading

gabi18 commented Sep 27, 2021 •

edited

Loading

gabi18 commented Sep 28, 2021 •

edited

Loading

gabi18 commented Sep 28, 2021 •

edited

Loading

gabi18 commented Oct 11, 2021 •

edited

Loading

gabi18 commented Oct 12, 2021 •

edited

Loading