Problem with upgrading files_encryption from OC 7 => OC 8 #14012

Closed
Beskhue opened this Issue Feb 9, 2015 · 45 comments

Comments

Projects
None yet
@Beskhue

Beskhue commented Feb 9, 2015

Steps to reproduce

  1. Have a working installation of owncloud version 7.0.4.2 with files_encryption enabled.
  2. Attempt to upgrade to owncloud 8.0.0.7.

Expected behaviour

The upgrade should succeed.

Actual behaviour

The upgrade hangs. After disabling maintenance mode and trying again, it still hangs.

After disabling files_encryption through the occ console, the upgrade succeeded. However, encrypted files are not decrypted (and owncloud does not prompt users to decrypt their files). After re-enabling files_encryption, owncloud shows the upgrade-screen again. Upgrading still fails.

Server configuration

Operating system: Ubuntu 12.04

Web server: Apache 2.4.9

Database: MySQL

PHP version: 5.5.12

ownCloud version: 7.0.4.2=>8.0.0.7

Updated from an older ownCloud or fresh install: During upgrading

List of activated apps:

  • files
  • files_encryption
  • files_external
  • files_pdfviewer
  • files_sharing
  • files_texteditor
  • files_trashbin
  • files_versions
  • files_videoviewer
  • firstrunwizard
  • gallery
  • templateeditor
  • updater

The content of config/config.php:

<?php
$CONFIG = array (
  'instanceid' => 'oc2197ab4a5f',
  'trusted_domains' =>
  array (
    0 => 'kepow.org',
  ),
  'datadirectory' => '/www/kepow/owncloud_data',
  'dbtype' => 'mysql',
  'version' => '8.0.0.7',
  'dbname' => 'owncloud',
  'dbhost' => 'localhost',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'oc_Thomas',
  'installed' => true,
  'forcessl' => true,
  'mail_smtpmode' => 'php',
  'mail_smtpname' => 'thomas',
  'maintenance' => true,
  'theme' => '',
);

Are you using external storage, if yes which one: No.

Are you using encryption: Yes.

Client configuration

Browser: Issue occurs on firefox and chrome

Operating system: Windows

Logs

Web server error log

No errors are logged during the upgrade process.

ownCloud log (data/owncloud.log)

First try:

{"reqId":"230bd67050902ddeb36c9af31787d650","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#282","level":3,"time":"2015-02-09T20:51:06+00:00"}
{"reqId":"230bd67050902ddeb36c9af31787d650","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#283","level":3,"time":"2015-02-09T20:51:06+00:00"}
{"reqId":"230bd67050902ddeb36c9af31787d650","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#284","level":3,"time":"2015-02-09T20:51:06+00:00"}
{"reqId":"230bd67050902ddeb36c9af31787d650","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#287","level":3,"time":"2015-02-09T20:51:06+00:00"}
{"reqId":"230bd67050902ddeb36c9af31787d650","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#288","level":3,"time":"2015-02-09T20:51:06+00:00"}
{"reqId":"230bd67050902ddeb36c9af31787d650","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#289","level":3,"time":"2015-02-09T20:51:06+00:00"}
{"reqId":"230bd67050902ddeb36c9af31787d650","remoteAddr":"84.85.215.83","app":"PHP","message":"opendir(\/www\/kepow\/owncloud_data\/files_encryption\/keyfiles): failed to open dir: No such file or directory at \/www\/kepow\/owncloud\/lib\/private\/files\/storage\/local.php#78","level":3,"time":"2015-02-09T20:51:06+00:00"}
{"reqId":"230bd67050902ddeb36c9af31787d650","remoteAddr":"84.85.215.83","app":"core","message":"unable to rename, file does not exists : files_encryption\/Jordi.private.key","level":3,"time":"2015-02-09T20:53:38+00:00"}

Retry:

{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#282","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#283","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#284","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#287","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#288","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#289","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"copy(\/www\/kepow\/owncloud_data\/owncloud_private_key): failed to open stream: No such file or directory at \/www\/kepow\/owncloud\/lib\/private\/files\/storage\/local.php#211","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"copy(\/www\/kepow\/owncloud_data\/public-keys): failed to open stream: No such file or directory at \/www\/kepow\/owncloud\/lib\/private\/files\/storage\/local.php#211","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"opendir(\/www\/kepow\/owncloud_data\/public-keys): failed to open dir: No such file or directory at \/www\/kepow\/owncloud\/lib\/private\/files\/storage\/local.php#78","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"opendir(\/www\/kepow\/owncloud_data\/files_encryption\/keyfiles): failed to open dir: No such file or directory at \/www\/kepow\/owncloud\/lib\/private\/files\/storage\/local.php#78","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"opendir(\/www\/kepow\/owncloud_data\/owncloud_private_key): failed to open dir: No such file or directory at \/www\/kepow\/owncloud\/lib\/private\/files\/storage\/local.php#78","level":3,"time":"2015-02-09T20:55:39+00:00"}

Browser log

Using //@ to indicate sourceMappingURL pragmas is deprecated. Use //# instead jquery.min.js:1
Using //@ to indicate sourceMappingURL pragmas is deprecated. Use //# instead jquery-migrate.min.js:3
Content Security Policy: The page's settings blocked the loading of a resource at self ("script-src https://kepow.org 'unsafe-eval'"). index.php
Use of getPreventDefault() is deprecated.  Use defaultPrevented instead. jquery.min.js:5
@MorrisJobke

This comment has been minimized.

Show comment
Hide comment
@MorrisJobke

MorrisJobke Feb 9, 2015

Member

cc @schiesbn

Member

MorrisJobke commented Feb 9, 2015

cc @schiesbn

@DeepDiver1975 DeepDiver1975 added this to the 8.0.1-next-maintenance milestone Feb 9, 2015

@doits

This comment has been minimized.

Show comment
Hide comment
@doits

doits Feb 9, 2015

I had the same "problem" with the hang but eventually the upgrade did not hang but instead was simply taking a really long time (> 30 minutes). strace showed the update process was opening each encrypted file (that's why I know it was doing something and therefore I did not abort it) and it was doing it at about 5 files per second ... so if you have a lot of number of files it can take some time. Maybe that's the cause here?

Edit: The file the upgrade opened were all located in the subfolder data/username/files_encryption/ and ended with .key or fileKey or .shareKey - so I assume it encryption had to be updated.

doits commented Feb 9, 2015

I had the same "problem" with the hang but eventually the upgrade did not hang but instead was simply taking a really long time (> 30 minutes). strace showed the update process was opening each encrypted file (that's why I know it was doing something and therefore I did not abort it) and it was doing it at about 5 files per second ... so if you have a lot of number of files it can take some time. Maybe that's the cause here?

Edit: The file the upgrade opened were all located in the subfolder data/username/files_encryption/ and ended with .key or fileKey or .shareKey - so I assume it encryption had to be updated.

@Beskhue

This comment has been minimized.

Show comment
Hide comment
@Beskhue

Beskhue Feb 9, 2015

It's certainly a possibility. I've had the upgrader running for a while (~1 hour), and I don't have a large number of files per se. I'm currently running it again and am logging the system calls using strace. It's still going strong (and quick), but I am seeing some errors, such as:

[pid 27995] 23:52:12.429875 lstat("/www/kepow/owncloud_data/Jordi/encryption_migration_backup_2015-02-09_22-50-58/share-keys/***/***/***/***.Jordi.shareKey", 0x7fff33790390) = -1 ENOENT (No such file or directory)

I also noticed that a problem with the private key associated with the user Jordi was reported in the errors logged in owncloud.log for the first upgrade attempt.

Beskhue commented Feb 9, 2015

It's certainly a possibility. I've had the upgrader running for a while (~1 hour), and I don't have a large number of files per se. I'm currently running it again and am logging the system calls using strace. It's still going strong (and quick), but I am seeing some errors, such as:

[pid 27995] 23:52:12.429875 lstat("/www/kepow/owncloud_data/Jordi/encryption_migration_backup_2015-02-09_22-50-58/share-keys/***/***/***/***.Jordi.shareKey", 0x7fff33790390) = -1 ENOENT (No such file or directory)

I also noticed that a problem with the private key associated with the user Jordi was reported in the errors logged in owncloud.log for the first upgrade attempt.

@schiessle

This comment has been minimized.

Show comment
Hide comment
@schiessle

schiessle Feb 9, 2015

Member

The upgrade will rename all encryption keys. If you have a lot of files this can take some time. We needed to re-structure the way the keys are stored to avoid file name collisions for some rare username-filename combinations. Before the upgrade script starts to rename the encryption keys it creates a backup of all your keys. So nothing should be lost. If you know how the key where stored in OC7 you can restore the backup and try to run the migration again with the occ script.

If you are not sure how to restore the backup I can give you more details tomorrow.

Member

schiessle commented Feb 9, 2015

The upgrade will rename all encryption keys. If you have a lot of files this can take some time. We needed to re-structure the way the keys are stored to avoid file name collisions for some rare username-filename combinations. Before the upgrade script starts to rename the encryption keys it creates a backup of all your keys. So nothing should be lost. If you know how the key where stored in OC7 you can restore the backup and try to run the migration again with the occ script.

If you are not sure how to restore the backup I can give you more details tomorrow.

@Beskhue

This comment has been minimized.

Show comment
Hide comment
@Beskhue

Beskhue Feb 9, 2015

I am not entirely sure how to restore the backup. Because of several upgrade attempts I now have a number of folders with names such as encryption_migration_backup_2015-02-09_16-07-16. I believe the folders from the first attempt contain everything I need.

Should I just copy the contents of all these folders from the first attempt back to their parent directories (i.e., cp -a ./ ../) and then remove the backup folders?

Beskhue commented Feb 9, 2015

I am not entirely sure how to restore the backup. Because of several upgrade attempts I now have a number of folders with names such as encryption_migration_backup_2015-02-09_16-07-16. I believe the folders from the first attempt contain everything I need.

Should I just copy the contents of all these folders from the first attempt back to their parent directories (i.e., cp -a ./ ../) and then remove the backup folders?

@kkretsch

This comment has been minimized.

Show comment
Hide comment
@kkretsch

kkretsch Feb 10, 2015

Having a similar problem I switched back to 704. I have nearly 10GB of encrypted media which seems to upgrade for such a long time OC9 might even be released befaore it finishes.

Any hints how to upgrade in a reasonable timeframe?

Having a similar problem I switched back to 704. I have nearly 10GB of encrypted media which seems to upgrade for such a long time OC9 might even be released befaore it finishes.

Any hints how to upgrade in a reasonable timeframe?

@schiessle

This comment has been minimized.

Show comment
Hide comment
@schiessle

schiessle Feb 10, 2015

Member

@Beskhue yes, take the backup from the first attempt because this should contain the state from OC7. Every user should have such a backup folder. Just that we don't lose any state I would make a manual backup from each users files_encryption folder in the current state, then delete the content of the files_encryption folder. Now copy the content of encryption_migration_backup_<date> from the first attempt for each users to files_encryption folder.

Also in data/ you should have a system wide 'files_encryption' folder for which also backups with the pattern encryption_migration_backup_<date> should exists. Like for the users first backup the existing files_encryption folder and then copy the backup to the system wide files_encryption folder.

Afterwards try to run the migration again ./occ encryption:migrate-keys. I would also suggest to stop your web server before you perform this steps, just to make sure that no update gets triggered somewhere in between or that any sync client tries to read some files etc.

Member

schiessle commented Feb 10, 2015

@Beskhue yes, take the backup from the first attempt because this should contain the state from OC7. Every user should have such a backup folder. Just that we don't lose any state I would make a manual backup from each users files_encryption folder in the current state, then delete the content of the files_encryption folder. Now copy the content of encryption_migration_backup_<date> from the first attempt for each users to files_encryption folder.

Also in data/ you should have a system wide 'files_encryption' folder for which also backups with the pattern encryption_migration_backup_<date> should exists. Like for the users first backup the existing files_encryption folder and then copy the backup to the system wide files_encryption folder.

Afterwards try to run the migration again ./occ encryption:migrate-keys. I would also suggest to stop your web server before you perform this steps, just to make sure that no update gets triggered somewhere in between or that any sync client tries to read some files etc.

@schiessle

This comment has been minimized.

Show comment
Hide comment
@schiessle

schiessle Feb 10, 2015

Member

@kkretsch we need to rename all encryption keys, there is no way arround it. Depending on how many files you have and how often they are shared you have more or less keys we need to migrate. But the keys are quite small files, we don't touch the actual files. So the crucial factor is the number if files and how often they are shared, not the size of the files. Renaming the keys should be possible in a reasonable time, depending on you hard disc performance. I would suggest to shutdown your web server and trigger the update with the occ command line tool to avoid any timeouts.

Member

schiessle commented Feb 10, 2015

@kkretsch we need to rename all encryption keys, there is no way arround it. Depending on how many files you have and how often they are shared you have more or less keys we need to migrate. But the keys are quite small files, we don't touch the actual files. So the crucial factor is the number if files and how often they are shared, not the size of the files. Renaming the keys should be possible in a reasonable time, depending on you hard disc performance. I would suggest to shutdown your web server and trigger the update with the occ command line tool to avoid any timeouts.

@inotriel

This comment has been minimized.

Show comment
Hide comment
@inotriel

inotriel Feb 10, 2015

Maybe this is the "problem" I mentioned here #13992 (comment)

I have about 7 oder 8 thousand files in my oc and the harddisk might not be the fastest in the world. So waiting for about 30 minutes maybe was just to little time.

Is there any chance to see if occ is still working or stuck?

Maybe this is the "problem" I mentioned here #13992 (comment)

I have about 7 oder 8 thousand files in my oc and the harddisk might not be the fastest in the world. So waiting for about 30 minutes maybe was just to little time.

Is there any chance to see if occ is still working or stuck?

@schiessle

This comment has been minimized.

Show comment
Hide comment
@schiessle

schiessle Feb 10, 2015

Member

You could monitor the files_encryption folder of your users the size of some of the folders should change from time to time if it still runs. Also the backup folders should appear over time for the users.

Member

schiessle commented Feb 10, 2015

You could monitor the files_encryption folder of your users the size of some of the folders should change from time to time if it still runs. Also the backup folders should appear over time for the users.

@inotriel

This comment has been minimized.

Show comment
Hide comment
@inotriel

inotriel Feb 10, 2015

Good idea, thank you. I'll give it another try tonight!

Good idea, thank you. I'll give it another try tonight!

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Feb 10, 2015

I'm running the occ upgrade cmd now for 3h !!! I have something about 5GB total. I didn't have such an issue before with the encryption enabled. The process runs with about 3 to 8% CPU usage and I would like to know what he's doing.... oh wait. It just finished ... lol.

I think it's definately a good idea to shut down the apache server. That's what I just did and the process finished right after... Could that be correlated?

ghost commented Feb 10, 2015

I'm running the occ upgrade cmd now for 3h !!! I have something about 5GB total. I didn't have such an issue before with the encryption enabled. The process runs with about 3 to 8% CPU usage and I would like to know what he's doing.... oh wait. It just finished ... lol.

I think it's definately a good idea to shut down the apache server. That's what I just did and the process finished right after... Could that be correlated?

@kkretsch

This comment has been minimized.

Show comment
Hide comment
@kkretsch

kkretsch Feb 10, 2015

After removing two old entries in the Database that "local::/..." stuff with old path not existing any more I reran it on a clean version 7 restore using the command line and after under an hour it went thru without any error. The machine was using it2 100% cpu for that task, but there where still some spare cores left :-)

After removing two old entries in the Database that "local::/..." stuff with old path not existing any more I reran it on a clean version 7 restore using the command line and after under an hour it went thru without any error. The machine was using it2 100% cpu for that task, but there where still some spare cores left :-)

@khlschrnk

This comment has been minimized.

Show comment
Hide comment
@khlschrnk

khlschrnk Feb 10, 2015

If have about 135k (not including the files_encrytion folder) files that use about 75GB. I started the ./occ upgrade 12 hours ago and it is still running on an idle pc.
As you can see in the attachment (output of iotop) that the process is still running.
I am not sure if it is unusual to have the highest IO in [jbd2/md2-8]. Maybe my software raid is causing issues?
My CPU load is around 1-2% for ./occ.

As a rough calculation I expect the renaming process per file to be way faster than 50ms. With respect to the number of files it should not exceed 2 hours total. So in my opinion there is something broken.

I don't find any other usefull information in my logs.

occ upgrade

If have about 135k (not including the files_encrytion folder) files that use about 75GB. I started the ./occ upgrade 12 hours ago and it is still running on an idle pc.
As you can see in the attachment (output of iotop) that the process is still running.
I am not sure if it is unusual to have the highest IO in [jbd2/md2-8]. Maybe my software raid is causing issues?
My CPU load is around 1-2% for ./occ.

As a rough calculation I expect the renaming process per file to be way faster than 50ms. With respect to the number of files it should not exceed 2 hours total. So in my opinion there is something broken.

I don't find any other usefull information in my logs.

occ upgrade

@doits

This comment has been minimized.

Show comment
Hide comment
@doits

doits Feb 10, 2015

If you want to see what it is doing (which files operating and how fast), use strace:

strace -e open -p PID

# for example (not tested if this really gets the pid but I think it should)
strace -e open -p `ps ax | grep occ | grep -v grep | sed 's/ .*//' | head -n 1`

For me it was about 3 file per second only.

doits commented Feb 10, 2015

If you want to see what it is doing (which files operating and how fast), use strace:

strace -e open -p PID

# for example (not tested if this really gets the pid but I think it should)
strace -e open -p `ps ax | grep occ | grep -v grep | sed 's/ .*//' | head -n 1`

For me it was about 3 file per second only.

@jknockaert

This comment has been minimized.

Show comment
Hide comment
@jknockaert

jknockaert Feb 11, 2015

Contributor

I have a very similar issue. However, the key migration stopped after two or three minutes. After restoring the key backup as explained by @schiesbn, I tried to restart the migration using ./occ encryption:migrate-keys, but that gives a InvalidArgumentException: "There are no commands defined in the "encryption" namespace." Not sure how to proceed now.

Contributor

jknockaert commented Feb 11, 2015

I have a very similar issue. However, the key migration stopped after two or three minutes. After restoring the key backup as explained by @schiesbn, I tried to restart the migration using ./occ encryption:migrate-keys, but that gives a InvalidArgumentException: "There are no commands defined in the "encryption" namespace." Not sure how to proceed now.

@khlschrnk

This comment has been minimized.

Show comment
Hide comment
@khlschrnk

khlschrnk Feb 11, 2015

Finished. It took me 22 hours to complete my 135k files upgrade.

Sync clients are syncing again on all platforms but my webinterface stays blank. No idea why but i guess this is a seperate issue ...

Finished. It took me 22 hours to complete my 135k files upgrade.

Sync clients are syncing again on all platforms but my webinterface stays blank. No idea why but i guess this is a seperate issue ...

@inotriel

This comment has been minimized.

Show comment
Hide comment
@inotriel

inotriel Feb 11, 2015

OMG. So if that is a point of reference I should wait some hours. We are using oc in a productive environment so ... well, it will be a little bit complicated.

OMG. So if that is a point of reference I should wait some hours. We are using oc in a productive environment so ... well, it will be a little bit complicated.

@Robstaaaa

This comment has been minimized.

Show comment
Hide comment
@Robstaaaa

Robstaaaa Feb 11, 2015

I ran into a similar issue:

My library is about 60 GB including lots of small files, so I expect the process to take quite long. However, the update process stops with an fatal error after short time (1h).

  • Apache is stopped
  • Command is: sudo -u www-data php /var/www/owncloud/occ upgrade

Errors were slightly different for consecutive executions of the same command:

  • 1st try: PHP Fatal error: Maximum execution time of 3600 seconds exceeded in /var/www/owncloud/lib/private/files/filesystem.php on line 516
  • 2nd try: PHP Fatal error: Maximum execution time of 3600 seconds exceeded in /var/www/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php on line 104
  • 3rd try: PHP Fatal error: Maximum execution time of 3600 seconds exceeded in /var/www/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php on line 91

Any advice?

[EDIT] I could not find this limitation of 3600 seconds for my php configuration. In php.ini the default value for max_execution_time is 30 seconds, in the owncloud directories I could not find anything that overwrites the default.

I ran into a similar issue:

My library is about 60 GB including lots of small files, so I expect the process to take quite long. However, the update process stops with an fatal error after short time (1h).

  • Apache is stopped
  • Command is: sudo -u www-data php /var/www/owncloud/occ upgrade

Errors were slightly different for consecutive executions of the same command:

  • 1st try: PHP Fatal error: Maximum execution time of 3600 seconds exceeded in /var/www/owncloud/lib/private/files/filesystem.php on line 516
  • 2nd try: PHP Fatal error: Maximum execution time of 3600 seconds exceeded in /var/www/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php on line 104
  • 3rd try: PHP Fatal error: Maximum execution time of 3600 seconds exceeded in /var/www/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php on line 91

Any advice?

[EDIT] I could not find this limitation of 3600 seconds for my php configuration. In php.ini the default value for max_execution_time is 30 seconds, in the owncloud directories I could not find anything that overwrites the default.

@jknockaert

This comment has been minimized.

Show comment
Hide comment
@jknockaert

jknockaert Feb 11, 2015

Contributor

Running ./occ upgrade does complete successfully (after 2 hours). However, after doing so I get the message "Encryption App is enabled but your keys are not initialized, please log-out and log-in again". In the log it says "Session does not contain a private key, maybe your login password changed" and "private key exists but public key is missing". I definitely did not change any password. Could something have gone wrong because the database schema update was run again on an already updated database?

Contributor

jknockaert commented Feb 11, 2015

Running ./occ upgrade does complete successfully (after 2 hours). However, after doing so I get the message "Encryption App is enabled but your keys are not initialized, please log-out and log-in again". In the log it says "Session does not contain a private key, maybe your login password changed" and "private key exists but public key is missing". I definitely did not change any password. Could something have gone wrong because the database schema update was run again on an already updated database?

@peperunas

This comment has been minimized.

Show comment
Hide comment

+1

@tflidd

This comment has been minimized.

Show comment
Hide comment
@tflidd

tflidd Feb 11, 2015

Contributor

What is your recommendation for users on shared hosting solutions? They often won't have ssh-access and no execution times of several hours. Wait for OC 8.0.1?

Contributor

tflidd commented Feb 11, 2015

What is your recommendation for users on shared hosting solutions? They often won't have ssh-access and no execution times of several hours. Wait for OC 8.0.1?

@karlitschek

This comment has been minimized.

Show comment
Hide comment
@karlitschek

karlitschek Feb 11, 2015

Member

@schiesbn

Member

karlitschek commented Feb 11, 2015

@schiesbn

@Robstaaaa

This comment has been minimized.

Show comment
Hide comment
@Robstaaaa

Robstaaaa Feb 11, 2015

My solution if you run into the 3600 second timeouts:

Go to the file: "/var/www/owncloud/lib/base.php" and change the 3600 second timeout settings to 0.

After that, the upgrade went through without any errors and all encrypted files are available again in the updated owncloud version without further issues. For me it took about 2 hours.

The real pain is that parameters for php can be configured in various places which is irritating. Potential places which I checked first were the following, where I found settings which eventually were overwritten:

  • /etc/php
  • /var/www/owncloud/.htaccess
  • /var/www/owncloud/config/config.php

My solution if you run into the 3600 second timeouts:

Go to the file: "/var/www/owncloud/lib/base.php" and change the 3600 second timeout settings to 0.

After that, the upgrade went through without any errors and all encrypted files are available again in the updated owncloud version without further issues. For me it took about 2 hours.

The real pain is that parameters for php can be configured in various places which is irritating. Potential places which I checked first were the following, where I found settings which eventually were overwritten:

  • /etc/php
  • /var/www/owncloud/.htaccess
  • /var/www/owncloud/config/config.php
@jknockaert

This comment has been minimized.

Show comment
Hide comment
@jknockaert

jknockaert Feb 12, 2015

Contributor

I finally managed to finish the upgrade. I misunderstood the explanation by @schiesbn: there should be no system wide files_encryption folder in /data; rather should the back up of system-wide folders (like public keys) be placed directly in /data. And ./occ encryption:migrate-keys is only available after a previous upgrade to 8 was completed, otherwise run ./occ upgrade. This was definitely one of the rougher owncloud upgrades, taking two hours on a small scale server. I doubt there's any reasonable application of this upgrade for large scale servers.

Contributor

jknockaert commented Feb 12, 2015

I finally managed to finish the upgrade. I misunderstood the explanation by @schiesbn: there should be no system wide files_encryption folder in /data; rather should the back up of system-wide folders (like public keys) be placed directly in /data. And ./occ encryption:migrate-keys is only available after a previous upgrade to 8 was completed, otherwise run ./occ upgrade. This was definitely one of the rougher owncloud upgrades, taking two hours on a small scale server. I doubt there's any reasonable application of this upgrade for large scale servers.

@armaccloud

This comment has been minimized.

Show comment
Hide comment
@armaccloud

armaccloud Feb 12, 2015

Hi @jknockaert,
Thanks for sharing some info about your steps; how did you fix the "Encryption App is enabled but your keys are not initialized, please log-out and log-in again" issue?

In my environment there is a files_encryption folder in /data; I don't understand what you mean with "there should be no system wide files_encryption folder in /data; rather should the back up of system-wide folders (like public keys) be placed directly in /data." What does that structure look like?

On my system, I had to run the occ encryption:migrate-keys option. After a 2,5h run, I get the same error message in ownCloud about the reinitialization, however when I log in again, it shows the message again and no files can be opened.

Thanks.

Hi @jknockaert,
Thanks for sharing some info about your steps; how did you fix the "Encryption App is enabled but your keys are not initialized, please log-out and log-in again" issue?

In my environment there is a files_encryption folder in /data; I don't understand what you mean with "there should be no system wide files_encryption folder in /data; rather should the back up of system-wide folders (like public keys) be placed directly in /data." What does that structure look like?

On my system, I had to run the occ encryption:migrate-keys option. After a 2,5h run, I get the same error message in ownCloud about the reinitialization, however when I log in again, it shows the message again and no files can be opened.

Thanks.

@jknockaert

This comment has been minimized.

Show comment
Hide comment
@jknockaert

jknockaert Feb 13, 2015

Contributor

@armaccloud Before running ./occ encryption:migrate-keys again, restore the pre-upgrade folder layout as pointed out by @schiesbn; this means that the content of /data/<user>/encryption_migration_backup_<date> should go to /data/<user>/files_encryption; however, the content of /data/encryption_migration_backup_<date> should go to /data (not /data/files_encryption). If you have multiple values for <date>, use the oldest one (i.e. the first one created upon the initial upgrade to 8). Note that there is a /data/encryption_migration_backup_<date>/files_encryption, but it should be empty, as in pre-OC8 installations there was no use for /data/files_encryption.

Contributor

jknockaert commented Feb 13, 2015

@armaccloud Before running ./occ encryption:migrate-keys again, restore the pre-upgrade folder layout as pointed out by @schiesbn; this means that the content of /data/<user>/encryption_migration_backup_<date> should go to /data/<user>/files_encryption; however, the content of /data/encryption_migration_backup_<date> should go to /data (not /data/files_encryption). If you have multiple values for <date>, use the oldest one (i.e. the first one created upon the initial upgrade to 8). Note that there is a /data/encryption_migration_backup_<date>/files_encryption, but it should be empty, as in pre-OC8 installations there was no use for /data/files_encryption.

@armaccloud

This comment has been minimized.

Show comment
Hide comment
@armaccloud

armaccloud Feb 13, 2015

Many thanks to you both; I have managed to run the script. However it only performed the action on some files, the majority was not executed. Only the following couple of folders have been created and they are not complete by far;

keys

The following errors are appearing in the logs;

{"reqId":"54248f7778847","app":"OC\\Files\\Cache\\Scanner","message":"!!! Path 'files\/Photos\/2007\/Oud en Nieuw 06-07\/01012007175.mp4' is not readable !!!","level":0,"time":"2014-09-25T21:56:07+00:00","method":"PUT","url":"\/owncloud\/remote.php\/webdav\/Photos\/2007\/Oud%20en%20Nieuw%2006-07\/01012007175.mp4"} {"reqId":"54248f7e98453","app":"OC\\Files\\Cache\\Scanner","message":"!!! Path 'files\/Photos\/2007\/Oud en Nieuw 06-07\/311220063585.jpg' is not readable !!!","level":0,"time":"2014-09-25T21:56:14+00:00","method":"PUT","url":"\/owncloud\/remohte.php\/webdav\/Photos\/2007\/Oud%20en%20Nieuw%2006-07\/311220063585.jpg"}

This might have to do with the folder path, which in the above example is the following:
Photo's and Video's\2007\Oud en Nieuw\... Could it have an issue with the ' in the folder names?

Also, the folder Photo's and Video's used to be called Photo's & Video's (was renamed a long time before I did the upgrade) and the keyfiles folder seems to contain both foders (see screenshot). That might also be creating issues.

keyfiles

Makes it strange though, why e.g. the folders Study and Work are also not been converted.
Any ideas?

Many thanks to you both; I have managed to run the script. However it only performed the action on some files, the majority was not executed. Only the following couple of folders have been created and they are not complete by far;

keys

The following errors are appearing in the logs;

{"reqId":"54248f7778847","app":"OC\\Files\\Cache\\Scanner","message":"!!! Path 'files\/Photos\/2007\/Oud en Nieuw 06-07\/01012007175.mp4' is not readable !!!","level":0,"time":"2014-09-25T21:56:07+00:00","method":"PUT","url":"\/owncloud\/remote.php\/webdav\/Photos\/2007\/Oud%20en%20Nieuw%2006-07\/01012007175.mp4"} {"reqId":"54248f7e98453","app":"OC\\Files\\Cache\\Scanner","message":"!!! Path 'files\/Photos\/2007\/Oud en Nieuw 06-07\/311220063585.jpg' is not readable !!!","level":0,"time":"2014-09-25T21:56:14+00:00","method":"PUT","url":"\/owncloud\/remohte.php\/webdav\/Photos\/2007\/Oud%20en%20Nieuw%2006-07\/311220063585.jpg"}

This might have to do with the folder path, which in the above example is the following:
Photo's and Video's\2007\Oud en Nieuw\... Could it have an issue with the ' in the folder names?

Also, the folder Photo's and Video's used to be called Photo's & Video's (was renamed a long time before I did the upgrade) and the keyfiles folder seems to contain both foders (see screenshot). That might also be creating issues.

keyfiles

Makes it strange though, why e.g. the folders Study and Work are also not been converted.
Any ideas?

@Beskhue

This comment has been minimized.

Show comment
Hide comment
@Beskhue

Beskhue Feb 14, 2015

I ran the upgrade process again (thank Zeus for screen) and it finished successfully. It took 10+ hours.

Thanks for your help, everyone!

Beskhue commented Feb 14, 2015

I ran the upgrade process again (thank Zeus for screen) and it finished successfully. It took 10+ hours.

Thanks for your help, everyone!

@ghost ghost referenced this issue Feb 17, 2015

Closed

Upgrade taking a long time #14279

@lupa18

This comment has been minimized.

Show comment
Hide comment
@lupa18

lupa18 Feb 18, 2015

I have the same problem. I upgraded by running:

sudo apt-get upgrade

Env:

  • OS: Ubuntu 14.04.1
  • Kernel: 3.2.0-67-generic

and now?

lupa18 commented Feb 18, 2015

I have the same problem. I upgraded by running:

sudo apt-get upgrade

Env:

  • OS: Ubuntu 14.04.1
  • Kernel: 3.2.0-67-generic

and now?

@eminaksehirli

This comment has been minimized.

Show comment
Hide comment
@eminaksehirli

eminaksehirli Feb 23, 2015

Hello, I've managed to complete the upgrade. I'm using owncloud on a shared hosting and I had to mirror the installation on my local machine, do the upgrade, and redeploy. The update script ran around 12 hours and used more than 7 gigs of memory. With these requirements, I think there should be a warning on the upgrade page. I documented my process here: http://memin.tk/informatics/2015/02/22/Upgrading-to-Owncloud-8.html

Hello, I've managed to complete the upgrade. I'm using owncloud on a shared hosting and I had to mirror the installation on my local machine, do the upgrade, and redeploy. The update script ran around 12 hours and used more than 7 gigs of memory. With these requirements, I think there should be a warning on the upgrade page. I documented my process here: http://memin.tk/informatics/2015/02/22/Upgrading-to-Owncloud-8.html

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Feb 27, 2015

Member

@icewind1991 can you please have a look at the encryption migration - https://github.com/owncloud/core/blob/master/apps/files_encryption/lib/migration.php ?

The migration is using files view which is know to be performance intensive compared to using the storage directly - might this help here as well?

The high memory consumption is a bit frightening as well 🙊

Member

DeepDiver1975 commented Feb 27, 2015

@icewind1991 can you please have a look at the encryption migration - https://github.com/owncloud/core/blob/master/apps/files_encryption/lib/migration.php ?

The migration is using files view which is know to be performance intensive compared to using the storage directly - might this help here as well?

The high memory consumption is a bit frightening as well 🙊

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Feb 27, 2015

Member

I had the same "problem" with the hang but eventually the upgrade did not hang but instead was simply taking a really long time (> 30 minutes). strace showed the update process was opening each encrypted file (that's why I know it was doing something and therefore I did not abort it) and it was doing it at about 5 files per second ... so if you have a lot of number of files it can take some time. Maybe that's the cause here?

5 files a second ?! How can it be that slow ?

Member

PVince81 commented Feb 27, 2015

I had the same "problem" with the hang but eventually the upgrade did not hang but instead was simply taking a really long time (> 30 minutes). strace showed the update process was opening each encrypted file (that's why I know it was doing something and therefore I did not abort it) and it was doing it at about 5 files per second ... so if you have a lot of number of files it can take some time. Maybe that's the cause here?

5 files a second ?! How can it be that slow ?

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Feb 27, 2015

Member

If I remember well, some keys are cached within a single PHP call to speed up the upgrade and some file operations. If run during the upgrade, I wonder if that cache's size is increasing when there are many files. That cache should probably be adapted to have invalidation logic.

Member

PVince81 commented Feb 27, 2015

If I remember well, some keys are cached within a single PHP call to speed up the upgrade and some file operations. If run during the upgrade, I wonder if that cache's size is increasing when there are many files. That cache should probably be adapted to have invalidation logic.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Feb 27, 2015

Member

I tried an upgrade with 1600 files, here is the result: https://blackfire.io/profiles/b07deb06-7a94-4e67-9479-a9e9798f2867/graph

Member

PVince81 commented Feb 27, 2015

I tried an upgrade with 1600 files, here is the result: https://blackfire.io/profiles/b07deb06-7a94-4e67-9479-a9e9798f2867/graph

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Feb 27, 2015

Member

Looks like it's spending time trying to propagate/calculate the folder sizes, even for "files_encryption" ?! That should be blocked and maybe rescanned at the end.

That might be what is causing the 255K database queries...

Member

PVince81 commented Feb 27, 2015

Looks like it's spending time trying to propagate/calculate the folder sizes, even for "files_encryption" ?! That should be blocked and maybe rescanned at the end.

That might be what is causing the 255K database queries...

@icewind1991

This comment has been minimized.

Show comment
Hide comment
@icewind1991

icewind1991 Feb 27, 2015

Member

#14573 Should greatly reduce the time the migration takes, does anyone here with a large amount of encrypted data have the opportunity to retest the migration with that patch

Member

icewind1991 commented Feb 27, 2015

#14573 Should greatly reduce the time the migration takes, does anyone here with a large amount of encrypted data have the opportunity to retest the migration with that patch

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Feb 27, 2015

Member

I also noticed that the keys are copied but not moved from the old "keyfiles" to the new "keys" folder. Maybe moving them (if possible) that could provide an additional boost.

I'll retest with that PR first.

Member

PVince81 commented Feb 27, 2015

I also noticed that the keys are copied but not moved from the old "keyfiles" to the new "keys" folder. Maybe moving them (if possible) that could provide an additional boost.

I'll retest with that PR first.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Feb 27, 2015

Member

The PR works well, I also improved it further.

Member

PVince81 commented Feb 27, 2015

The PR works well, I also improved it further.

@DeepDiver1975 DeepDiver1975 modified the milestones: 8.0.2-next-maintenance, 8.0.1-current-maintenance Mar 2, 2015

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Mar 2, 2015

Member

too late for 8.0.1 -> 8.0.2

Member

DeepDiver1975 commented Mar 2, 2015

too late for 8.0.1 -> 8.0.2

@ghost ghost referenced this issue Mar 23, 2015

Closed

upgrade takes too much time #15133

@rjaeckel

This comment has been minimized.

Show comment
Hide comment
@rjaeckel

rjaeckel Mar 26, 2015

Contributor

I also had the Issue that after the Timeout occured, the encryption-keys got currupted.

For anyone who needs to recove the previous keys as well, here is a script generating the actions to be made to recover the pre-update state

#!/bin/bash
### !!! needs to be run as webserver-user
### !!! EDIT this pattern to match all user directories in OC data directory
userpattern='^(.{5}|cloud-administration-user)$'
# recover user keys
for username in $(ls -1tr|egrep -o $userpattern); do
  backup=$(ls -1tr $username|egrep '^encryption_migration_backup'|head -n1)
  if [ "x" != "x$backup" ]; then
    # echo "$username --> $backup"
    # create backup of broken state, just in case
    echo "mv $username/files_encryption $username/files_encryption.broken"
    # recreate the renamed folder
    echo "mkdir -p $username/files_encryption"
    # recover the pre-update state
    echo "cp -r $username/$backup/* $username/files_encryption"
  fi
done

# recover global encryption directories
globbackup=$(ls -1tr|egrep '^encryption_migration_backup'|head -n1)
for bu in $(ls -1 $globbackup); do
  echo "mv $bu $bu.broken"
  echo "cp -r $globbackup/$bu $bu"
done
Contributor

rjaeckel commented Mar 26, 2015

I also had the Issue that after the Timeout occured, the encryption-keys got currupted.

For anyone who needs to recove the previous keys as well, here is a script generating the actions to be made to recover the pre-update state

#!/bin/bash
### !!! needs to be run as webserver-user
### !!! EDIT this pattern to match all user directories in OC data directory
userpattern='^(.{5}|cloud-administration-user)$'
# recover user keys
for username in $(ls -1tr|egrep -o $userpattern); do
  backup=$(ls -1tr $username|egrep '^encryption_migration_backup'|head -n1)
  if [ "x" != "x$backup" ]; then
    # echo "$username --> $backup"
    # create backup of broken state, just in case
    echo "mv $username/files_encryption $username/files_encryption.broken"
    # recreate the renamed folder
    echo "mkdir -p $username/files_encryption"
    # recover the pre-update state
    echo "cp -r $username/$backup/* $username/files_encryption"
  fi
done

# recover global encryption directories
globbackup=$(ls -1tr|egrep '^encryption_migration_backup'|head -n1)
for bu in $(ls -1 $globbackup); do
  echo "mv $bu $bu.broken"
  echo "cp -r $globbackup/$bu $bu"
done
@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Mar 30, 2015

Member

the long execution time of the key migration will be fixed with 8.0.3

Member

DeepDiver1975 commented Mar 30, 2015

the long execution time of the key migration will be fixed with 8.0.3

@rsling

This comment has been minimized.

Show comment
Hide comment
@rsling

rsling May 13, 2015

Well, it was not fixed, at least not for me! I tried multiple times to upgrade from 7 to 8.0.3 on a Debian system. The installation has 5 users and 100GB total, some folders with many small files. After 4 days, I interrupted the procedure. strace showed that the updater kept working on tens of thousands of encryption keys. This is completely useless!

rsling commented May 13, 2015

Well, it was not fixed, at least not for me! I tried multiple times to upgrade from 7 to 8.0.3 on a Debian system. The installation has 5 users and 100GB total, some folders with many small files. After 4 days, I interrupted the procedure. strace showed that the updater kept working on tens of thousands of encryption keys. This is completely useless!

@rjaeckel

This comment has been minimized.

Show comment
Hide comment
@rjaeckel

rjaeckel May 26, 2015

Contributor

@DeepDiver1975
what about considering pcntl as an option? this would make it possible to use the whole computing power... there are several libraries here on github providing php threads this way.

Contributor

rjaeckel commented May 26, 2015

@DeepDiver1975
what about considering pcntl as an option? this would make it possible to use the whole computing power... there are several libraries here on github providing php threads this way.

@hostingnuggets

This comment has been minimized.

Show comment
Hide comment
@hostingnuggets

hostingnuggets Sep 30, 2015

This issue has not been fixed, see #19319

This issue has not been fixed, see #19319

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment