Standalone updater. Initial commit #187

Merged
merged 1 commit into from Dec 21, 2015

Conversation

Projects
None yet
3 participants
@VicDeo
Member

VicDeo commented Nov 17, 2015

As per owncloud/core#18721

Steps are taken from owncloud/core#18721 (comment)

  • upgrade:info Info. Show what is going to happen with explanation.
  • upgrade:checkSystem System check. System health and if dependencies are OK (we also count the number of files and DB entries and give time estimations based on hardcoded estimation)
  • upgrade:checkpoint Create or restore owncloud files
  • upgrade:cleanCache clean all caches
  • upgrade:detect Detect
      1. currently existing code,
      1. version in config.php,
      1. online available verison.
    • ASK what to do? (download, upgrade, abort, …)
  • upgrade:disableNotShippedApps Disable not shipped apps
  • upgrade:enableNotShippedApps try reenable and upgrade 3rdparty/non shipped apps. (one app after another)
  • upgrade:executeCoreUpgradeScripts execute core upgrade scripts [danger, might take long]
  • upgrade:maintenanceMode Toggle maintenance mode
    [danger, might take long]
  • upgrade:restartWebServer Please Restart Web Server
  • upgrade:upgradeShippedApps upgrade shipped apps [danger, might take long]
  • upgrade:start executes
    'upgrade:info'
    'upgrade:checkSystem'
    'upgrade:maintenanceMode --on'
    'upgrade:detect'
    'upgrade:backupDb'
    'upgrade:backupData'
    'upgrade:preUpgradeRepair'
    'upgrade:dbUpgrade simulation'],
    'upgrade:dbUpgrade'
    'upgrade:disableNotShippedApps'
    'upgrade:executeCoreUpgradeScript'
    'upgrade:upgradeShippedApps'
    'upgrade:enableNotShippedApps'
    'upgrade:cleanCache'
    'upgrade:postUpgradeRepair'
    'upgrade:restartWebServer'
    'upgrade:updateConfig'
    'upgrade:maintenanceMode --off'
    'upgrade:postUpgradeCleanup'

Not listed in the comment above but done:

  • downloading
  • checking md5 for package
  • unpacking (zip only)
  • logging
  • check if there are enough permissions to replace files

Postponed for 9.0
moved to #204

@VicDeo

This comment has been minimized.

Show comment
Hide comment
@VicDeo

This comment has been minimized.

Show comment
Hide comment
@VicDeo

VicDeo Nov 17, 2015

Member

As of today it is able to

  • search for occ and version.php one level above it's path
  • read ownCloud config via occ config:list --output "json"
  • construct a feed URL from version,edition,build,etc
  • download a file from the feed
  • get md5 checksum (by adding .md5 postfix to the package URL)
  • validate md5
  • location and file name is always the same so if there is already a package, it just validates md5
  • extract it via console unzip and fallback to PHP built-in ZipArchive (currently extraction path is /tmp but this is going to be changed very soon)

all exceptions are logged into updater.log with a full stacktrace
usage is limited by 8.2+ due to lack of implementation of occ config:list for versions below this one

Member

VicDeo commented Nov 17, 2015

As of today it is able to

  • search for occ and version.php one level above it's path
  • read ownCloud config via occ config:list --output "json"
  • construct a feed URL from version,edition,build,etc
  • download a file from the feed
  • get md5 checksum (by adding .md5 postfix to the package URL)
  • validate md5
  • location and file name is always the same so if there is already a package, it just validates md5
  • extract it via console unzip and fallback to PHP built-in ZipArchive (currently extraction path is /tmp but this is going to be changed very soon)

all exceptions are logged into updater.log with a full stacktrace
usage is limited by 8.2+ due to lack of implementation of occ config:list for versions below this one

@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="UTF-8"?>

This comment has been minimized.

@PVince81

PVince81 Nov 17, 2015

Member

remove "nbproject" folder ?

@PVince81

PVince81 Nov 17, 2015

Member

remove "nbproject" folder ?

This comment has been minimized.

@VicDeo

VicDeo Nov 17, 2015

Member

and add .idea ? ;)

@VicDeo

VicDeo Nov 17, 2015

Member

and add .idea ? ;)

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Nov 17, 2015

Member

I scrolled through the code and it looks good so far.

Would it maybe be better to create a new app/repo for this ?
If we merge this PR we "destroy" the old updater app, so we have to be 100% that the new app replacement will be fully ready for 9.0.

Member

PVince81 commented Nov 17, 2015

I scrolled through the code and it looks good so far.

Would it maybe be better to create a new app/repo for this ?
If we merge this PR we "destroy" the old updater app, so we have to be 100% that the new app replacement will be fully ready for 9.0.

@PVince81 PVince81 added this to the 9.0-current milestone Nov 17, 2015

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Nov 17, 2015

Member

Would it maybe be better to create a new app/repo for this ?

we did discuss this before - since the current updater app will stop to exist starting 9.0 this is the natural approach

Member

DeepDiver1975 commented Nov 17, 2015

Would it maybe be better to create a new app/repo for this ?

we did discuss this before - since the current updater app will stop to exist starting 9.0 this is the natural approach

CONTRIBUTING.md
@@ -1,26 +0,0 @@
-## Submitting issues

This comment has been minimized.

@DeepDiver1975

DeepDiver1975 Nov 17, 2015

Member

we should keep this

@DeepDiver1975

DeepDiver1975 Nov 17, 2015

Member

we should keep this

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Nov 17, 2015

Member

So people on web hosters (no CLI) will not be able to upgrade the code to 9.0 and will have to do it manually with the tarball ? And then with 9.1 or later there will be a new web UI so it will become easy again ?

(reading from owncloud/core#18721 (comment))

the first implementation will be console based only

Member

PVince81 commented Nov 17, 2015

So people on web hosters (no CLI) will not be able to upgrade the code to 9.0 and will have to do it manually with the tarball ? And then with 9.1 or later there will be a new web UI so it will become easy again ?

(reading from owncloud/core#18721 (comment))

the first implementation will be console based only

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Nov 17, 2015

Member

So people on web hosters (no CLI) will not be able to upgrade the code to 9.0 and will have to do it manually with the tarball ? And then with 9.1 or later there will be a new web UI so it will become easy again ?

We need to have a web ui for the updater for 9.0 as well . we simply started out with the console.

Member

DeepDiver1975 commented Nov 17, 2015

So people on web hosters (no CLI) will not be able to upgrade the code to 9.0 and will have to do it manually with the tarball ? And then with 9.1 or later there will be a new web UI so it will become easy again ?

We need to have a web ui for the updater for 9.0 as well . we simply started out with the console.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Dec 1, 2015

Member

@VicDeo what's left to be done ?

Member

PVince81 commented Dec 1, 2015

@VicDeo what's left to be done ?

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Dec 1, 2015

Member

Would be good to have a list of checkboxes for all that it can already do and what's left to be done (for the 9.0 scoped things)

Member

PVince81 commented Dec 1, 2015

Would be good to have a list of checkboxes for all that it can already do and what's left to be done (for the 9.0 scoped things)

@VicDeo

This comment has been minimized.

Show comment
Hide comment
@VicDeo

VicDeo Dec 1, 2015

Member

@PVince81 I guess no one made a decision what exactly is in the scope of 9.0
What I'm doing now is trying to gain functionality the current Updater app has.
I added the checkboxes according to all steps requested by @karlitschek. All of them have generic implementation atm. (and do nothing useful).

I added checkpoint command, that is needed to revert unsuccessful upgrade attempt and some other things

Member

VicDeo commented Dec 1, 2015

@PVince81 I guess no one made a decision what exactly is in the scope of 9.0
What I'm doing now is trying to gain functionality the current Updater app has.
I added the checkboxes according to all steps requested by @karlitschek. All of them have generic implementation atm. (and do nothing useful).

I added checkpoint command, that is needed to revert unsuccessful upgrade attempt and some other things

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Dec 1, 2015

Member

Okay thanks. So looks like there is still a lot to do.

In the meantime I guess that some could be called through occ first and then moved to this app later ?

Member

PVince81 commented Dec 1, 2015

Okay thanks. So looks like there is still a lot to do.

In the meantime I guess that some could be called through occ first and then moved to this app later ?

@VicDeo

This comment has been minimized.

Show comment
Hide comment
@VicDeo

VicDeo Dec 1, 2015

Member

@PVince81 according to @DeepDiver1975 proposal as much as possible (~ everything) should be implemented via occ command.
e.g. executeCoreUpgradeScripts command just executes occ upgrade

Member

VicDeo commented Dec 1, 2015

@PVince81 according to @DeepDiver1975 proposal as much as possible (~ everything) should be implemented via occ command.
e.g. executeCoreUpgradeScripts command just executes occ upgrade

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Dec 15, 2015

Member

From a first run experience - there has to be an updater.sh or what so ever - we might still want to research into the phar concept once we have a free minute

Member

DeepDiver1975 commented Dec 15, 2015

From a first run experience - there has to be an updater.sh or what so ever - we might still want to research into the phar concept once we have a free minute

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Dec 15, 2015

Member

@karlitschek what do you think about adding the pure installation use case to this tool as well?

Installing an instance is basically the same as upgrade without the backup 😉

Member

DeepDiver1975 commented Dec 15, 2015

@karlitschek what do you think about adding the pure installation use case to this tool as well?

Installing an instance is basically the same as upgrade without the backup 😉

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Dec 15, 2015

Member

@VicDeo invocation of occ did fail for me because occ had no execution flag.
To be on the save side we might want to run php console.php instead ?

Member

DeepDiver1975 commented Dec 15, 2015

@VicDeo invocation of occ did fail for me because occ had no execution flag.
To be on the save side we might want to run php console.php instead ?

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Dec 15, 2015

Member

we need to add a version check - I was testing this on a 8.0 instance and got:

The instance is not installed - and it broke because of that.

$ ./application.php 



  [UnexpectedValueException]                                                                                            
  Could not parse a response for config:list. Please check if the current shell user can run occ command. Raw output:   
  An unhandled exception has been thrown:                                                                               
  Exception: Not installed in /home/deepdiver/Development/ownCloud/test_updater/lib/base.php:223                        
  Stack trace:                                                                                                          
  #0 /home/deepdiver/Development/ownCloud/test_updater/lib/base.php(558): OC::checkInstalled()                          
  #1 /home/deepdiver/Development/ownCloud/test_updater/lib/base.php(1039): OC::init()                                   
  #2 /home/deepdiver/Development/ownCloud/test_updater/console.php(12): require_once('/home/deepdiver...')              
  #3 /home/deepdiver/Development/ownCloud/test_updater/occ(11): require_once('/home/deepdiver...')                      
  #4 {main}                                                                                                             


Member

DeepDiver1975 commented Dec 15, 2015

we need to add a version check - I was testing this on a 8.0 instance and got:

The instance is not installed - and it broke because of that.

$ ./application.php 



  [UnexpectedValueException]                                                                                            
  Could not parse a response for config:list. Please check if the current shell user can run occ command. Raw output:   
  An unhandled exception has been thrown:                                                                               
  Exception: Not installed in /home/deepdiver/Development/ownCloud/test_updater/lib/base.php:223                        
  Stack trace:                                                                                                          
  #0 /home/deepdiver/Development/ownCloud/test_updater/lib/base.php(558): OC::checkInstalled()                          
  #1 /home/deepdiver/Development/ownCloud/test_updater/lib/base.php(1039): OC::init()                                   
  #2 /home/deepdiver/Development/ownCloud/test_updater/console.php(12): require_once('/home/deepdiver...')              
  #3 /home/deepdiver/Development/ownCloud/test_updater/occ(11): require_once('/home/deepdiver...')                      
  #4 {main}                                                                                                             


@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Dec 15, 2015

Member

furthermore - we need to output anything more user friendly - no stack traces and clear sentences

Member

DeepDiver1975 commented Dec 15, 2015

furthermore - we need to output anything more user friendly - no stack traces and clear sentences

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Dec 15, 2015

Member

And this is where we require a version check 😉


[error] 


  [Symfony\Component\Process\Exception\ProcessFailedException]                                              
  The command "/home/deepdiver/Development/ownCloud/test_updater/occ config:list  --output "json"" failed.  
  Exit Code: 1(General error)                                                                               
  Output:                                                                                                   
  ================                                                                                          
  Error Output:                                                                                             
  ================                                                                                          

    [InvalidArgumentException]                                                                              
    There are no commands defined in the "config" namespace.                                                




Exception trace:
 () at /home/deepdiver/Development/ownCloud/test_updater/updater/src/Utils/OccRunner.php:47
 Owncloud\Updater\Utils\OccRunner->run() at /home/deepdiver/Development/ownCloud/test_updater/updater/src/Utils/OccRunner.php:53
 Owncloud\Updater\Utils\OccRunner->runJson() at /home/deepdiver/Development/ownCloud/test_updater/updater/src/Utils/ConfigReader.php:94
 Owncloud\Updater\Utils\ConfigReader->load() at /home/deepdiver/Development/ownCloud/test_updater/updater/src/Utils/ConfigReader.php:45
 Owncloud\Updater\Utils\ConfigReader->init() at /home/deepdiver/Development/ownCloud/test_updater/updater/src/Console/Application.php:92
 Owncloud\Updater\Console\Application->doRun() at /home/deepdiver/Development/ownCloud/test_updater/updater/vendor/symfony/console/Application.php:120
 Symfony\Component\Console\Application->run() at /home/deepdiver/Development/ownCloud/test_updater/updater/application.php:42
Member

DeepDiver1975 commented Dec 15, 2015

And this is where we require a version check 😉


[error] 


  [Symfony\Component\Process\Exception\ProcessFailedException]                                              
  The command "/home/deepdiver/Development/ownCloud/test_updater/occ config:list  --output "json"" failed.  
  Exit Code: 1(General error)                                                                               
  Output:                                                                                                   
  ================                                                                                          
  Error Output:                                                                                             
  ================                                                                                          

    [InvalidArgumentException]                                                                              
    There are no commands defined in the "config" namespace.                                                




Exception trace:
 () at /home/deepdiver/Development/ownCloud/test_updater/updater/src/Utils/OccRunner.php:47
 Owncloud\Updater\Utils\OccRunner->run() at /home/deepdiver/Development/ownCloud/test_updater/updater/src/Utils/OccRunner.php:53
 Owncloud\Updater\Utils\OccRunner->runJson() at /home/deepdiver/Development/ownCloud/test_updater/updater/src/Utils/ConfigReader.php:94
 Owncloud\Updater\Utils\ConfigReader->load() at /home/deepdiver/Development/ownCloud/test_updater/updater/src/Utils/ConfigReader.php:45
 Owncloud\Updater\Utils\ConfigReader->init() at /home/deepdiver/Development/ownCloud/test_updater/updater/src/Console/Application.php:92
 Owncloud\Updater\Console\Application->doRun() at /home/deepdiver/Development/ownCloud/test_updater/updater/vendor/symfony/console/Application.php:120
 Symfony\Component\Console\Application->run() at /home/deepdiver/Development/ownCloud/test_updater/updater/application.php:42
@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Dec 15, 2015

Member
  • application.php list does not output the list of commands
  • application.php without any argument should list an introduction and kick off the upgrade
Member

DeepDiver1975 commented Dec 15, 2015

  • application.php list does not output the list of commands
  • application.php without any argument should list an introduction and kick off the upgrade
@VicDeo

This comment has been minimized.

Show comment
Hide comment
@VicDeo

VicDeo Dec 15, 2015

Member

@DeepDiver1975 it should, after the commit More stability if occ fails early (23a4c2c)

Member

VicDeo commented Dec 15, 2015

@DeepDiver1975 it should, after the commit More stability if occ fails early (23a4c2c)

@VicDeo

This comment has been minimized.

Show comment
Hide comment
@VicDeo

VicDeo Dec 15, 2015

Member

I mean application.php list does not output the list of commands

Member

VicDeo commented Dec 15, 2015

I mean application.php list does not output the list of commands

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Dec 21, 2015

Member

Let's merge this in now and continue with PRs against master to push the functionalities

Member

DeepDiver1975 commented Dec 21, 2015

Let's merge this in now and continue with PRs against master to push the functionalities

DeepDiver1975 added a commit that referenced this pull request Dec 21, 2015

Merge pull request #187 from owncloud/new-updater
Standalone updater. Initial commit

@DeepDiver1975 DeepDiver1975 merged commit 542ddb7 into master Dec 21, 2015

2 checks passed

Scrutinizer 342 Issues, 27 Patches
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@DeepDiver1975 DeepDiver1975 deleted the new-updater branch Dec 21, 2015

@PVince81 PVince81 referenced this pull request in owncloud/core Jan 8, 2016

Open

Non-disruptive Upgrades #18721

@VicDeo VicDeo referenced this pull request Jan 8, 2016

Merged

More unit tests and enhancements #213

2 of 6 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment