New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Non-disruptive Upgrades #18721

Open
MTRichards opened this Issue Aug 31, 2015 · 42 comments

Comments

@MTRichards
Contributor

MTRichards commented Aug 31, 2015

Goal:

  • Upgrade NEVER leads to a non working ownCloud installation. (ownCloud should never be in a state where it is not working / can't be recovered)
  • Done very fast (1min) (long running steps should happen before or after asynchronously)
  • Works from CLI and web
  • Always tells the admin whats happening now, what happens next and how long it will take.
  • Offers roll-back

Approach:

  • New isolated updater, all code files are located in /updater, no interaction with the rest of ownCloud.
  • No code, configuration, database or anything else used from core is required. (- included with normal ‘required’ to make it isolated)
  • Shipped with ownCloud core
  • Can be called by CLI or web (updater/update.php or /updater)
  • Offer interactive or automatic mode (interactive recommended)
  • Offers to do a /data backup if possible
  • Offers to do a db backup if possible
  • Offers to do a full rollback if failed or offers a Retry if a steps fails temporarily.
  • Support the admin with cleaning caches/backup
  • Can detect failed app upgrades and can move failed apps into a ‘incompatible-apps’ folder
  • Included tipps and tricks and hints
  • Included links to documentation for more information
  • Upgrade steps should be split up into pre/post steps and optional/required steps. For example mimetype conversion can happen when an user requests his files instead of doing 16 hour queries on update.
  • ownCloud should check itself for integrity problems. So it ownCloud and the DB schema are unmodified or changed.
  • It should be possible to export schema migration statements to do the DB migration asynchronously.
    Currently: On-the fly schema migration, diff
    Wanted: Have predictable database upgrade changes, move to define the migration steps
  • DOES NEVER BREAK

Concrete process:

  • tbd

Todos:

  • Check how wordpress handles it
  • Check how Piwik handles it
  • Go through existing issues to see if we catch all problems
  • somehow make sure that the admin doesn't miss new config options
@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Aug 31, 2015

Ref to an older issue at the documentation tracker: owncloud/documentation#394 which contains some basic discussions.

ghost commented Aug 31, 2015

Ref to an older issue at the documentation tracker: owncloud/documentation#394 which contains some basic discussions.

@Xenopathic

This comment has been minimized.

Show comment
Hide comment
@Xenopathic

Xenopathic Aug 31, 2015

Member

Two parts to an upgrade: code upgrade and DB upgrade.

Code upgrades need to ensure the server only sees one of two states - full old code or full new code. If some classes are new while others are still not updated then things will break. The ways to do this are with filesystem snapshots and atomic upgrades (supported by btrfs and other modern filesystems), where a snapshot of the filesystem is taken, changes made to it while the PHP process still sees the snapshot, then updating the snapshot to reflect the changed files.

DB upgrades are somewhat more difficult. I imagine the easiest thing to start with is setting oC to read-only, so no DB writes can occur. This would greatly help keeping things consistent. Then, just like code upgrades, the schema changes need to be atomic. I think transactions can help here? In fact, ideally schema updates and repair steps would all happen in one atomic operation, so the DB goes from pre-upgrade to post-upgrade and no running instance sees anything in between. I'm not sure how feasible this is though.

Finally is the problem of synchronising code upgrades and DB upgrades. When the code is upgraded but the DB is still old, ownCloud cannot work. Likewise the other way around. Perhaps the best way to handle this is to prepare filesystem snapshots on each server, perform the atomic database upgrade, then apply the filesystem snapshots. The only downtime during this would be after the database upgrade but before the filesystem snapshot update, which should be short as applying snapshots is usually very fast. Unfortunately the only way I see online DB upgrades working properly is by only allowing read-only access for the short duration.

Member

Xenopathic commented Aug 31, 2015

Two parts to an upgrade: code upgrade and DB upgrade.

Code upgrades need to ensure the server only sees one of two states - full old code or full new code. If some classes are new while others are still not updated then things will break. The ways to do this are with filesystem snapshots and atomic upgrades (supported by btrfs and other modern filesystems), where a snapshot of the filesystem is taken, changes made to it while the PHP process still sees the snapshot, then updating the snapshot to reflect the changed files.

DB upgrades are somewhat more difficult. I imagine the easiest thing to start with is setting oC to read-only, so no DB writes can occur. This would greatly help keeping things consistent. Then, just like code upgrades, the schema changes need to be atomic. I think transactions can help here? In fact, ideally schema updates and repair steps would all happen in one atomic operation, so the DB goes from pre-upgrade to post-upgrade and no running instance sees anything in between. I'm not sure how feasible this is though.

Finally is the problem of synchronising code upgrades and DB upgrades. When the code is upgraded but the DB is still old, ownCloud cannot work. Likewise the other way around. Perhaps the best way to handle this is to prepare filesystem snapshots on each server, perform the atomic database upgrade, then apply the filesystem snapshots. The only downtime during this would be after the database upgrade but before the filesystem snapshot update, which should be short as applying snapshots is usually very fast. Unfortunately the only way I see online DB upgrades working properly is by only allowing read-only access for the short duration.

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Sep 1, 2015

Member

Code update is of zero problem: web Server nodes to be updated will be taken offline.

Anything else is highly critical. No immediate ideas for how it could be done.

What we should AIM for is a more manageable upgrade process which is actually testable and reproducible.

Let me once more reference doctrine migrations to be used instead of this mixture of Schema diffing magic plus wild php scripting.

Member

DeepDiver1975 commented Sep 1, 2015

Code update is of zero problem: web Server nodes to be updated will be taken offline.

Anything else is highly critical. No immediate ideas for how it could be done.

What we should AIM for is a more manageable upgrade process which is actually testable and reproducible.

Let me once more reference doctrine migrations to be used instead of this mixture of Schema diffing magic plus wild php scripting.

@MTRichards MTRichards changed the title from Near Zero Downtime Upgrades to Non-disruptive Upgrades Sep 1, 2015

@zijlstra-it

This comment has been minimized.

Show comment
Hide comment
@zijlstra-it

zijlstra-it Sep 2, 2015

I'd like to add to this that being able to "produce" a complete DB change script, so not only the schema changes, would be very helpful to being able to test and time the databases updates on a test instance of the database.
For me it's unfeasible to have a test environment with the same number of accounts and data, but a database copy is no problem and a system with the same specs either, so if I would be able to run ALL database changes for an upgrade on a test DB, I can time the process and test if it works at all in our case.
I used the "db:generate-change-script" to test our upgrade to 7.06 from 6.06 but it only covered the schema changes, not the table updates due to changed behaviour of OC, e.g. the "Shared" folder which changed from 6 to 7.

zijlstra-it commented Sep 2, 2015

I'd like to add to this that being able to "produce" a complete DB change script, so not only the schema changes, would be very helpful to being able to test and time the databases updates on a test instance of the database.
For me it's unfeasible to have a test environment with the same number of accounts and data, but a database copy is no problem and a system with the same specs either, so if I would be able to run ALL database changes for an upgrade on a test DB, I can time the process and test if it works at all in our case.
I used the "db:generate-change-script" to test our upgrade to 7.06 from 6.06 but it only covered the schema changes, not the table updates due to changed behaviour of OC, e.g. the "Shared" folder which changed from 6 to 7.

@MTRichards

This comment has been minimized.

Show comment
Hide comment
@MTRichards

MTRichards Sep 18, 2015

Contributor

Disable an app after it fails to load to remove WSOD - prototype: #18599 (comment)

Contributor

MTRichards commented Sep 18, 2015

Disable an app after it fails to load to remove WSOD - prototype: #18599 (comment)

@karlitschek

This comment has been minimized.

Show comment
Hide comment
@karlitschek

karlitschek Sep 25, 2015

Member

proposal for the concrete upgrade process:

upgrade++ NG 2.0

  • isolated upgrade in /upgrade
  • Web and CLI
  • All steps can be executed seperately with CLI options
  • default is interactive mode
  • write everything into an upgrade.log
  • write a status file to make resume possible.

Other:

  • Make it possible to export a DB migration sctipt

upgrade steps

  • Info. Show what is going to happen with explanation.
  • System check. System health and if dependencies are OK (we also count the number of files and DB entries and give time estimations based on hardcoded estimation)
  • maintainance mode on

Detect
- 1. currently existing code,
- 2. version in config.php,
- 3. online available verison.
(ASK) what to do? (download, upgrade, abort, …)

in case of download: (
- download new ownCloud release
- backup of old code (optionally)
- unpack tar and replace code
- exit and tell user to please restart upgrade script
)

  • Backup DB (optionally)
  • Backup data (optionally)
  • Repair and cleanup (pre upgrade, DB collations update, ..) [danger, might take long]
  • db schema upgrade simulated (optionally, can be done online in advance)
  • db schema upgrade real
    [danger, might take long]
  • disable 3rdparty/no shipped apps
  • execute core upgrade scripts
    [danger, might take long]
  • upgrade shipped apps [danger, might take long]
    • schema migration
    • app upgrade scripts
  • try reenable and upgrade 3rdparty/non shipped apps. (one app after another)
  • clean caches
  • repair and cleanup step 1 (post upgrade, repair legacy storage, ..) [danger, might take long]
  • restart webserver (ask, propably a good idea, just to be sure)
  • update config.php
  • maintainance mode off
  • repair and cleanup step 2 (online) [danger, might take long]

Meta todos for upcoming releases:

  • DB: no more table/column changes. Only add new tables/columns
  • Repair scripts: make possible to run in production
  • migration scripts: handle half migrated status in production
Member

karlitschek commented Sep 25, 2015

proposal for the concrete upgrade process:

upgrade++ NG 2.0

  • isolated upgrade in /upgrade
  • Web and CLI
  • All steps can be executed seperately with CLI options
  • default is interactive mode
  • write everything into an upgrade.log
  • write a status file to make resume possible.

Other:

  • Make it possible to export a DB migration sctipt

upgrade steps

  • Info. Show what is going to happen with explanation.
  • System check. System health and if dependencies are OK (we also count the number of files and DB entries and give time estimations based on hardcoded estimation)
  • maintainance mode on

Detect
- 1. currently existing code,
- 2. version in config.php,
- 3. online available verison.
(ASK) what to do? (download, upgrade, abort, …)

in case of download: (
- download new ownCloud release
- backup of old code (optionally)
- unpack tar and replace code
- exit and tell user to please restart upgrade script
)

  • Backup DB (optionally)
  • Backup data (optionally)
  • Repair and cleanup (pre upgrade, DB collations update, ..) [danger, might take long]
  • db schema upgrade simulated (optionally, can be done online in advance)
  • db schema upgrade real
    [danger, might take long]
  • disable 3rdparty/no shipped apps
  • execute core upgrade scripts
    [danger, might take long]
  • upgrade shipped apps [danger, might take long]
    • schema migration
    • app upgrade scripts
  • try reenable and upgrade 3rdparty/non shipped apps. (one app after another)
  • clean caches
  • repair and cleanup step 1 (post upgrade, repair legacy storage, ..) [danger, might take long]
  • restart webserver (ask, propably a good idea, just to be sure)
  • update config.php
  • maintainance mode off
  • repair and cleanup step 2 (online) [danger, might take long]

Meta todos for upcoming releases:

  • DB: no more table/column changes. Only add new tables/columns
  • Repair scripts: make possible to run in production
  • migration scripts: handle half migrated status in production
@MorrisJobke

This comment has been minimized.

Show comment
Hide comment
@MorrisJobke

MorrisJobke Sep 25, 2015

Member

Copy of the notes from our session at the conf - @karlitschek feel free to merge this with you comment:

Goals

  • Never kill OC
  • ownCloud should never be in a state where it is not working / can't be recovered
  • Fast
  • Interactive
    • No user feedback, should have more verbosity => How long does each step take / estimates / …
    • Offer advises such as "Please create a database backup"
    • Morris: Interactivity will create problems for enterprise deployments, more verbosity would be good. => Where is the problem? Deployment problem or ownCloud bug?
    • For example when occ runs show what will run and offer possibility to skip some migrations
  • Wordpress / Piwik-Reliability

Ideas

  • Isolated updater code
    • Self-contained, so that it does not shoot itself into the foot by accident. Such as replacing files that it depends on and then clearing OPCache.
  • CLI / WEB
    • share as most as possible code paths
  • App Enable Check
    • a.k.a disable apps that break stuff
    • Lukas: Not moving apps to "/incompatible-apps" due to security concerns ("Enterprises do want have a Read-only oC")
  • Pre/Post steps
    • For example mimetype conversion can happen when an user requests his files instead of doing 16 hour queries on update.
    • Have:
      • Pre-steps, before the update, required
      • Post-steps, after the update, can be done while it is running in production
      • Optional (maybe)
  • Caches
    • Ensure that all caches are cleaned (memcached / APCu / …)
    • => Already done by prepending the caching keys with the version
  • Backups
  • Handholding
    • Help users more, such as explaining how to restart their web server
  • Time estimations (see "Interactive" from "Goals")
  • Progress (see "Interactive" from "Goals")
    • Update steps might take from hours and there is no feedback, some customers believe these were bugs and stopped the update => Broken instances
  • Upgrading apps
    • Exception in apps that aren't tested properly/"interesting configuration" -> kills the whole upgrade run
  • Rollback on failure
    • Half-way migrated, something is missing, no rollback for this
    • Unlikely to be implemented, very hard to do right, can be done with snapshots / backups.
  • Integrity problems
    • some files are not upgraded
      • Verify if the code that is currently running, is exactly the code that is running (possible candidate: .htaccess because of hidden file)
    • Database
      • People are applying random patches to database scheme
  • App disabled during upgrade
    • Re-enable => Kills instance because of upgrade page
      • Enable an app => Get upgrade mode, enable another one via Ajax => Fails since in upgrade mode
  • Proper schema migration
    • Currently: On-the fly schema migration, diff
    • Wanted: Have predictable database upgrade changes, move to define the migration steps
  • Partial upgrade for some users/migration of users from old version instance to a new version instance owncloud/enterprise#686
    • Wanted: Have a test group that uses new app
      • Not possible to update that easily
      • Jörns idea: Have an old and new version and move users, for sharing use federated sharing
Member

MorrisJobke commented Sep 25, 2015

Copy of the notes from our session at the conf - @karlitschek feel free to merge this with you comment:

Goals

  • Never kill OC
  • ownCloud should never be in a state where it is not working / can't be recovered
  • Fast
  • Interactive
    • No user feedback, should have more verbosity => How long does each step take / estimates / …
    • Offer advises such as "Please create a database backup"
    • Morris: Interactivity will create problems for enterprise deployments, more verbosity would be good. => Where is the problem? Deployment problem or ownCloud bug?
    • For example when occ runs show what will run and offer possibility to skip some migrations
  • Wordpress / Piwik-Reliability

Ideas

  • Isolated updater code
    • Self-contained, so that it does not shoot itself into the foot by accident. Such as replacing files that it depends on and then clearing OPCache.
  • CLI / WEB
    • share as most as possible code paths
  • App Enable Check
    • a.k.a disable apps that break stuff
    • Lukas: Not moving apps to "/incompatible-apps" due to security concerns ("Enterprises do want have a Read-only oC")
  • Pre/Post steps
    • For example mimetype conversion can happen when an user requests his files instead of doing 16 hour queries on update.
    • Have:
      • Pre-steps, before the update, required
      • Post-steps, after the update, can be done while it is running in production
      • Optional (maybe)
  • Caches
    • Ensure that all caches are cleaned (memcached / APCu / …)
    • => Already done by prepending the caching keys with the version
  • Backups
  • Handholding
    • Help users more, such as explaining how to restart their web server
  • Time estimations (see "Interactive" from "Goals")
  • Progress (see "Interactive" from "Goals")
    • Update steps might take from hours and there is no feedback, some customers believe these were bugs and stopped the update => Broken instances
  • Upgrading apps
    • Exception in apps that aren't tested properly/"interesting configuration" -> kills the whole upgrade run
  • Rollback on failure
    • Half-way migrated, something is missing, no rollback for this
    • Unlikely to be implemented, very hard to do right, can be done with snapshots / backups.
  • Integrity problems
    • some files are not upgraded
      • Verify if the code that is currently running, is exactly the code that is running (possible candidate: .htaccess because of hidden file)
    • Database
      • People are applying random patches to database scheme
  • App disabled during upgrade
    • Re-enable => Kills instance because of upgrade page
      • Enable an app => Get upgrade mode, enable another one via Ajax => Fails since in upgrade mode
  • Proper schema migration
    • Currently: On-the fly schema migration, diff
    • Wanted: Have predictable database upgrade changes, move to define the migration steps
  • Partial upgrade for some users/migration of users from old version instance to a new version instance owncloud/enterprise#686
    • Wanted: Have a test group that uses new app
      • Not possible to update that easily
      • Jörns idea: Have an old and new version and move users, for sharing use federated sharing
@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Sep 25, 2015

Member

Note regarding pre/post steps: everything that is done "lazily" needs to be properly protected by locking to make sure the migration isn't done twice and causing unpredictable results.
Same for anything that is done "at login time", considering that multiple logins can happen at the same time.

Member

PVince81 commented Sep 25, 2015

Note regarding pre/post steps: everything that is done "lazily" needs to be properly protected by locking to make sure the migration isn't done twice and causing unpredictable results.
Same for anything that is done "at login time", considering that multiple logins can happen at the same time.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Sep 25, 2015

Member

Also to note:

  • the current app architecture doesn't allow for generating SQL statements because apps can provide generic PHP code in "appinfo/update.php" which is simply run when updating the app. To fix this, the apps would need to be told that we only want to get the SQL from the possible update and not actually perform it. Or deprecate "appinfo/update.php" and expect apps to implement the RepairStep interface for better compliance with the process. (currently only core can provide/register RepairStep instances)
  • some repair steps are not running SQL queries directly but using OC core code which contains the logic necessary. We might need to decouple the repair steps from the core code, if possible. (and make this a guideline when writing new repair steps)
  • some repair steps might operate directly on files (not purely SQL/DB changes). I'm not aware of any that does so at the moment: the encryption app usually requires the admin to run an additional "./occ" command to operate on files.
Member

PVince81 commented Sep 25, 2015

Also to note:

  • the current app architecture doesn't allow for generating SQL statements because apps can provide generic PHP code in "appinfo/update.php" which is simply run when updating the app. To fix this, the apps would need to be told that we only want to get the SQL from the possible update and not actually perform it. Or deprecate "appinfo/update.php" and expect apps to implement the RepairStep interface for better compliance with the process. (currently only core can provide/register RepairStep instances)
  • some repair steps are not running SQL queries directly but using OC core code which contains the logic necessary. We might need to decouple the repair steps from the core code, if possible. (and make this a guideline when writing new repair steps)
  • some repair steps might operate directly on files (not purely SQL/DB changes). I'm not aware of any that does so at the moment: the encryption app usually requires the admin to run an additional "./occ" command to operate on files.
@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Sep 25, 2015

Member

As discussed previously with @schiesbn, long running repair steps (or on-demand) might work, but they would also need to be able to run in many future versions of OC.
Consider the following scenario

  • setup is currently on OC 9.0
  • admin decides to update to OC 12.0 (without skipping major versions)
  • repair step "FixSomething" is run in the background for all users without disrupting functionality
  • whenever a user accesses a folder, "FixMimeType" is applied (for example)
  • admin upgrades to OC 9.1
  • repair step "FixSomething" hasn't finished, so need to continue running
  • new repair step "FixSomethingAfterFixSomething" (needs "FixSomething") to finish first
  • "FixMimeType" hasn't been run for all files because not every user has viewed all their (unchanged) folders
  • admin upgrades to OC 9.2
  • repair step "FixSomething" still running
  • ...
  • admin upgrades to OC 12
  • at this point, the repair steps should still be able to continue running, including the "lazy" fixes like "FixMimeType".

This is only a potential issue that can happen, so we need to consider that when reaching OC 12, some database entries might still be in the state pre-OC 9, so the code that repairs them needs to remain in place. (and potentially need to be retested, hopefully automatically)

Basically, this brings the more detailed requirement:

  • old repair steps (code) still need to be available in the newest versions and be able to run / continue running
  • lazy/on-demand repair steps need to work in the newest versions
  • always assume that the DB might have unrepaired entries from older updates
Member

PVince81 commented Sep 25, 2015

As discussed previously with @schiesbn, long running repair steps (or on-demand) might work, but they would also need to be able to run in many future versions of OC.
Consider the following scenario

  • setup is currently on OC 9.0
  • admin decides to update to OC 12.0 (without skipping major versions)
  • repair step "FixSomething" is run in the background for all users without disrupting functionality
  • whenever a user accesses a folder, "FixMimeType" is applied (for example)
  • admin upgrades to OC 9.1
  • repair step "FixSomething" hasn't finished, so need to continue running
  • new repair step "FixSomethingAfterFixSomething" (needs "FixSomething") to finish first
  • "FixMimeType" hasn't been run for all files because not every user has viewed all their (unchanged) folders
  • admin upgrades to OC 9.2
  • repair step "FixSomething" still running
  • ...
  • admin upgrades to OC 12
  • at this point, the repair steps should still be able to continue running, including the "lazy" fixes like "FixMimeType".

This is only a potential issue that can happen, so we need to consider that when reaching OC 12, some database entries might still be in the state pre-OC 9, so the code that repairs them needs to remain in place. (and potentially need to be retested, hopefully automatically)

Basically, this brings the more detailed requirement:

  • old repair steps (code) still need to be available in the newest versions and be able to run / continue running
  • lazy/on-demand repair steps need to work in the newest versions
  • always assume that the DB might have unrepaired entries from older updates

@MTRichards MTRichards added this to the 9.0-next milestone Sep 25, 2015

@VicDeo

This comment has been minimized.

Show comment
Hide comment
@VicDeo

VicDeo Sep 25, 2015

Member

what make the life painful right now:

  • users may have some unrelated items in the webserver root directory (webalazer logs, etc) or other places. And there is not possibility to detect whether any file belongs to the owncloud or not.
    IMO some sort of a centralized storage that knows exact list of files for any version of owncloud is mandatory if we want to introduce skipping versions.
  • movable 3rdparty dir and several apps directories (writable/not writable)
Member

VicDeo commented Sep 25, 2015

what make the life painful right now:

  • users may have some unrelated items in the webserver root directory (webalazer logs, etc) or other places. And there is not possibility to detect whether any file belongs to the owncloud or not.
    IMO some sort of a centralized storage that knows exact list of files for any version of owncloud is mandatory if we want to introduce skipping versions.
  • movable 3rdparty dir and several apps directories (writable/not writable)
@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Sep 25, 2015

Member
  • making the web based updater is tricky if we want it to spawn processes when driving the update, popen() and exec() might be blocked in shared hosters which are the main target for such a web UI
Member

PVince81 commented Sep 25, 2015

  • making the web based updater is tricky if we want it to spawn processes when driving the update, popen() and exec() might be blocked in shared hosters which are the main target for such a web UI
@VicDeo

This comment has been minimized.

Show comment
Hide comment
@VicDeo

VicDeo Oct 27, 2015

Member

Wordpress reliability... hm
I would say it is reliable because it doesn't use schema and supports nothing but MySQL out of a box https://github.com/WordPress/WordPress/blob/master/wp-admin/includes/upgrade.php#L478

UPD I wonder how much time these steps will take for e.g. 1000 users having 10000 posts...

Member

VicDeo commented Oct 27, 2015

Wordpress reliability... hm
I would say it is reliable because it doesn't use schema and supports nothing but MySQL out of a box https://github.com/WordPress/WordPress/blob/master/wp-admin/includes/upgrade.php#L478

UPD I wonder how much time these steps will take for e.g. 1000 users having 10000 posts...

@MorrisJobke

This comment has been minimized.

Show comment
Hide comment
Member

MorrisJobke commented Oct 27, 2015

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Oct 28, 2015

Member

Wordpress also has update issues, just read the tweets https://twitter.com/search?q=wordpress%20update%20break&src=typd

Anyway, comparing to Wordpress is not the point. The point is simply to make OC updates more reliable.

Member

PVince81 commented Oct 28, 2015

Wordpress also has update issues, just read the tweets https://twitter.com/search?q=wordpress%20update%20break&src=typd

Anyway, comparing to Wordpress is not the point. The point is simply to make OC updates more reliable.

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Oct 28, 2015

Member

Anyway, comparing to Wordpress is not the point. The point is simply to make OC updates more reliable.

indeed - but there was the statement: "Other do upgrade without problems - like wordpress ..."

Member

DeepDiver1975 commented Oct 28, 2015

Anyway, comparing to Wordpress is not the point. The point is simply to make OC updates more reliable.

indeed - but there was the statement: "Other do upgrade without problems - like wordpress ..."

@MorrisJobke

This comment has been minimized.

Show comment
Hide comment
@MorrisJobke

MorrisJobke Oct 28, 2015

Member

UPD I wonder how much time these steps will take for e.g. 1000 users having 10000 posts...

It's simply not possible to compare Wordpress&Piwik with ownCloud because the amount of data are completely different. We need to find our way to make it reliable.

Member

MorrisJobke commented Oct 28, 2015

UPD I wonder how much time these steps will take for e.g. 1000 users having 10000 posts...

It's simply not possible to compare Wordpress&Piwik with ownCloud because the amount of data are completely different. We need to find our way to make it reliable.

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Nov 6, 2015

Member

@karlitschek @cmonteroluque @PVince81 regarding the location of the new tool which is under development: I propose to put it into the existing updater app repo.

The updater app as we ship it until 8.2 will no longer be used and the code can continue to live in the stableX branches. Master will hold the new tool.

Objections?

Member

DeepDiver1975 commented Nov 6, 2015

@karlitschek @cmonteroluque @PVince81 regarding the location of the new tool which is under development: I propose to put it into the existing updater app repo.

The updater app as we ship it until 8.2 will no longer be used and the code can continue to live in the stableX branches. Master will hold the new tool.

Objections?

@karlitschek

This comment has been minimized.

Show comment
Hide comment
@karlitschek

karlitschek Nov 6, 2015

Member

Works for me. We will move it into /updater of the owncloud root in the tar files release.

Member

karlitschek commented Nov 6, 2015

Works for me. We will move it into /updater of the owncloud root in the tar files release.

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Nov 6, 2015

Member

Works for me. We will move it into /updater of the owncloud root in the tar files release.

THX - @VicDeo please prepare a PR these days - THX

Member

DeepDiver1975 commented Nov 6, 2015

Works for me. We will move it into /updater of the owncloud root in the tar files release.

THX - @VicDeo please prepare a PR these days - THX

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Nov 6, 2015

Member

It would be good to raise a separate ticket for the requirements/acceptance criteria that are scoped for 9.0 to get a clearer picture. I thought we had a list already, can't seem to find it.
Thanks.

Member

PVince81 commented Nov 6, 2015

It would be good to raise a separate ticket for the requirements/acceptance criteria that are scoped for 9.0 to get a clearer picture. I thought we had a list already, can't seem to find it.
Thanks.

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Nov 6, 2015

Member

@PVince81 just to document the current apporach:

  • the first implementation will be console based only
  • capabilities for web based user interaction is being thought off during implementation of the first step
  • steps to be implemented are #18721 (comment)
  • the tool will be developed isolated from owncloud core as self contained web and console project
  • it is expected to have the tool running in a webspace root in a subfolder 'updater'
  • all steps in it's first implementation will hold simple test out put to tell the user what to do 😉
  • interaction with the existing/installed/updated core is to be done via the occ command line tool (unclear in case of web based approach)

@VicDeo did I miss anything? THX

Member

DeepDiver1975 commented Nov 6, 2015

@PVince81 just to document the current apporach:

  • the first implementation will be console based only
  • capabilities for web based user interaction is being thought off during implementation of the first step
  • steps to be implemented are #18721 (comment)
  • the tool will be developed isolated from owncloud core as self contained web and console project
  • it is expected to have the tool running in a webspace root in a subfolder 'updater'
  • all steps in it's first implementation will hold simple test out put to tell the user what to do 😉
  • interaction with the existing/installed/updated core is to be done via the occ command line tool (unclear in case of web based approach)

@VicDeo did I miss anything? THX

@cmonteroluque

This comment has been minimized.

Show comment
Hide comment
@cmonteroluque
Contributor

cmonteroluque commented Nov 10, 2015

@VicDeo

This comment has been minimized.

Show comment
Hide comment
@VicDeo

VicDeo Nov 10, 2015

Member

@DeepDiver1975 Looks like the description is complete

Member

VicDeo commented Nov 10, 2015

@DeepDiver1975 Looks like the description is complete

@VicDeo

This comment has been minimized.

Show comment
Hide comment
@VicDeo

VicDeo Nov 10, 2015

Member

Notes:

  • right now I simply add .md5 postfix to the download URL to obtain the package checksum location.
    But I really appreciate #20056 ;)
  • It would be great to have a certain non zero exit code for the case when occ was run by improper user https://github.com/owncloud/core/blob/master/console.php#L63
    Otherwise I would stop the process with something like
    'Can not parse occ response, it says: raw_output_here' because it is not possible to determine what exactly is wrong.
  • it's the right time to start preparing the filelists for each and every released versions
  • target PHP version?
Member

VicDeo commented Nov 10, 2015

Notes:

  • right now I simply add .md5 postfix to the download URL to obtain the package checksum location.
    But I really appreciate #20056 ;)
  • It would be great to have a certain non zero exit code for the case when occ was run by improper user https://github.com/owncloud/core/blob/master/console.php#L63
    Otherwise I would stop the process with something like
    'Can not parse occ response, it says: raw_output_here' because it is not possible to determine what exactly is wrong.
  • it's the right time to start preparing the filelists for each and every released versions
  • target PHP version?
@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Nov 17, 2015

Member

it's the right time to start preparing the filelists for each and every released versions

@LukasReschke this sounds like a reuse case of the signing mechanism

Member

DeepDiver1975 commented Nov 17, 2015

it's the right time to start preparing the filelists for each and every released versions

@LukasReschke this sounds like a reuse case of the signing mechanism

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Nov 17, 2015

Member

target PHP version?

5.4, 5.5, 5.6, 7

Member

DeepDiver1975 commented Nov 17, 2015

target PHP version?

5.4, 5.5, 5.6, 7

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Nov 17, 2015

Member

It would be great to have a certain non zero exit code for the case when occ was run by improper user https://github.com/owncloud/core/blob/master/console.php#L63
Otherwise I would stop the process with something like
'Can not parse occ response, it says: raw_output_here' because it is not possible to determine what exactly is wrong.

pull request welcome 😉

Member

DeepDiver1975 commented Nov 17, 2015

It would be great to have a certain non zero exit code for the case when occ was run by improper user https://github.com/owncloud/core/blob/master/console.php#L63
Otherwise I would stop the process with something like
'Can not parse occ response, it says: raw_output_here' because it is not possible to determine what exactly is wrong.

pull request welcome 😉

@PVince81 PVince81 added the overview label Nov 17, 2015

@VicDeo VicDeo referenced this issue Nov 17, 2015

Merged

Standalone updater. Initial commit #187

16 of 21 tasks complete
@VicDeo

This comment has been minimized.

Show comment
Hide comment
@VicDeo

VicDeo Nov 18, 2015

Member

@DeepDiver1975

@LukasReschke this sounds like a reuse case of the signing mechanism

I don't want to nag user for using something unsigned.
What I need is a complete list of files that I am allowed to remove.
And I should admit Wordpress has a perfect approach here: there are 3 directories and a dozen of files.
https://github.com/WordPress/WordPress

No multiple app directories, no movable 3rdparty or other sexy features leading to unpredictable results.

Member

VicDeo commented Nov 18, 2015

@DeepDiver1975

@LukasReschke this sounds like a reuse case of the signing mechanism

I don't want to nag user for using something unsigned.
What I need is a complete list of files that I am allowed to remove.
And I should admit Wordpress has a perfect approach here: there are 3 directories and a dozen of files.
https://github.com/WordPress/WordPress

No multiple app directories, no movable 3rdparty or other sexy features leading to unpredictable results.

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 Nov 18, 2015

Member

The full list of files which are contained in a release are part of the signing mechanism.
This should give a good starting point.

Member

DeepDiver1975 commented Nov 18, 2015

The full list of files which are contained in a release are part of the signing mechanism.
This should give a good starting point.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Feb 2, 2016

Member

Just thought about something: what about servers that have no access to the Internet, how could they benefit from the updater ? I think there should be an option to specify a custom URL for downloading the tarball. An admin can then either put "file:///" or the URL of a local intranet resource that will deliver the new tarball to the instance.

Even simpler would be to be able to pass the path of the tarball when running the upgrade command.

This would also make it possible to test upgrades more easily by building "dummy" tarballs for the next version instead of requiring a remote server to deliver test tarballs.

Member

PVince81 commented Feb 2, 2016

Just thought about something: what about servers that have no access to the Internet, how could they benefit from the updater ? I think there should be an option to specify a custom URL for downloading the tarball. An admin can then either put "file:///" or the URL of a local intranet resource that will deliver the new tarball to the instance.

Even simpler would be to be able to pass the path of the tarball when running the upgrade command.

This would also make it possible to test upgrades more easily by building "dummy" tarballs for the next version instead of requiring a remote server to deliver test tarballs.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Feb 11, 2016

Member

Basic version of the updater CLI + web UI was implemented.

@VicDeo can you post a link to the summary of what was done so far ?

We can then move this ticket to 9.1 and/or raise separate tickets for the remaining tasks.

Member

PVince81 commented Feb 11, 2016

Basic version of the updater CLI + web UI was implemented.

@VicDeo can you post a link to the summary of what was done so far ?

We can then move this ticket to 9.1 and/or raise separate tickets for the remaining tasks.

@PVince81 PVince81 modified the milestones: 9.1-next, 9.0-current Feb 23, 2016

@VicDeo VicDeo referenced this issue Feb 23, 2016

Closed

Some small design issues #250

5 of 5 tasks complete

@VicDeo VicDeo modified the milestones: 9.0-current, 9.1-next Feb 23, 2016

@cmonteroluque cmonteroluque modified the milestones: 9.0.1-next-maintenance, 9.0-current Feb 24, 2016

@cmonteroluque

This comment has been minimized.

Show comment
Hide comment
@cmonteroluque
Contributor

cmonteroluque commented Feb 24, 2016

aligned with
owncloud/updater#250

@cmonteroluque cmonteroluque modified the milestones: 9.1-current, 9.0.1-current-maintenance Mar 9, 2016

@MTRichards

This comment has been minimized.

Show comment
Hide comment
@MTRichards

MTRichards Mar 21, 2016

Contributor

owncloud/appstore-issues#4 Related as part of the update process.

Contributor

MTRichards commented Mar 21, 2016

owncloud/appstore-issues#4 Related as part of the update process.

@VicDeo

This comment has been minimized.

Show comment
Hide comment
@VicDeo

VicDeo Mar 21, 2016

Member

@MTRichards owncloud/appstore-issues#4 is addressed in owncloud/administration-internal#12 but I'm not sure what decision was made about the latter.
Looks like the discussion is stalled there

Member

VicDeo commented Mar 21, 2016

@MTRichards owncloud/appstore-issues#4 is addressed in owncloud/administration-internal#12 but I'm not sure what decision was made about the latter.
Looks like the discussion is stalled there

@MTRichards

This comment has been minimized.

Show comment
Hide comment
@MTRichards

MTRichards Apr 1, 2016

Contributor

Estimation: 6 weeks

cc:/ @PVince81

Contributor

MTRichards commented Apr 1, 2016

Estimation: 6 weeks

cc:/ @PVince81

@PVince81 PVince81 modified the milestones: 9.1, 9.1.1 Jul 18, 2016

@PVince81 PVince81 modified the milestones: 9.2, 9.1.1 Sep 21, 2016

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Sep 21, 2016

Member

moving topic to 9.2.

The separated updater has already improved a lot.

Member

PVince81 commented Sep 21, 2016

moving topic to 9.2.

The separated updater has already improved a lot.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Dec 8, 2016

Member

I think the only stuff in scope for 10.0 right now is making the updater app be able to upgrade over multiple versions. And that will only work starting with 10.0 once Doctrine migrations are used for any upgrades.

@DeepDiver1975 @VicDeo correct ?

Member

PVince81 commented Dec 8, 2016

I think the only stuff in scope for 10.0 right now is making the updater app be able to upgrade over multiple versions. And that will only work starting with 10.0 once Doctrine migrations are used for any upgrades.

@DeepDiver1975 @VicDeo correct ?

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Apr 11, 2017

Member

I'm moving this to backlog.

Will need to be rescheduled once we intend to work on the other remaining tasks. (there are lots of them, will likely need separate estimations and scheduling)

@DeepDiver1975 @butonic @hodyroff @pmaier1

Member

PVince81 commented Apr 11, 2017

I'm moving this to backlog.

Will need to be rescheduled once we intend to work on the other remaining tasks. (there are lots of them, will likely need separate estimations and scheduling)

@DeepDiver1975 @butonic @hodyroff @pmaier1

@PVince81 PVince81 modified the milestones: backlog, 10.0 Apr 11, 2017

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Jan 30, 2018

Member

removing p1 as this is not in focus currently

Member

PVince81 commented Jan 30, 2018

removing p1 as this is not in focus currently

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment