New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document upgrade/installation procedures for Git checkouts (with composer.phar install --no-dev and git submodule update --init --recursive) #8717

Closed
nekohayo opened this Issue Sep 5, 2015 · 14 comments

Comments

Projects
None yet
4 participants
@nekohayo

nekohayo commented Sep 5, 2015

A little bird told me I could upgrade from one version straight to another without going through intermediate version upgrades. However this failed for me today:

$ php --version
    PHP 5.4.43 (cli) (built: Jul 20 2015 11:17:14)

$ mysql --version
    mysql  Ver 14.14 Distrib 5.5.42-37.1,
    for Linux (x86_64) using readline 5.1

$ git remote update
    […]
$ nano config/config.ini.php 
$ git checkout 2.14.3
    Checking out files: 100% (9089/9089), done.
    Previous HEAD position was 92bfb24...
    HEAD is now at 08f113d... 2.14.3

$ git log
    commit 08f113d9c4edae6182bfd458e63016f49a7dd024
    Date:   Wed Aug 5 21:05:54 2015 +0200
        2.14.3
    […]

$ ./console core:update
    Fatal error: Class 'Piwik\Common' not found in ./console on line 17

And yet I did not make any hacks/modifications to my setup:

$ git status
    # Not currently on any branch.
    # Untracked files:
    #
    #   error_log
    #   misc/cron/error_log
    #   themes/

So I reverted back to my previous version:

$ git checkout 2.0.3
    Previous HEAD position was 08f113d... 2.14.3
    HEAD is now at 92bfb24...
@sgiehl

This comment has been minimized.

Show comment
Hide comment
@sgiehl

sgiehl Sep 5, 2015

Member

When checking out from git, you might need to do a composer install to update the dependencies

Member

sgiehl commented Sep 5, 2015

When checking out from git, you might need to do a composer install to update the dependencies

@nekohayo

This comment has been minimized.

Show comment
Hide comment
@nekohayo

nekohayo Sep 5, 2015

Ah, you mean running php composer.phar install? That was not documented anywhere, so I would suggest adding a notice about that in http://piwik.org/docs/installation/ and http://piwik.org/docs/update/#the-manual-three-step-update

However, I've encountered a subproblem now:

$ php composer.phar install
    Warning: This development build of composer is over 30 days old.
    It is recommended to update it by running "composer.phar self-update" to get the latest version.

$ php composer.phar self-update
    Updating to version 4590012a57d829f18bfb819ce7041671ba2b0a67.
        Downloading: 100%         
    Use composer self-update --rollback to return to version 76833335152d17632e1afe277a9edbbb232d3a1f

$ php composer.phar install
    > Piwik\Composer\ScriptHandler::cleanXhprof
    Loading composer repositories with package information
    Installing dependencies (including require-dev) from lock file
    Nothing to install or update
    Generating autoload files
    > Piwik\Composer\ScriptHandler::buildXhprof
    Building xhprof...
    Fatal error: configure script failed! View tmp/xhprof-logs/configure.log for more info.

Contents of that log file:

    checking for grep that handles long lines and -e... /bin/grep
    checking for egrep... /bin/grep -E
    checking for a sed that does not truncate output... /bin/sed
    checking for cc... no
    checking for gcc... no
    configure: error: in `vendor/facebook/xhprof/extension':
    configure: error: no acceptable C compiler found in $PATH
    See `config.log' for more details.

Shared hosts (or most locked-down web servers) will not have C compilers lying around, for security reasons. Can we do something about that?

nekohayo commented Sep 5, 2015

Ah, you mean running php composer.phar install? That was not documented anywhere, so I would suggest adding a notice about that in http://piwik.org/docs/installation/ and http://piwik.org/docs/update/#the-manual-three-step-update

However, I've encountered a subproblem now:

$ php composer.phar install
    Warning: This development build of composer is over 30 days old.
    It is recommended to update it by running "composer.phar self-update" to get the latest version.

$ php composer.phar self-update
    Updating to version 4590012a57d829f18bfb819ce7041671ba2b0a67.
        Downloading: 100%         
    Use composer self-update --rollback to return to version 76833335152d17632e1afe277a9edbbb232d3a1f

$ php composer.phar install
    > Piwik\Composer\ScriptHandler::cleanXhprof
    Loading composer repositories with package information
    Installing dependencies (including require-dev) from lock file
    Nothing to install or update
    Generating autoload files
    > Piwik\Composer\ScriptHandler::buildXhprof
    Building xhprof...
    Fatal error: configure script failed! View tmp/xhprof-logs/configure.log for more info.

Contents of that log file:

    checking for grep that handles long lines and -e... /bin/grep
    checking for egrep... /bin/grep -E
    checking for a sed that does not truncate output... /bin/sed
    checking for cc... no
    checking for gcc... no
    configure: error: in `vendor/facebook/xhprof/extension':
    configure: error: no acceptable C compiler found in $PATH
    See `config.log' for more details.

Shared hosts (or most locked-down web servers) will not have C compilers lying around, for security reasons. Can we do something about that?

@sgiehl

This comment has been minimized.

Show comment
Hide comment
@sgiehl

sgiehl Sep 5, 2015

Member

Try php composer.phar install --no-dev, if you are not going to use this installation for piwik development.

Member

sgiehl commented Sep 5, 2015

Try php composer.phar install --no-dev, if you are not going to use this installation for piwik development.

@nekohayo

This comment has been minimized.

Show comment
Hide comment
@nekohayo

nekohayo Sep 5, 2015

Ah yes, that worked... but it seems there's still a bug left to investigate in the process:

$ php composer.phar install --no-dev
> Piwik\Composer\ScriptHandler::cleanXhprof
Loading composer repositories with package information
Installing dependencies from lock file
Nothing to install or update
Generating autoload files
> Piwik\Composer\ScriptHandler::buildXhprof

Running the DB upgrade script:

$ php console core:update

    *** Update ***
    Database Upgrade Required

    Your Piwik database is out-of-date, and must be upgraded before you can continue.
    Piwik database will be upgraded from version 2.0.3 to the new version 2.14.3.

    [a megaton of upgrades occur]
    Done. [61 / 751]

    ****************************************
      Piwik has been successfully updated!  
    ****************************************

$ php console core:update # again to be sure

    *** Update ***
    Database Upgrade Required

    Your Piwik database is out-of-date, and must be upgraded before you can continue.
    […]

A database upgrade is required. Execute update? (y/N) y

Starting the database upgrade process now. This may take a while, so please be patient.

    *** Update ***
    Database Upgrade Required

    Your Piwik database is out-of-date, and must be upgraded before you can continue.

    The following dimensions will be updated: log_link_visit_action.idaction_content_interaction, log_link_visit_action.idaction_content_name, log_link_visit_action.idaction_content_piece, log_link_visit_action.idaction_content_target.

    The database upgrade process may take a while, so please be patient.

  Executing ALTER TABLE `piwik_log_link_visit_action` ADD COLUMN `idaction_content_interaction` INTEGER(10) UNSIGNED DEFAULT NULL, ADD COLUMN `idaction_content_name` INTEGER(10) UNSIGNED DEFAULT NULL, ADD COLUMN `idaction_content_piece` INTEGER(10) UNSIGNED DEFAULT NULL, ADD COLUMN `idaction_content_target` INTEGER(10) UNSIGNED DEFAULT NULL... Done. [1 / 1]

****************************************
  Piwik has been successfully updated!  
****************************************

$ php console core:update # and again, since it turns out it needed to be run twice, as shown above

*************************************
  Everything is already up to date.  
*************************************

But now, the bug is this: when visiting the web interface, no data shows up, and I get this error shown in the Admin settings:

WARNING: plugins/CustomVariables/Archiver.php(19): Warning - require_once(libs/PiwikTracker/PiwikTracker.php): failed to open stream: No such file or directory

Again, not sure why this would happen, it seems like the file really is missing:

$ git status
    # Not currently on any branch.
    # Untracked files:
    #
    #   error_log
    #   misc/cron/error_log
    #   themes/

$ ls libs/        
HTML  MaxMindGeoIP  PiwikTracker  README.md  Zend  bower_components  javascript  jqplot  jquery  pChart  sparkline  upgradephp

$ ls libs/PiwikTracker/

nekohayo commented Sep 5, 2015

Ah yes, that worked... but it seems there's still a bug left to investigate in the process:

$ php composer.phar install --no-dev
> Piwik\Composer\ScriptHandler::cleanXhprof
Loading composer repositories with package information
Installing dependencies from lock file
Nothing to install or update
Generating autoload files
> Piwik\Composer\ScriptHandler::buildXhprof

Running the DB upgrade script:

$ php console core:update

    *** Update ***
    Database Upgrade Required

    Your Piwik database is out-of-date, and must be upgraded before you can continue.
    Piwik database will be upgraded from version 2.0.3 to the new version 2.14.3.

    [a megaton of upgrades occur]
    Done. [61 / 751]

    ****************************************
      Piwik has been successfully updated!  
    ****************************************

$ php console core:update # again to be sure

    *** Update ***
    Database Upgrade Required

    Your Piwik database is out-of-date, and must be upgraded before you can continue.
    […]

A database upgrade is required. Execute update? (y/N) y

Starting the database upgrade process now. This may take a while, so please be patient.

    *** Update ***
    Database Upgrade Required

    Your Piwik database is out-of-date, and must be upgraded before you can continue.

    The following dimensions will be updated: log_link_visit_action.idaction_content_interaction, log_link_visit_action.idaction_content_name, log_link_visit_action.idaction_content_piece, log_link_visit_action.idaction_content_target.

    The database upgrade process may take a while, so please be patient.

  Executing ALTER TABLE `piwik_log_link_visit_action` ADD COLUMN `idaction_content_interaction` INTEGER(10) UNSIGNED DEFAULT NULL, ADD COLUMN `idaction_content_name` INTEGER(10) UNSIGNED DEFAULT NULL, ADD COLUMN `idaction_content_piece` INTEGER(10) UNSIGNED DEFAULT NULL, ADD COLUMN `idaction_content_target` INTEGER(10) UNSIGNED DEFAULT NULL... Done. [1 / 1]

****************************************
  Piwik has been successfully updated!  
****************************************

$ php console core:update # and again, since it turns out it needed to be run twice, as shown above

*************************************
  Everything is already up to date.  
*************************************

But now, the bug is this: when visiting the web interface, no data shows up, and I get this error shown in the Admin settings:

WARNING: plugins/CustomVariables/Archiver.php(19): Warning - require_once(libs/PiwikTracker/PiwikTracker.php): failed to open stream: No such file or directory

Again, not sure why this would happen, it seems like the file really is missing:

$ git status
    # Not currently on any branch.
    # Untracked files:
    #
    #   error_log
    #   misc/cron/error_log
    #   themes/

$ ls libs/        
HTML  MaxMindGeoIP  PiwikTracker  README.md  Zend  bower_components  javascript  jqplot  jquery  pChart  sparkline  upgradephp

$ ls libs/PiwikTracker/
@sgiehl

This comment has been minimized.

Show comment
Hide comment
@sgiehl

sgiehl Sep 5, 2015

Member

Ok. Seems the submodule is missing. Try git submodule update --init --recursive

Member

sgiehl commented Sep 5, 2015

Ok. Seems the submodule is missing. Try git submodule update --init --recursive

@nekohayo

This comment has been minimized.

Show comment
Hide comment
@nekohayo

nekohayo Sep 5, 2015

That worked! Woohoo :)

I get the overall feeling that it'd be pretty nice to have a "If you're installing/upgrading from Git…" (sub)page in the docs to list all those gotchas... It might others some trouble!

nekohayo commented Sep 5, 2015

That worked! Woohoo :)

I get the overall feeling that it'd be pretty nice to have a "If you're installing/upgrading from Git…" (sub)page in the docs to list all those gotchas... It might others some trouble!

@nekohayo nekohayo changed the title from Upgrading directly from 2.0.3 to 2.14.3 fails to run console core:update script - fatal error: class 'Piwik\Common' not found to Document upgrade/installation procedures for Git checkouts (with composer.phar install --no-dev and git submodule update --init --recursive) Sep 5, 2015

@sgiehl

This comment has been minimized.

Show comment
Hide comment
@sgiehl

sgiehl Sep 5, 2015

Member
Member

sgiehl commented Sep 5, 2015

@nekohayo

This comment has been minimized.

Show comment
Hide comment
@nekohayo

nekohayo Sep 5, 2015

I understand it's not the most common/straightforward technique, but it's the "safest" for me as I run on a shared host with a sizeable database (~1.4 GB used by Piwik) so I need to be able to test & revert versions :)

nekohayo commented Sep 5, 2015

I understand it's not the most common/straightforward technique, but it's the "safest" for me as I run on a shared host with a sizeable database (~1.4 GB used by Piwik) so I need to be able to test & revert versions :)

@tsteur

This comment has been minimized.

Show comment
Hide comment
@tsteur

tsteur Sep 7, 2015

Member

Here's the FAQ re using Piwik via Git but it's not recommended for security reasons etc http://piwik.org/faq/how-to-install/faq_18271/

Member

tsteur commented Sep 7, 2015

Here's the FAQ re using Piwik via Git but it's not recommended for security reasons etc http://piwik.org/faq/how-to-install/faq_18271/

@tsteur

This comment has been minimized.

Show comment
Hide comment
@tsteur

tsteur Sep 7, 2015

Member

Closing as I think it's resolved, please reopen or comment if not

Member

tsteur commented Sep 7, 2015

Closing as I think it's resolved, please reopen or comment if not

@tsteur tsteur closed this Sep 7, 2015

@tsteur tsteur added this to the 2.15.0 milestone Sep 7, 2015

@tsteur tsteur added the answered label Sep 7, 2015

@tsteur tsteur self-assigned this Sep 7, 2015

@sgiehl

This comment has been minimized.

Show comment
Hide comment
@sgiehl

sgiehl Sep 7, 2015

Member

Maybe we should extend our git:pull command, as it only allows pulling the master... For real deployment maybe it would be better to checkout a specific version

Member

sgiehl commented Sep 7, 2015

Maybe we should extend our git:pull command, as it only allows pulling the master... For real deployment maybe it would be better to checkout a specific version

@tsteur

This comment has been minimized.

Show comment
Hide comment
@tsteur

tsteur Sep 7, 2015

Member

OK reopening... Not sure if our git:pull command can handle this but I presume so.

Member

tsteur commented Sep 7, 2015

OK reopening... Not sure if our git:pull command can handle this but I presume so.

@tsteur tsteur reopened this Sep 7, 2015

@tsteur tsteur removed the answered label Sep 7, 2015

@tsteur tsteur removed their assignment Sep 7, 2015

@tsteur tsteur removed this from the 2.15.0 milestone Sep 7, 2015

@nekohayo

This comment has been minimized.

Show comment
Hide comment
@nekohayo

nekohayo Sep 7, 2015

Ah that FAQ is interesting! It's missing a few bits that I learned in this bug report though.
Also, note that while I use git, I am not running master; I use your git release tags to go from one stable release to another, which I think should be just as safe (if not safer) than any other method, especially for someone who is at ease with "git checkout" and not ever using "git pull".

So instead of:

cd path/to/piwik
# Clone the Git repository 
git clone https://github.com/piwik/piwik.git .
# Setup composer libraries
curl -sS https://getcomposer.org/installer | php 
php composer.phar install
# Disable Piwik development mode
./console development:disable

Maybe it could say:

cd path/to/piwik
# Clone the Git repository 
git clone https://github.com/piwik/piwik.git .
# Optional (if you know what you're doing!): switch to a particular stable release instead of master (ex: 2.14.3):
git checkout 2.14.3

# Setup composer libraries
curl -sS https://getcomposer.org/installer | php 
php composer.phar install --no-dev
# Disable Piwik development mode
./console development:disable

Complemented by:

If you are upgrading by switching between stable release branches/tags, when doing an upgrade you can do something like this:

git checkout THE_RELEASE_TAG
git submodule update --init --recursive
php composer.phar install --no-dev

Hoping that's useful.

nekohayo commented Sep 7, 2015

Ah that FAQ is interesting! It's missing a few bits that I learned in this bug report though.
Also, note that while I use git, I am not running master; I use your git release tags to go from one stable release to another, which I think should be just as safe (if not safer) than any other method, especially for someone who is at ease with "git checkout" and not ever using "git pull".

So instead of:

cd path/to/piwik
# Clone the Git repository 
git clone https://github.com/piwik/piwik.git .
# Setup composer libraries
curl -sS https://getcomposer.org/installer | php 
php composer.phar install
# Disable Piwik development mode
./console development:disable

Maybe it could say:

cd path/to/piwik
# Clone the Git repository 
git clone https://github.com/piwik/piwik.git .
# Optional (if you know what you're doing!): switch to a particular stable release instead of master (ex: 2.14.3):
git checkout 2.14.3

# Setup composer libraries
curl -sS https://getcomposer.org/installer | php 
php composer.phar install --no-dev
# Disable Piwik development mode
./console development:disable

Complemented by:

If you are upgrading by switching between stable release branches/tags, when doing an upgrade you can do something like this:

git checkout THE_RELEASE_TAG
git submodule update --init --recursive
php composer.phar install --no-dev

Hoping that's useful.

@mattab

This comment has been minimized.

Show comment
Hide comment
@mattab

mattab Sep 20, 2015

Member

Thanks @nekohayo this was really useful!

I've updated the FAQ to explain the two use cases: using Bleeding edge master, or Deploying a specific release. Also added --no-dev to instructions and the git submodule init call. see http://piwik.org/faq/how-to-install/faq_18271/

Member

mattab commented Sep 20, 2015

Thanks @nekohayo this was really useful!

I've updated the FAQ to explain the two use cases: using Bleeding edge master, or Deploying a specific release. Also added --no-dev to instructions and the git submodule init call. see http://piwik.org/faq/how-to-install/faq_18271/

@mattab mattab closed this Sep 20, 2015

@mattab mattab added this to the 2.15.0 milestone Sep 20, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment