Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Installer to check that all files are uploaded correctly during install / upgrades #1097

Closed
robocoder opened this Issue · 19 comments

3 participants

@robocoder
Collaborator

Support issue: users who upload (eg recursive ftp put) an already extracted Piwik folder. Files missing or corrupt.

Enhancement:

  • build script: generate file manifest with filesizes (and optional checksum)
  • installation/upgrade: iterate thru manifest and verify integrity of installation
@mattab
Owner

Marking it as feature request. This is an interesting improvement (I've seen it in Gallery installer before), but can take many seconds to execute, and adds a bit of overhead on the release process. If someone wants to grab the task and build it, please let us know.

@robocoder
Collaborator

Change to build script:

  cp piwik/misc/How\ to\ install\ Piwik.html .
  cp piwik/misc/WebAppGallery/*.xml .

+ find . -printf '%s ' -exec md5sum {} \; >config/manifest
+ 
  zip -r piwik-$VERSION.zip piwik How\ to\ install\ Piwik.html  > /dev/null 2> /dev/null
@robocoder
Collaborator

Revised manifest generator for build script:

find piwik -type f -printf '%s ' -exec md5sum {} \; | fgrep -v 'manifest.inc.php' | sed '1,$ s/\([0-9]*\) \([a-z0-9]*\) *piwik\/\(.*\)/\t\t"\3" => array("\1", "\2"),/; 1 s/^/<?php\nclass Manifest {\n\tstatic $files=array(\n/; $ s/$/\n\t);\n}/' > piwik/config/manifest.inc.php

Timings from an (ancient) Athlon 1.4GHz test box:

  • Generating the manifest.inc.php took 11.117 seconds on a clean (svn export) folder.
  • Loading manifest.inc.php, iterating through the files, and running file_exists/file_size/md5_file took 0.541 seconds.
@mattab
Owner

For the UI part, I suggest that if the test fails, a red icon appears in the system checklist page. If it passes, it is just one more green item in the list (with required functions and librairies).

@robocoder
Collaborator

(In [1756]) refs #1097 - file integrity check (optional) added to Installation. A warning is given if manifest.inc.php or md5_file() don't exist.

Matt: please update the build script with the 'find' in comment:6 and then mark this ticket as fixed.

@robocoder
Collaborator

Attachment: Standalone integrity test script
testintegrity.php

@robocoder
Collaborator

Attachment: Sample (generated) manifest.inc.php (using piwik 0.5.4)
manifest.inc.php

@robocoder
Collaborator

(In [1757]) refs #1097 - exclude development environments; translate exception messages

more php docs

@robocoder
Collaborator

(In [1758]) refs #1097 - reorganize code to better accommodate an integrity check after an upgrade (not yet implemented in CoreUpdater)

@mattab
Owner

before marking as fixed, integrity check must happen during upgrades too. I will update the release script once this is code complete, nice work on this, this was a very frequent support issue.

@robocoder
Collaborator

(In [1764]) refs #1097 - check file integrity after updates; simplify some messages (e.g., md5 sum)

@robocoder
Collaborator

(In [1765]) refs #1097 - refactoring

@anonymous-piwik-user

I think this will be a really great feature.
Thanks a real lot !

@mattab
Owner

Anthon, is the piwik code ready for this new feature? is the only thing left the one line update to the build script?

@robocoder
Collaborator

Matt: yes, it's ready.

While you're at it, please also add a line to the build script to remove unit test directories under the plugins folder.

  rm -rf piwik/tests/*
+ find piwik/plugins -name tests -type d -exec rm -rf {} \;
  mv README.txt piwik/tests/
@mattab
Owner

Anthon,

I added the find to delete the tests plugins folders and I get:

find: piwik/plugins/UsersManager/tests: No such file or directory
find: piwik/plugins/UserSettings/tests: No such file or directory
find: piwik/plugins/Actions/tests: No such file or directory
find: piwik/plugins/LanguagesManager/tests: No such file or directory
find: piwik/plugins/UserCountry/tests: No such file or directory
find: piwik/plugins/AnonymizeIP/tests: No such file or directory
find: piwik/plugins/SitesManager/tests: No such file or directory

is that expected?

@mattab
Owner

good stuff Anthon! closing

@robocoder
Collaborator

Yes, expected because 'find' is diving through folders while we remove it. We can suppress the messages using redirection, e.g, 2> /dev/null, or use xargs (if you don't mind the extra process being spawned), e.g.,

find piwik/plugins -name tests -type d -exec rm -rf {} \; 2>/dev/null

find piwik/plugins -name tests -type d -print0 | xargs -0 /bin/rm -rf
@mattab
Owner

(In [1944]) Refs #1097
adding explanation when failure occurs, inviting user to reupload the files

@robocoder robocoder added this to the Piwik 0.5.5 milestone
@mattab mattab was assigned by robocoder
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.