Installer to check that all files are uploaded correctly during install / upgrades #1097

robocoder opened this Issue Jan 1, 2010 · 19 comments

3 participants


Support issue: users who upload (eg recursive ftp put) an already extracted Piwik folder. Files missing or corrupt.


  • build script: generate file manifest with filesizes (and optional checksum)
  • installation/upgrade: iterate thru manifest and verify integrity of installation
Piwik Open Source Analytics member

Marking it as feature request. This is an interesting improvement (I've seen it in Gallery installer before), but can take many seconds to execute, and adds a bit of overhead on the release process. If someone wants to grab the task and build it, please let us know.


Change to build script:

  cp piwik/misc/How\ to\ install\ Piwik.html .
  cp piwik/misc/WebAppGallery/*.xml .

+ find . -printf '%s ' -exec md5sum {} \; >config/manifest
  zip -r piwik-$ piwik How\ to\ install\ Piwik.html  > /dev/null 2> /dev/null

Revised manifest generator for build script:

find piwik -type f -printf '%s ' -exec md5sum {} \; | fgrep -v '' | sed '1,$ s/\([0-9]*\) \([a-z0-9]*\) *piwik\/\(.*\)/\t\t"\3" => array("\1", "\2"),/; 1 s/^/<?php\nclass Manifest {\n\tstatic $files=array(\n/; $ s/$/\n\t);\n}/' > piwik/config/

Timings from an (ancient) Athlon 1.4GHz test box:

  • Generating the took 11.117 seconds on a clean (svn export) folder.
  • Loading, iterating through the files, and running file_exists/file_size/md5_file took 0.541 seconds.
Piwik Open Source Analytics member

For the UI part, I suggest that if the test fails, a red icon appears in the system checklist page. If it passes, it is just one more green item in the list (with required functions and librairies).


(In [1756]) refs #1097 - file integrity check (optional) added to Installation. A warning is given if or md5_file() don't exist.

Matt: please update the build script with the 'find' in comment:6 and then mark this ticket as fixed.


Attachment: Standalone integrity test script


Attachment: Sample (generated) (using piwik 0.5.4)


(In [1757]) refs #1097 - exclude development environments; translate exception messages

more php docs


(In [1758]) refs #1097 - reorganize code to better accommodate an integrity check after an upgrade (not yet implemented in CoreUpdater)

Piwik Open Source Analytics member

before marking as fixed, integrity check must happen during upgrades too. I will update the release script once this is code complete, nice work on this, this was a very frequent support issue.


(In [1764]) refs #1097 - check file integrity after updates; simplify some messages (e.g., md5 sum)


(In [1765]) refs #1097 - refactoring


I think this will be a really great feature.
Thanks a real lot !

Piwik Open Source Analytics member

Anthon, is the piwik code ready for this new feature? is the only thing left the one line update to the build script?


Matt: yes, it's ready.

While you're at it, please also add a line to the build script to remove unit test directories under the plugins folder.

  rm -rf piwik/tests/*
+ find piwik/plugins -name tests -type d -exec rm -rf {} \;
  mv README.txt piwik/tests/
Piwik Open Source Analytics member


I added the find to delete the tests plugins folders and I get:

find: piwik/plugins/UsersManager/tests: No such file or directory
find: piwik/plugins/UserSettings/tests: No such file or directory
find: piwik/plugins/Actions/tests: No such file or directory
find: piwik/plugins/LanguagesManager/tests: No such file or directory
find: piwik/plugins/UserCountry/tests: No such file or directory
find: piwik/plugins/AnonymizeIP/tests: No such file or directory
find: piwik/plugins/SitesManager/tests: No such file or directory

is that expected?

Piwik Open Source Analytics member

good stuff Anthon! closing


Yes, expected because 'find' is diving through folders while we remove it. We can suppress the messages using redirection, e.g, 2> /dev/null, or use xargs (if you don't mind the extra process being spawned), e.g.,

find piwik/plugins -name tests -type d -exec rm -rf {} \; 2>/dev/null

find piwik/plugins -name tests -type d -print0 | xargs -0 /bin/rm -rf
Piwik Open Source Analytics member

(In [1944]) Refs #1097
adding explanation when failure occurs, inviting user to reupload the files

@robocoder robocoder added this to the Piwik 0.5.5 milestone Jul 8, 2014
@mattab mattab was assigned by robocoder Jul 8, 2014
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment