Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Installer to check that all files are uploaded correctly during install / upgrades #1097

Closed
robocoder opened this Issue · 19 comments

3 participants

Anthon Pang Matthieu Aubry Anonymous Piwik user
Anthon Pang
Collaborator

Support issue: users who upload (eg recursive ftp put) an already extracted Piwik folder. Files missing or corrupt.

Enhancement:

  • build script: generate file manifest with filesizes (and optional checksum)
  • installation/upgrade: iterate thru manifest and verify integrity of installation
Matthieu Aubry
Owner

Marking it as feature request. This is an interesting improvement (I've seen it in Gallery installer before), but can take many seconds to execute, and adds a bit of overhead on the release process. If someone wants to grab the task and build it, please let us know.

Anthon Pang
Collaborator

Change to build script:

  cp piwik/misc/How\ to\ install\ Piwik.html .
  cp piwik/misc/WebAppGallery/*.xml .

+ find . -printf '%s ' -exec md5sum {} \; >config/manifest
+ 
  zip -r piwik-$VERSION.zip piwik How\ to\ install\ Piwik.html  > /dev/null 2> /dev/null
Anthon Pang
Collaborator

Revised manifest generator for build script:

find piwik -type f -printf '%s ' -exec md5sum {} \; | fgrep -v 'manifest.inc.php' | sed '1,$ s/\([0-9]*\) \([a-z0-9]*\) *piwik\/\(.*\)/\t\t"\3" => array("\1", "\2"),/; 1 s/^/<?php\nclass Manifest {\n\tstatic $files=array(\n/; $ s/$/\n\t);\n}/' > piwik/config/manifest.inc.php

Timings from an (ancient) Athlon 1.4GHz test box:

  • Generating the manifest.inc.php took 11.117 seconds on a clean (svn export) folder.
  • Loading manifest.inc.php, iterating through the files, and running file_exists/file_size/md5_file took 0.541 seconds.
Matthieu Aubry
Owner

For the UI part, I suggest that if the test fails, a red icon appears in the system checklist page. If it passes, it is just one more green item in the list (with required functions and librairies).

Anthon Pang
Collaborator

(In [1756]) refs #1097 - file integrity check (optional) added to Installation. A warning is given if manifest.inc.php or md5_file() don't exist.

Matt: please update the build script with the 'find' in comment:6 and then mark this ticket as fixed.

Anthon Pang
Collaborator

Attachment: Standalone integrity test script
testintegrity.php

Anthon Pang
Collaborator

Attachment: Sample (generated) manifest.inc.php (using piwik 0.5.4)
manifest.inc.php

Anthon Pang
Collaborator

(In [1757]) refs #1097 - exclude development environments; translate exception messages

more php docs

Anthon Pang
Collaborator

(In [1758]) refs #1097 - reorganize code to better accommodate an integrity check after an upgrade (not yet implemented in CoreUpdater)

Matthieu Aubry
Owner

before marking as fixed, integrity check must happen during upgrades too. I will update the release script once this is code complete, nice work on this, this was a very frequent support issue.

Anthon Pang
Collaborator

(In [1764]) refs #1097 - check file integrity after updates; simplify some messages (e.g., md5 sum)

Anthon Pang
Collaborator

(In [1765]) refs #1097 - refactoring

Anonymous Piwik user

I think this will be a really great feature.
Thanks a real lot !

Matthieu Aubry
Owner

Anthon, is the piwik code ready for this new feature? is the only thing left the one line update to the build script?

Anthon Pang
Collaborator

Matt: yes, it's ready.

While you're at it, please also add a line to the build script to remove unit test directories under the plugins folder.

  rm -rf piwik/tests/*
+ find piwik/plugins -name tests -type d -exec rm -rf {} \;
  mv README.txt piwik/tests/
Matthieu Aubry
Owner

Anthon,

I added the find to delete the tests plugins folders and I get:

find: piwik/plugins/UsersManager/tests: No such file or directory
find: piwik/plugins/UserSettings/tests: No such file or directory
find: piwik/plugins/Actions/tests: No such file or directory
find: piwik/plugins/LanguagesManager/tests: No such file or directory
find: piwik/plugins/UserCountry/tests: No such file or directory
find: piwik/plugins/AnonymizeIP/tests: No such file or directory
find: piwik/plugins/SitesManager/tests: No such file or directory

is that expected?

Matthieu Aubry
Owner

good stuff Anthon! closing

Anthon Pang
Collaborator

Yes, expected because 'find' is diving through folders while we remove it. We can suppress the messages using redirection, e.g, 2> /dev/null, or use xargs (if you don't mind the extra process being spawned), e.g.,

find piwik/plugins -name tests -type d -exec rm -rf {} \; 2>/dev/null

find piwik/plugins -name tests -type d -print0 | xargs -0 /bin/rm -rf
Matthieu Aubry
Owner

(In [1944]) Refs #1097
adding explanation when failure occurs, inviting user to reupload the files

Anthon Pang robocoder added this to the Piwik 0.5.5 milestone
Matthieu Aubry mattab was assigned by robocoder
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.