Archive randomly fails to process and data show zeros for one day/period

[Zefling]

Hi,

I have strange results in week mode :
http://ikilote.net/Galeries/Autres/Divers/Piwik_Week.png
A week is not managed as it should, no problem for days and months

[mattab]

@Zefling Thanks for the report! This is surely an issue if Piwik fails to process data for some periods. Could you try the following: run the command from: http://piwik.org/faq/how-to/faq_59/

It may fix the issue for you, let me know?

[mattab]

The goal of this issue is to automate the process of fixing this error and also learning when it occurs.

Proposed steps:

• create a console command that detects the zero-visit archives and/or missing archives
• log as error the number of missing archives and which periods/segments
• create a console command that fixes the data for this period and triggers core:archive command with proper parameters.

[Zefling]

Sorry for the delay of my response, the [Debug] mode corrects this problem. It's not user-friendly, but it's work. Thank you.

[mgazdzik]

We have encountered prety similar problem on more than one instance. Here are what steps have been taken and what are symptoms:

• after first detection of such problem (days were ok, weeks and months were missing. After searching database, no 'done' flags were found for missing periods, except for last 3-4 weeks, which were intact). I dropped all archives and forced re-archiving for all periods all the way back till creation of first site. Right after archiving data looked ok, but after some time (couple days) it disappeared. Archive droping and re-archiving was done three times and issue persists.
• Similar problem was also located with 'returning visitor' segment archives. Recent data was fine, but there was huge gap in past. All archiving logs were ok, no errors or any other trace of error. After manually processing "returning visitor" archives (Using VisitorFrequency.get() ) all data poped up and now it's available.
• Similar situation occurs on more instances, also no sign of error in archiving logs.

[mattab]

If data disappears, somehow it means that we are deleting the data.

Notes to self

1. check whether we can guess a buggy codepath, from any of the functions in Piwik that delete archive data:
   1. deleteArchivesWithPeriodRange
   2. deleteArchiveIds
   3. deletePreviousArchiveStatus
   4. Db::deleteAllRows
2. maybe we should at least log more info about archives being deleted

[mattab]

Let's wait to reproduce this bug with 2.8.1 to confirm it is still here

maybe #5805 could have helped with this bug.

Note: on the instance the "Delete old report" and "delete old log" feature was disabled.

[mattab]

Moving this out of 2.9.0 until we have confirmation that this bug still occurs in 2.8.3 - please comment if this bug is reproduced in 2.8.3

[mattab]

AFAIK this is now working. If anyone still experiences "no data" gap in the stats even after re-processing the archives, please comment here! cc @quba @mgazdzik