Stats Archive can miss records that arrive at the CMS a long time after they are recorded #2341
Comments
dasgarner
added a commit
to dasgarner/xibo-cms
that referenced
this issue
Oct 9, 2020
…on date. Also no reason why the interface returns an array. xibosignage/xibo#2341
dasgarner
added a commit
to xibosignage/xibo-cms
that referenced
this issue
Oct 22, 2020
* Fix issue with MongoDbTimeSeriesStore so that we don't count if we don't need to (if we aren't requesting a page) xibosignage/xibo#2339 * See if we need to reconnect to MySQL after a long stats archive operation in Mongo (during which MySQL is essentially idle). xibosignage/xibo#2339 * Further improvements to archiver for larger datasets xibosignage/xibo#2339 * Streamline getting the earliest date/time xibosignage/xibo#2339 * Earliest date/time should be based on start date rather than collection date. Also no reason why the interface returns an array. xibosignage/xibo#2341 * Add logging to StatsArchiveTask to track down performance issues. xibosignage/xibo#2339 * Don't do an extra sort unless we're paging. xibosignage/xibo#2339 * Extra logging xibosignage/xibo#2339 * Extra logging xibosignage/xibo#2339 * Add the ability to console log. xibosignage/xibo#2339 * Move MySQL connection refresh forwards to immediately after the CSV processing loop. xibosignage/xibo#2339 * Try to make MySQL logging resilient to a wait timeout (clear stored statement and try again) xibosignage/xibo#2339 * Move default connect mysql refresh back after ZIP, and catch type of exception. xibosignage/xibo#2339 * Catch error exception from MySQL (which isn't captured by PDO) xibosignage/xibo#2339 * commit after library add, and refresh connection after delete xibosignage/xibo#2339
dasgarner
added a commit
to dasgarner/xibo-cms
that referenced
this issue
Oct 22, 2020
* Fix issue with MongoDbTimeSeriesStore so that we don't count if we don't need to (if we aren't requesting a page) xibosignage/xibo#2339 * See if we need to reconnect to MySQL after a long stats archive operation in Mongo (during which MySQL is essentially idle). xibosignage/xibo#2339 * Further improvements to archiver for larger datasets xibosignage/xibo#2339 * Streamline getting the earliest date/time xibosignage/xibo#2339 * Earliest date/time should be based on start date rather than collection date. Also no reason why the interface returns an array. xibosignage/xibo#2341 * Add logging to StatsArchiveTask to track down performance issues. xibosignage/xibo#2339 * Don't do an extra sort unless we're paging. xibosignage/xibo#2339 * Extra logging xibosignage/xibo#2339 * Extra logging xibosignage/xibo#2339 * Add the ability to console log. xibosignage/xibo#2339 * Move MySQL connection refresh forwards to immediately after the CSV processing loop. xibosignage/xibo#2339 * Try to make MySQL logging resilient to a wait timeout (clear stored statement and try again) xibosignage/xibo#2339 * Move default connect mysql refresh back after ZIP, and catch type of exception. xibosignage/xibo#2339 * Catch error exception from MySQL (which isn't captured by PDO) xibosignage/xibo#2339 * commit after library add, and refresh connection after delete xibosignage/xibo#2339
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The first thing we do is get the earliest date in the table by statDate (which is the date/time we recorded the stat).
We use this date to decide the first period to archive, and then we archive on start/end date.
We expect the statDate to be after the start date always, ergo we will always chop off some records from the archive (if they are later than the period).
The text was updated successfully, but these errors were encountered: