Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warn users about possible slow processing when viewing a report that has a lot of rows (for example Pages reports can be slow to render when showing >1K rows) #20164

Open
peteruniversum opened this issue Jan 6, 2023 · 7 comments
Labels
c: Performance For when we could improve the performance / speed of Matomo. c: Usability For issues that let users achieve a defined goal more effectively or efficiently. Enhancement For new feature suggestions that for example enhance Matomo's cabapilities..
Milestone

Comments

@peteruniversum
Copy link

E.g. https://anyofour3bigmatomoinstances.universum.com**/index.php?module=CoreHome&action=index&date=yesterday&period=day&idSite=2#?period=month&date=2022-12-01&idSite=2&category=General_Actions&subcategory=General_Pages**
-> Super slow.

Our customers want yearly reporting, which does absolutely not work in those reports (Pages/Page Titles). Application delivers 73Mbs of data to be rendered in the browser, which of course will not work.

-> Is this to be expected or might there be a config issue on our end?

System Check

`

Click to view System Check

Mandatory checks

PHP version >= 7.2.5:

✔ 8.1.13

PDO extension:

PDO\MYSQL extension:

MYSQLI extension:

Other required extensions:

✔ zlib ✔ json ✔ filter ✔ hash ✔ session

Required functions:

✔ debug_backtrace ✔ eval ✔ hash ✔ gzcompress ✔ gzuncompress ✔ pack

Required PHP configuration (php.ini):

✔ session.auto_start = 0 ✔ max_execution_time = 0 OR = -1 OR >= 30

Directories with write access:

✔ $DOC_ROOT/tmp ✔ $DOC_ROOT/tmp/assets ✔ $DOC_ROOT/tmp/cache ✔ $DOC_ROOT/tmp/climulti ✔ $DOC_ROOT/tmp/latest ✔ $DOC_ROOT/tmp/logs ✔ $DOC_ROOT/tmp/sessions ✔ $DOC_ROOT/tmp/tcpdf ✔ $DOC_ROOT/tmp/templates_c

Directories with write access for Tag Manager:

✔ $DOC_ROOT/js

Optional checks

Required Private Directories:

✔ All private directories are inaccessible from the internet.

Recommended Private Directories:

✔ All private directories are inaccessible from the internet.

File integrity:

⚠ Warning: File integrity check failed and reported some errors. You should fix this issue and then refresh this page until it shows no error.

Files were found in your Matomo, but we didn't expect them.
--> Please delete these files to prevent errors.

64-bit PHP Binary:

Tracker status:

Memory limit:

✔ 2048M

Time zone:

Open URL:

✔ curl

PageSpeed is turned off:

GD > 2.x + FreeType (graphics):

Other extensions:

✔ json ✔ libxml ✔ dom ✔ SimpleXML ✔ openssl

Other functions:

✔ shell_exec ✔ set_time_limit ✔ mail ✔ parse_ini_file ✔ glob ✔ gzopen ✔ md5_file

Filesystem:

Set up Cron - Managing processes via CLI:

✔ Ok

Last Successful Archiving Completion:

✔ The archiving process completed successfully 12:06:37 ago.

Database abilities:

✔ UTF8mb4 charset ✔ LOAD DATA INFILE ✔ CREATE TEMPORARY TABLES ✔ Changing transaction isolation level

Max Packet Size:

Forced SSL Connection:

Geolocation:

⚠ Warning: The default location provider determines the country visitors connect from based on their selected language. This is not very accurate, so install and use a geolocation database.

Update over HTTPS:

Writable JavaScript Tracker ("/matomo.js" & "/piwik.js"):

Informational results

Matomo Version:

4.13.0

Matomo Update History:

4.4.1,4.2.1,4.1.1,3.14.1,

Matomo Install Version:

Unknown - pre 3.8.

Latest Available Version:

4.13.0

Is Git Deployment:

0

PHP_OS:

Linux

PHP_BINARY:

/usr/sbin/php-fpm8.1

PHP SAPI:

fpm-fcgi

Timezone Version:

0.system

PHP Timezone:

UTC

PHP Time:

1672996225

PHP Datetime:

2023-01-06 09:10:25

PHP INI max_execution_time:

300s

PHP INI post_max_size:

10G

PHP INI max_input_vars:

1000

PHP INI zlib.output_compression:

Curl Version:

7.74.0, OpenSSL/1.1.1n

Suhosin Installed:

0

DB Prefix:

matomo_

DB Charset:

utf8mb4

DB Adapter:

PDO\MYSQL

MySQL Version:

10.5.18-MariaDB-0+deb11u1

Num Tables:

393

Browser Segment Archiving Enabled:

0

Development Mode Enabled:

0

Internet Enabled:

1

Multi Server Environment:

0

Auto Update Enabled:

1

Custom User Path:

0

Custom Include Path:

0

Release Channel:

latest_stable

Plugins Activated:

API, Actions, Annotations, BulkTracking, Contents, CoreAdminHome, CoreConsole, CoreHome, CorePluginsAdmin, CoreUpdater, CoreVisualizations, CoreVue, CustomDimensions, CustomJsTracker, CustomVariables 4.1.1, Dashboard, DevicePlugins, DevicesDetection, Diagnostics, Ecommerce, Events, FacebookPageWidgetByAmperage 1.0.7, Feedback, GeoIp2, Goals, Heartbeat, ImageGraph, Insights, Installation, Intl, IntranetMeasurable, LanguagesManager, Live, Login, MarketingCampaignsReporting 4.1.3, Marketplace, MobileMessaging, Monolog, Morpheus, MultiSites, Overlay, PagePerformance, PrivacyManager, ProfessionalServices, Proxy, Referrers, Resolution, RssWidget, SEO, ScheduledReports, SegmentEditor, SitesManager, TagManager, Tour, Transitions, TwoFactorAuth, UserCountry, UserCountryMap, UserId, UserLanguage, UsersManager, VisitFrequency, VisitTime, VisitorInterest, VisitsSummary, WebsiteMeasurable, Widgetize

Plugins Deactivated:

ArchiveSite 0.1.1, DBStats, MobileAppMeasurable, Provider 4.0.5

Plugins Invalid:

Server Info:

Apache

Had visits in last 1 day:

1

Had visits in last 3 days:

1

Had visits in last 5 days:

1

Archive Time Last Started:

2023-01-06 09:00:01

Archive Time Last Finished:

2023-01-05 21:03:48

User Agent:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36

Browser Language:

de,en,nl,it

Total Invalidation Count:

366

In Progress Invalidation Count:

0

Scheduled Invalidation Count:

366

Earliest invalidation ts_started:

Latest invalidation ts_started:

Earliest invalidation ts_invalidated:

2023-01-05 01:00:01

Latest invalidation ts_invalidated:

2023-01-06 04:02:43

Number of segment invalidations:

311

Number of plugin invalidations:

0

List of plugins being invalidated:

Anonymize Referrer:

Do Not Track enabled:

1

`
@peteruniversum peteruniversum added Potential Bug Something that might be a bug, but can't be reproduced (yet). To Triage An issue awaiting triage by a Matomo core team member labels Jan 6, 2023
@bx80
Copy link
Contributor

bx80 commented Jan 8, 2023

Hi @peteruniversum,

It should definitely be possible to view yearly page reports. If the browser is being sent report data then it sounds like the archiving has completed successfully and the report data has been generated. However 73MiB is obviously far too much data to be sent to the browser, it could be that there is a config setting cause this.

When Matomo generates reports it used a 'maximum rows' value to show the top x rows, after which all other rows are grouped together into a single row. For sites with a lot of pages this prevents reports being generated with millions of rows.

Could you check the values you have in config.ini.php for the following settings?

By default no report will return more than 50,000 rows:

archiving_ranking_query_row_limit = 50000

Actions (pages) reports will have a maximum of 500 rows:

datatable_archiving_maximum_rows_actions = 500

Each page in the report will have a maximum of 100 sub-pages shown (500 pages x 100 sub-pages = 50,000)

datatable_archiving_maximum_rows_subtable_actions = 100

@bx80 bx80 added the Waiting for user feedback Indicates the Matomo team is waiting for feedback from the author or other users. label Jan 8, 2023
@sgiehl sgiehl removed the To Triage An issue awaiting triage by a Matomo core team member label Feb 9, 2023
@ansgarbecker
Copy link

ansgarbecker commented Mar 8, 2023

I'm not sure if it's the same cause, but we are also experiencing slow report rendering, even with a browser crash. In E-Commerce > Products. Some computers seem to handle it without issue, some just show a loading indicator for a minute before the browser stops working. Chrome e.g. says "out of memory":
grafik

Edit: the 3 above mentioned config options are exactly on their defaults: archiving_ranking_query_row_limit = 50000, datatable_archiving_maximum_rows_actions = 500, datatable_archiving_maximum_rows_subtable_actions = 100

@bx80
Copy link
Contributor

bx80 commented Mar 9, 2023

Thanks for the extra info @ansgarbecker, Normally reports are paginated to show 5 - 500 rows at a time. Is this issue just occurring when showing all rows?
image

@ansgarbecker
Copy link

That's a good point. I was just trying out the "all" item, which now leads to high memory consumption in my Firefox. I had never touched the pager setting before, so previsouly it was "10", and I had no issue with that. My colleague probably had a higher setting here, so that could be the cause. I already asked her what she did but she could not remember. I'm attempting to find out more and will report back here.

@ansgarbecker
Copy link

I just got a reply from my colleague, seems she indeed used the "all" pager. You can likely close the issue. Icing on the cake would be to add a warning about probably slow processing when user wants to use more than 1000 (?) rows per page. For me personally I would have known I clicked there so I would be aware there is probably an issue with loading data. But there seem to be users clicking wildly somewhere and afterwards forgetting what they did...

@mattab
Copy link
Member

mattab commented Mar 12, 2023

Thanks @ansgarbecker for creating the issue and your great suggestion to "add a warning about probably slow processing when user wants to use more than 1000 (?) rows per page."

We'll consider doing this indeed 👍

@mattab mattab changed the title Report Behaviour/Pages super slow over longer periods of time Warn users about possible slow processing when viewing a report that has a lot of rows (for example Pages reports can be slow to render when showing >1K rows) Mar 12, 2023
@mattab mattab added this to the 5.4.0 milestone Mar 12, 2023
@ansgarbecker
Copy link

Another solution would be to remove the stored filter_limit setting from the server side user preferences in the option table, into a session value. So the user would just have to remove cookies to solve this issue.

grafik

Or just not store the -1 (= "all") setting at all, but that would probably annoy the user.

@sgiehl sgiehl added Enhancement For new feature suggestions that for example enhance Matomo's cabapilities.. c: Usability For issues that let users achieve a defined goal more effectively or efficiently. and removed Waiting for user feedback Indicates the Matomo team is waiting for feedback from the author or other users. Potential Bug Something that might be a bug, but can't be reproduced (yet). labels Oct 9, 2023
@mattab mattab added c: Performance For when we could improve the performance / speed of Matomo. c: Data Integrity & Accuracy and removed c: Data Integrity & Accuracy labels Dec 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Performance For when we could improve the performance / speed of Matomo. c: Usability For issues that let users achieve a defined goal more effectively or efficiently. Enhancement For new feature suggestions that for example enhance Matomo's cabapilities..
Projects
None yet
Development

No branches or pull requests

5 participants