Reports over 1 month are empty

[KarellenX]

Hello,

Using the Team Kodi account, I cannot seem to generate a report greater than 1 month. When I attempt to generate a report for 3 months I end up with one of the following screens.

Is this a known issue?

[arp242]

Looks like the SQL query is timing out; so the dashboard gets rendered without data. Not sure why, because Kodi isn't even the largest site and another one is working fine 🤔

I'll take a look later this evening or tomorrow to fix it.

[KarellenX]

Thanks

[arp242]

The issue is that Kodi has quite a lot of different paths (e.g, /page1, /page2, about 40,000 more) and this causes some performance issues. There's no problem handling many different pageviews as such, but it's fairly slow if it's spread out over so many paths.

I have a database update ready which improves things a little bit, but not enough that I'm happy with it. I delayed updating the server as it requires a few hours and downtime to migrate stuff to the new table format, and I've been waiting until PostgreSQL 13 is available on Alpine Linux, since updating that also requires downtime. I'll do that sometime soon (hopefully Sunday, unless some problems come up with testing). After that, I can improve on the update further.

In the meanwhile, I increased the timeout for Kodi from 5 seconds to 30 seconds, which should be enough. I can't test this as I can't log in, so let me know if it doesn't work or if you're still having problems. This should at least give you access to the data while I'm improving this over the next week or so.

[KarellenX]

The issue is that Kodi has quite a lot of different paths (e.g, /page1, /page2, about 40,000 more)

I have no idea where 40,000 pages are coming from. Our wiki is not that big. Even though the image shows almost 14,000 pages, a lot of those I would not expect to be indexed (but maybe they are, I don't know). Even so 14,000 (10,516+3,221) is a long way off 40,000.

In the meanwhile, I increased the timeout for Kodi from 5 seconds to 30 seconds, which should be enough. I can't test this as I can't log in, so let me know if it doesn't work or if you're still having problems.

Yes it is working now, thanks. No issues apart from a bit slow, but it is only a few seconds, not tens of seconds so not really an issue. I only run the quarterly report a couple times a quarter.

I'll keep an eye out for the changes. Thanks!!

[arp242]

There's a lot of other pagevies other than the actual content pages:

• Search: /index.php?fulltext=1&profile=advanced&search=dosbox&title=Special%3ASearch – there's a few variations of this depending on which parameters were added, and this seems to be the main bulk of the pageviews.

• Diff views: /index.php?diff=159300&oldid=159298&title=Add-on%3AGME_Audio_Decoder and /index.php?curid=1014&diff=219918&oldid=212678&title=Apple_remote – there are quite a few of these as well.

• Some other UI actions such as delete, edit, talk pages, and history.

Also a lot of pages get loaded with __cf_chl_jschl_tk__ and __cf_chl_captcha_tk__ parameters, which are a CloudFlare thing, and are automatically removed as of yesterday (but I haven't removed it from existing pages yet).

All of this combined causes the number of unique paths to be much higher than the actual pages you have (it's actually higher than 40k: 58,578).

I don't know which data you're actually interested in, but if I load just the pages starting with /view/% and exclude those with query parameters then there are 8,643 left, which includes all the content pages, talk pages, various Special: pages e.g. (Special:Contributions, Special:WhatLinksHere), templates.

[KarellenX]

Thanks for that information. I had no idea that users were looking at diffs or that each search query was logged as a page.

At this time, the pages in /view/ is all I need to track what users are reading, and of those pages which ones need to be updated to satisfy the users needs. So go ahead and exclude anything with the query parameter. The remaining 8,643 pages will be more than enough for my needs.

Thanks!!

[KarellenX]

Is there any way that I can search for a particular page?

When I am deciding on whether to delete a page from the wiki, I need to check how popular the page is first as maybe I may need to replace it with a redirect, rather than a delete.

It would be easier to search for, say, https://kodi.wiki/view/Backup rather than using my existing method of "keep loading results until I find the page". (I usually give up after loading the first 1000 pages)

[arp242]

At this time, the pages in /view/ is all I need to track what users are reading, and of those pages which ones need to be updated to satisfy the users needs. So go ahead and exclude anything with the query parameter. The remaining 8,643 pages will be more than enough for my needs.

This isn't something I can really exclude on the backend side; although I can delete the existing pages if you want. If you don't want to send them, then changing the integration to something like this would be the easiest:

<script>
window.goatcounter = {
	path: function() {
		// Just the path name, without query parameters.
		var p = location.pathname || '/'

		// Send only content pages starting with /view/
		if (p.indexOf('/view/') !== 0)
			return null
		return p
	}
}
</script>
<script data-goatcounter="https://teamkodi.goatcounter.com/count" async src="//gc.zgo.at/count.js"></script>

Is there any way that I can search for a particular page?

There's the "Filter paths" on the top-right; I think this is what you're looking for?

[KarellenX]

This isn't something I can really exclude on the backend side; although I can delete the existing pages if you want. If you don't want to send them, then changing the integration to something like this would be the easiest:

Ok, whatever you need to do. From your previous post you made it sound as if it was possible

but if I load just the pages starting with /view/% and exclude those with query parameters then there are 8,643 left,

---

There's the "Filter paths" on the top-right; I think this is what you're looking for?

Oops. Exactly what I was looking for. Sorry, was a bit blind :)

[arp242]

I made a few changes in the last two weeks which should make everything a bit faster; it's still on-going work, especially for cases where you have tens of thousands of different paths, but it should be better.

Another change is that you can now configure which parts to display on the dashboard, and disabling parts you don't use will speed things up a bit. In particular, I don't think you're using the "Totals" and that chart is quite slow to render, so disabling that will speed everything up.

I'll close this for now, but please let me know if you're still having issues and I'll look in to it.

[KarellenX]

Thanks @arp242

I'll adjust my settings. I mentioned previously I don't need charts (and they are more of an annoyance than anything else) but I don't see a setting to disable charts. Will that be coming in future?

I'll let you know if there are any issues.

[arp242]

I don't see a setting to disable charts. Will that be coming in future?

I'm currently working on a feature which allows you to set a default view so that this loads by default (#258):

Then you can select "text view" and "quarter", click "save default view", and that will load when you open "teamkodi.goatcounter.com", instead of the hard-coded defaults.

Mostly works already, just need to sort out some details (which is always 80% of the work, innit? 🙃) Maybe I'll finish it tonight or otherwise tomorrow.

[arp242]

And this is now live too.

[KarellenX]

I tried the new "Save default view" setting, but I receive the following error message. Also, I don't see a setting to completely remove the graphs. Did I misunderstand you were going to make removing them a setting?

[arp242]

Hm, it works for all my sites; not sure why it doesn't for you; I'll have a look in the morning.

You mean removing the text graphs? I wasn't really planning on it – I thought you meant setting the default so that it would render as text. Why do you want them removed? I understand you're not using them, but are they causing any issues?

[KarellenX]

No, not causing any issue for me. A few times you have stated that generating graphs can slow down the page, so just letting you know I don't use them, so you can remove them. But it is no issue if they stay.

[arp242]

Oh, I think I know why it doesn't save: it's because I manually set the page limit to 100, above the normally allowed amount; when it saves the settings it tries to validate the data and that fails as the page limit is above 25. It's just a problem for you. Ehm, need to think about a solution for that – I don't really want to set the upper limit to 100 as it causes performance issues on slower devices, but OTOH I also don't want to prevent people from loading a bunch of data if they really want to. I'll need to make some stuff with a warning, or something.

[arp242]

A few times you have stated that generating graphs can slow down the page

That's just for the "normal" view, not the "text" view; on the text view they're pretty much free performance-wise.

[arp242]

Oh, I think I know why it doesn't save: it's because I manually set the page limit to 100, above the normally allowed amount; when it saves the settings it tries to validate the data and that fails as the page limit is above 25. It's just a problem for you. Ehm, need to think about a solution for that – I don't really want to set the upper limit to 100 as it causes performance issues on slower devices, but OTOH I also don't want to prevent people from loading a bunch of data if they really want to. I'll need to make some stuff with a warning, or something.

This is fixed too now; you can add up to 100 pages, but above 25 you'll get a warning on the settings screen.

[KarellenX]

That works now. No warning was displayed for the 100 page list, though.

Thank you :)