Filter Page URLs

[metmarkosaric]

"I have some pages that contain UUIDs that are specific to users. I don't care which date or UUID is called and want to aggregate them.

Example:
Before: /account/550e8400-e29b-11d4-a716-446655440000
After: /account/{uuid}

A simple search and replace functionality with regex would be great!"

UPDATE: This is now live: Specify a custom location to aggregate pages that contain identifiers.

[pelme]

This would be helpful from a privacy and analytics perspective for us.

[marcobusemann]

Same here. Would be nice to specify multiple patterns to aggregate page URLs. Currently having dynamic URL parts creates a mess in the statistic.

[aradabir007]

It is indeed creating a mess and completely unnecessary to show stats for those URLs. Would love to see this feature added anytime soon!

[jaimeiniesta]

This would be very useful to filter stats on different sections. For example, I'd like to see stats of the most visited pages of our Blog, in this case I would filter web pages starting with http://example.com/blog

[metmarkosaric]

yeah this is what i meant. i just want to search the data to give me more filtering options to analyse the traffic, i don't want to change anything in the data that is stored. cloudflare/ga/search console pretty much all have this same thing. click "manual filter", select if you want to include/exclude and then choose the metric and type what you want. i think the use case where you want to change the data stored is very rare compared to the one where you just want to analyse the data.

[captainrobbo]

Another approach occurred to me. Would it make more sense to use (abuse?) custom goals and properties for this? e.g. on pages relating to my competition, add

plausible('viewed_competition', {props: {competition: '/x/2021/EST/tallinn_marathon'}, language: 'ET'})

Sorry but I haven't really explored custom properties yet, so I don't know what kind of reports they produce. But it seems to me the website developer will know how to classify a page when they serve it, and this might be simpler than having 'watch lists' of custom URL patterns.

[ukutaht]

Yes @captainrobbo and actually @Vigasaurus is landing a PR on master next week that would help you to do this even without custom props: #750. It's not the ideal solution but using pageview goals for this means you don't need to redeploy the script or anything like that.

[metmarkosaric]

@jaimeiniesta your specific request about tracking blog posts is now possible using the new pageview goals. see how we do it on our own live demo and read how to do it yourself.

[jaimeiniesta]

Thanks! Just tried it and it's exactly what I was needing 💪

[captainrobbo]

I see the PR - can't comment on implementation as I am new to Elixir, but this looks absolutely perfect for our needs! We are happy to test aggressively as soon as it is there, and make our dashboard public.

…

On Fri, 26 Feb 2021 at 09:01, Uku Taht ***@***.***> wrote: Yes @captainrobbo <https://github.com/captainrobbo> and actually @Vigasaurus <https://github.com/Vigasaurus> is landing a PR on master next week that would help you to do this even without custom props: #750 <#750>. It's not the ideal solution but using pageview goals for this means you don't need to redeploy the script or anything like that. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#136 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACVQ5ZSN56SS624I5YZQ6D3TA5PNPANCNFSM4YELLYIQ> .

-- Andy Robinson Managing Director, ReportLab Wimbletech Zone 2 35 Wimbledon Hill Road London SW19 7NB, UK Tel +44-20-8191-7277

[captainrobbo]

A followup: I found it quite easy to use custom goals to pull out total hits on each competition we're hosting. Each of these subtotals represents hits on a whole forest of different pages in the last 30min, possibly on different subdomains and with different language prefixes. I'm sure the ability to process URLs as you ingest them will be great, but this is an easy way to achieve the same thing.

Plausible is awesome!

[ffont]

Any news on this? We're interested in that feature too. We can use Goals as a temporary workaround I guess, but without aggregation of pages the "Top pages" statistic is useless and consuming unnecessary resources.
Thanks a lot, so far Plausible has been great!

[scratchyone]

This would be very helpful for me too, and currently goals don't work correctly for this because I still end up seeing user's various uuids in the top pages list, which ends up de-anonymizing the data and gives me access to people's pages that are unlisted.

[tiblu]

When it comes to the initial use case of aggregating data, I FEEL like sending RAW data to Plausible and then Plausible UI enabling fancier aggregation functionality is the way to go. The RAW-er the data, the more into detail you can go processing it. IF you abstract the data, you lose level of detail. Never know what you want to query next. BUT, downside is that it makes Plausible UI more complicated and it is quite a development.

When it comes to privacy AND aggregation, I FEEL like ability to filter/modify the data that is being sent to Plausible is good for all parties. Implementation? Some kind of interceptor interface so User implemented function(method) gets called with data to be sent for modification.

[ukutaht]

Yeah I think we're in agreement. An ad-hoc query interface on top of the raw data would be most useful to all and we should build that first.

On-disk aggregation is necessary if you have personal data in URLs that you want to completely get rid of from your analytics.

Some kind of interceptor interface so User implemented function(method) gets called with data to be sent for modification.

Uh yeah I hadn't thought of this option as well. I assume you mean some sort of middleware on client-side? That would be super flexible.

[tiblu]

Yes, some kind of middleware on the client-side sounds good to me. I think its a low hanging fruit, relatively easy to implement and would benefit many. I have no specific ideas on the implementation tho.

[ukutaht]

would benefit many

Potentially. Most of our users are marketers not programmers. An implementation that doesn't require coding from the user would be much more popular, albeit less flexible. Luckily most transformations that people want to make can be expressed with simple page rules like /user/account/*. These kinds of filters/aggregations could be built into the backend so you don't have to code on the frontend.

[scratchyone]

Maybe both of these? Since being able to remove personal data on the client side is quite important but it’d also be very useful to be able to aggregate data with the UI for anything that isn’t PII

[ukutaht]

Yeah, I'm in favour of offering all of these while prioritizing solutions that are most accessible to the widest section of our users possible.

[mschoeffmann]

I agree with the exact same situation - especially for PWA sites.

Two cases:

1. as mentioned above:
   /account/550e8400-e29b-11d4-a716-446655440000 >>> /account/{uuid}
2. aggregate sub-pages/actions as well
   /account/11111111-1111-1111-1111-111111111111/action1 >>> /account/{uuid}/action1
/account/22222222-2222-2222-2222-222222222222/action1 >>> /account/{uuid}/action1
/account/22222222-2222-2222-2222-222222222222/action2 >>> /account/{uuid}/action2

This would be really helpful, because sometimes you want to count how often action1 happened, not action1 per uuid ...

At least masking the ID's before counting would help?

[appinteractive]

Any update on that?

[metmarkosaric]

have you tried our custom filter? it doesn't overwrite the URLs but it does allow you to group your pages using wildcards. see https://plausible.io/docs/filters-segments

[appinteractive]

have you tried our custom filter? it doesn't overwrite the URLs but it does allow you to group your pages using wildcards. see https://plausible.io/docs/filters-segments

Hey @metmarkosaric yeah I saw it on the screenshot but its not in the latest build (1.3.0 from April). So my question relates to the self hosted version I should say.

[thinkplaymake]

+1 on all of this, either post-capture aggregation, or the ability to track by a path you set in the path, i.e

/invite/hash1901892038910238920/view
/invite/hash2189382190389021921/view
/invite/hash1283902193021890382/view

I'd only want to track this as "/invite/view" (as an example), rather than hundreds of unique pageviews which are all effectively a single page view.

so i'd call something like:
plausible.track('/invite/view');

this might also solve for all the people asking for querystring support too, as they could call a custom page track.

[EmilePerron]

I would love to use this feature as well!

This could be implemented via some sort of client-side middleware as others have pointed out.

Actually, from my point of view, I feel like it could be done fairly simply.

I believe allowing for a location property/getter on window.plausible (and falling back on location.href when absent) would be enough to let developers add the implementation they need.

For example, if one wanted to simply specify a specific string as their location at load time, they could do something like this:

window.plausible.location = "/my/redacted/route";

If they needed a more complex implementation, they could define location as a getter instead:

Object.defineProperty(window.plausible, "location", {
  get: function() { 
    var url = window.location.href;

    // Custom logic that may or may not update `url` here

    return url;
  }
});

I'm not familiar with the Plausible codebase, but from a quick search it looks like this could be implemented by changing a single line in plausible.js at line 51:

payload.u = window.plausible.location || location.href

Obviously, this would also require that the feature be documented and that test cases be written, but it would still be a minor addition that is backwards compatible and quick to implement for both the Plausible team and the developers who would end up using it.

What do you all think?

I'd be happy to make a PR for this if the concept is approved.

[ukutaht]

@EmilePerron Thanks for your proposal!

I like your approach and it feels natural to me as well. Definitely a very flexible approach and much simpler than doing something on server-side.

Some thoughts that come to mind: We are totally obsessive about the size of the script. Anything that increases the size and is not used by default for everyone should be implemented as a script extension instead.

The only added weight in your proposal would be this line:

payload.u = window.plausible.location || location.href

However I don't think this would work if window.plausible.location is defined as a function, right? So it would have to check the type and decide whether to call the function, use the plain value or fall back to location.href.

The other thing I noticed is that your proposal expects window.plausible to be defined. For that you need the second line as shown here: https://plausible.io/docs/custom-event-goals. Not a huge deal but it feels sub-optimal if the user doesn't care about custom events, for example.

Thinking about these points I came up with a slightly modified API inspired by your proposal. If you want to provide a plain value (for ex. using the templating language from a server-generated page) you can use the standard configuration syntax:

<script data-domain="my-domain.com" data-location="https://my-domain.com/my-custom-location" src="https://plausible.io/js/plausible.custom-location.js"></script>

And if you want to generate it client-side:

<script>window.myLocationGetter = function() { return location.href + 'foo';  }</script>
<script data-domain="my-domain.com" data-get-location="myLocationeGetter" src="https://plausible.io/js/plausible.custom-location.js"></script>

The contract would be that data-get-location will provide the name of the function and if it exists of window, it will get executed and the return value used as the location value.

What do you think?

important We have to think about how this is going to play together with this request: #177. It's not obvious at all how to satisfy both with a clean API. I think we have to address both requests in one go in order not to paint ourselves into a corner. I didn't try to reconcile the two in this comment.

[EmilePerron]

@ukutaht Thanks for the speedy response!

My bad on the on the possibly missing window.plausible. I do like the data attribute configuration better: it's clean and simple.

And I get your obsession on file size: it's one of the things that makes Plausible Plausible, so better not break it!
Seing as this is an extra feature that many won't use, it makes sense to exclude it from the default script. So script extension it is.

Now, about compatibility with #177:
From what I can see, the URL that is sent by the tracker is the full URL, query string included.
The query string trimming is done server-side in PlausibleWeb.Api.ExternalController::create_event().

I think it's safe to assume that the implementation of the query parameter whitelist won't change this basic behaviour. It would likely be a script extension that adds a flag (or an array of whitelisted query parameters) to the payload, and the rest would still be handled server-side. Or, the whitelist would be handled in Plausible's dashboard, removing the need for a script extension. Either way, the basic behaviour described above stays the same: full URL is sent, and the server does the query filtering.

With that in mind, the only issue I can think of is that if the URL you specify via data-location/data-get-location contains a query string, it will always be trimmed just like URLs provided by the default behaviour. So there could be a discrepancy between your specified input and the data in Plausible.

However, seeing as query string trimming is already the default, this is not a new issue: it's still just #177. So when the query parameter whitelist (or other alternative) is eventually implemented, it will also work for the custom URLs introduced in this issue.

The only alternative I could think of would be to consider all query parameters in custom URLs as whitelisted, but this is likely to cause more issues than it solves, as it would shift the query-trimming responsibility to the users.

Let me know if I'm missing anything or if you have any more thoughts on the subject.

Cheers! 🤓

[EmilePerron]

@ukutaht Just pinging you to see if you have any more thoughts / oppositions to this. If not, I'll get to work on a PR in the upcoming days!

[ukutaht]

Hey,

I've thought about this a bit and I think your proposal sounds good. Compatibility with #177 isn't an issue because I think the query param allowlist is better implemented on the backend and integrated into our pipeline there.

[EmilePerron]

I've started to work on this in #1398 - interested parties can keep an eye on the development / progress there :)

[metmarkosaric]

This is now live! 🔥

You can now filter page URLs and specify custom locations. This is especially helpful to redact and aggregate multiple pages whose URLs contain identifiers specific to users. Learn how it works here: Specify a custom location to aggregate pages that contain identifiers.

Thanks to @EmilePerron for the contribution!

[ffont]

Hi everyone,

I've been trying to use this feature but I don't seem to be able to get it working. Should it be working with a self-hosted 1.4.4 plausible? I'm following what's written in this tutorial (https://plausible.io/docs/custom-locations) but still see the pageviews without the custom location.

Thanks!

[EmilePerron]

It looks like the 1.4.4 tag (which was created a month ago) does not include the custom locations, which were merged and launched ~ 3 months ago. Weird.

@ukutaht Are some features not included in the docker images for Plausible?

(perhaps it would be worth opening a separate issue/discussion for this).

[ukutaht]

@ukutaht Are some features not included in the docker images for Plausible?

Self-hosted is released twice a year or so. Please see the changelog for what is released and what is going in 1.5.

[ffont]

Thanks for the clarification @ukutaht ,
So considering 1.4.0 was released in October 2021, should we expect 1.5 around April?
Is it also a possibility to build our own image with the Dockerfile from master branch if we want to get the feature before the official release? Would it be advised?
Thanks again!

[ukutaht]

So considering 1.4.0 was released in October 2021, should we expect 1.5 around April?

Yes

Is it also a possibility to build our own image with the Dockerfile from master branch if we want to get the feature before the official release? Would it be advised?

It's possible but you're on your own :) we have made some config changes which aren't documented for self-hosters so likely it would break. Better to wait until we can cut a release with docs on how to install, configure and upgrade it.

[ffont]

Thanks @ukutaht, we'll wait for the next release!