Add more tracking features

[blaggacao]

Page View (frappe.website.doctype.web_page_view.web_page_view.make_view_log) is already great and a lot of things can be built on top of it through (mere) reports, such as:

• Funnels
• User Flows
• AB Test Performance
• Cohorts

And with more integration with the rest of ERPNext, such as revenue data, even:

• Multichannel Conversion Attribution

Others would require more analytical record, such as:

• Form Analytics — Form Analytics Doctype gathering statistics on Frappe Forms
• Media Analytics — Media Analytics Doctype gathering statistics on HTML5 video or audio tags

Other than that, typical tracking events, in addition to Page View, are CheckoutItems, Checkout (i.e. typically the definition of a "conversion"). See for example this matomo help article for a good enough hands-on understanding of tracking events.

These events are also very relevant for the so called Conversion API s of this world (Meta, TikTok, Google, Microsoft) which they use to increase the effectiveness of marketing campaigns via AI-backed continuous improvement models.

A Page Analytics Event could catch these whenever the user defines something like this on the respective action (e.g. click event of a button):

// entire thing automatically seeded in the builder UI?
frappe.call("frappe.website.doctype.web_page_event.web_page_event.make_event_log", {
        name: this.analytics_event_name, // automatically seeded in the builder UI?
	referrer: document.referrer,
	browser: browser.name,
	version: browser.version,
	user_tz: Intl.DateTimeFormat().resolvedOptions().timeZone,
	source: query_params.source,
	medium: query_params.medium,
	campaign: query_params.campaign,
	visitor_id: result.visitorId,
	... // maybe more
})

On the backend, a combined Analytics Settings could then manage the mappings of the available event data to the various Conversions API (Meta, TikTok, etc), as well as the credentials to push these data points to the campaign platform for AI-based campaign self-improvement.

I think that's a viable, out-of-current-builder-capability perspective and gameplan on how to bring this feature on-par quickly and cheaply with some of the industry leaders (such as Matomo or Google Analytics 4 / Google Tag Manager with server-side tagging). All the pieces are already in place, just needs connecting them.

cc @rmehta

[blaggacao]

List of Event Triggers available in my Matomo Instance's Tag Manager:

Clicks

• All Download Clicks (left, middle or right click)
• All Element Clicks (left, middle or right click)
• All Link Clicks (left, midlle or right click)

Pageviews

• DOM Ready
• PageView
• Window Loaded

User engagement

• Element Visibility (when a specific element becomes visible)
• Form Submit (when any form is sumbitted)
• Full screen (when window goes or leaves full screen)
• History Change (when the current url changes)
• Scroll Reach (when a certain scroll position is reached)
• User Interaction (configurable touche, move, scroll or mouse click event)
• Window Leave ("about to leave", when user mouse moves outside the window)
• Window Unload (just before the window is closed or user navigates to different page)

Others

• Custom Event
• JS Error (when a js error happens)
• Timer (on a specific interval)

---

I imagine most of these could be thought of check-boxes in the builder UI on the respective components, something like "Enable Tracking".

[blaggacao]

Here is a real live Matomo Tag Manager script

https://gist.github.com/blaggacao/fc8d33f7ba46ba4fa30e533146a114dd

I think the beauty of having this integrated into builder is that a lot of the bolt-on trigger logic won't be necessary because those triggers can just be included into the page source where ever it is desired. Server side push to Conversion APIs and others also makes the overhead of having external tool's client side procsessing obsolete. That is: instead of installing a Facebook Pixel tag, one would simply feed the data kraken server-to-server, directly from the back end (just as modern Google Tag Manager does in a cookieless [third-party] 2024 world).

What's a bit more interesting are the variables and internal data structures used. They might give us some clues as to what is UI configurable in matomo tag manager and what is actually tracked:

In a bit I'll be able to assess how Facebook Pixel plays into this to extract the exact correlation mechanism on how incoming platforms would express a "Click ID" or capture a "User Identity" [Possibly the finger-printed visitor ID is actually enough for most conversion APIs if enriched with some back-end user data such as email, address or IP]) 🤷

        window.MatomoTagManager.addContainer({
            "id": "fff...",
            "idsite": 1,
            "versionName": "",
            "revision": 0,
            "environment": "live",
            "tags": [{
                "id": 1,
                "type": "Matomo",
                "name": "fff......",
                "parameters": {
                    "matomoConfig": {
                        "name": "Matomo Configuration",
                        "type": "MatomoConfiguration",
                        "lookUpTable": [],
                        "defaultValue": "",
                        "parameters": {
                            "matomoUrl": "https:\/\/foobar.matomo.cloud",
                            "idSite": 1,
                            "enableLinkTracking": true,
                            "enableFormAnalytics": true,
                            "enableMediaAnalytics": true,
                            "enableFileTracking": false,
                            "enableCrossDomainLinking": false,
                            "crossDomainLinkingTimeout": "180",
                            "enableDoNotTrack": false,
                            "disablePerformanceTracking": false,
                            "enableJSErrorTracking": false,
                            "enableHeartBeatTimer": false,
                            "heartBeatTime": "15",
                            "trackAllContentImpressions": false,
                            "trackVisibleContentImpressions": false,
                            "disableCookies": false,
                            "requireConsent": false,
                            "requireCookieConsent": false,
                            "customCookieTimeOutEnable": false,
                            "customCookieTimeOut": "393",
                            "referralCookieTimeOut": "182",
                            "sessionCookieTimeOut": "30",
                            "setSecureCookie": false,
                            "cookieDomain": "",
                            "cookieNamePrefix": "_pk_",
                            "cookiePath": "",
                            "cookieSameSite": "Lax",
                            "disableBrowserFeatureDetection": false,
                            "domains": [],
                            "alwaysUseSendBeacon": false,
                            "disableAlwaysUseSendBeacon": false,
                            "userId": "",
                            "customDimensions": [],
                            "registerAsDefaultTracker": true,
                            "bundleTracker": true,
                            "jsEndpoint": "matomo.js",
                            "jsEndpointCustom": "custom.js",
                            "trackingEndpoint": "matomo.php",
                            "trackingEndpointCustom": "custom.php",
                            "appendToTrackingUrl": "",
                            "forceRequestMethod": false,
                            "requestMethod": "GET",
                            "requestContentType": "application\/x-www-form-urlencoded; charset=UTF-8",
                            "customRequestProcessing": ""
                        },
                        "Variable": "MatomoConfigurationVariable"
                    },
                    "trackingType": "pageview",
                    "idGoal": "",
                    "goalCustomRevenue": "",
                    "documentTitle": "",
                    "customUrl": "",
                    "isEcommerceView": false,
                    "productSKU": "",
                    "productName": "",
                    "categoryName": "",
                    "price": "",
                    "eventCategory": "",
                    "eventAction": "",
                    "eventName": "",
                    "eventValue": ""
                },
                "blockTriggerIds": [],
                "fireTriggerIds": [1],
                "fireLimit": "unlimited",
                "fireDelay": 0,
                "startDate": null,
                "endDate": null,
                "Tag": "MatomoTag"
            }],
            "triggers": [{
                "id": 1,
                "type": "PageView",
                "name": "PageView",
                "parameters": [],
                "conditions": [],
                "Trigger": "PageViewTrigger"
            }],
            "variables": [{
                "name": "MatomoConfiguration",
                "type": "MatomoConfiguration",
                "lookUpTable": [],
                "defaultValue": "",
                "parameters": {
                    "matomoUrl": "https:\/\/foobar.matomo.cloud",
                    "idSite": 1,
                    "enableLinkTracking": true,
                    "enableFormAnalytics": true,
                    "enableMediaAnalytics": true,
                    "enableFileTracking": false,
                    "enableCrossDomainLinking": false,
                    "crossDomainLinkingTimeout": "180",
                    "enableDoNotTrack": false,
                    "disablePerformanceTracking": false,
                    "enableJSErrorTracking": false,
                    "enableHeartBeatTimer": false,
                    "heartBeatTime": "15",
                    "trackAllContentImpressions": false,
                    "trackVisibleContentImpressions": false,
                    "disableCookies": false,
                    "requireConsent": false,
                    "requireCookieConsent": false,
                    "customCookieTimeOutEnable": false,
                    "customCookieTimeOut": "393",
                    "referralCookieTimeOut": "182",
                    "sessionCookieTimeOut": "30",
                    "setSecureCookie": false,
                    "cookieDomain": "",
                    "cookieNamePrefix": "_pk_",
                    "cookiePath": "",
                    "cookieSameSite": "Lax",
                    "disableBrowserFeatureDetection": false,
                    "domains": [],
                    "alwaysUseSendBeacon": false,
                    "disableAlwaysUseSendBeacon": false,
                    "userId": "",
                    "customDimensions": [],
                    "registerAsDefaultTracker": true,
                    "bundleTracker": true,
                    "jsEndpoint": "matomo.js",
                    "jsEndpointCustom": "custom.js",
                    "trackingEndpoint": "matomo.php",
                    "trackingEndpointCustom": "custom.php",
                    "appendToTrackingUrl": "",
                    "forceRequestMethod": false,
                    "requestMethod": "GET",
                    "requestContentType": "application\/x-www-form-urlencoded; charset=UTF-8",
                    "customRequestProcessing": ""
                },
                "Variable": "MatomoConfigurationVariable"
            }]
        }, Templates);

[blaggacao]

List of Variales from Matomo.

They are kind of the the "building blocks" for marketeer's programming via the "Tag Manager UI". As a developer, I understand that they are pretty much equivalent to the injection of a let foo = ...; block, just via UI.

But seeing the list might give an idea what type of tracking is actually needed (surely nobody needs the implementation a whole UI based programming language — which is what Tag Managers seem to be).

Time

• Time since pageload

Page Variables

• Custom JS (eureka!)
• DOM Element - gets value from dom element
• Data Layer - afaiu a fancy handle on localStorage
• First Party Coockie - reads from a cookie
• Referrer URL
• URL Parameter - a specific param
• URL Variable - current page url

SEO

• Page Meta Content - reads from meta tags

Utilities

• Constants

Others

• Custom Request processing function
• Matomo config (what you see above: configures the matomo analytics tracker)

I think all of these can be simply subsumed by injecting a custom tracking function into frappe.fel:

frappe.fel = (event_name) => ...;
// could become
frappe.fel = {% my_injected_tracker_function %} => (event_name, data) => ... /* continue with the default */;

In the Facebook case, I would want to extract the Facebook Click ID from the query parameter and make sure it's pushed as data payload. That's it. The backend facebook integration would then have to extract that freeform data from all events when talking to facebook conversion api. Simple.

---

Edit

I prototyped this in frappe/frappe@42e0280