-
-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Offline tracking in JavaScript API #9939
Comments
+1 |
This would be a nice feature indeed |
So...how long we would plan to have this functionality? E.g. 3.0 would be released when? |
Piwik 3.0 would be in about a year but this feature is not planned yet. Pull request or suggestions on how to implement it are always welcome 👍 |
Just looking for such a solution. This is tremendously important for mobile apps. In the JS tracking client, I have noticed a method called retryMissedPluginCalls() and the array missedPluginTrackerCalls. It could be interesting to hook it in some way in order to intercept the calls to the server in an offline state. Then, upon "online" event, we would call and retry missed calls. |
Offline tracking is super important nowadays for mobile apps, progressive web apps, .... If someone wanted to work on it I'm happy to give some support. I think it needs to be worked out what the best place is to save requests that couldn't be sent because the user is offline (eg localstorage, ...) and then we need to detect whether user is offline and when user goes online again. Some browsers have an API for that. There might be one problem when tracking the requests later. I think by default Piwik lets you only track requests up to 4 hours in the past without needing an authentication token. The time is customizable AFAIK but it might be something to take into account that requests older than 4 hours may have to be invalidated. |
Thank you for your prompt reply. I need to find some solution for this rather promptly, whether using Piwik or something else. Ideally the former though, due to its ability to easily deal with hybrid applications.
OK, I see. No problem.
Well, I don't think that this is so important. It should be left open to the developer to decide what to plug into the system. Everyone might have different preferences.
Ditto. Developer should supply this to make it easy for Piwik. What I mean by this is that there should be methods to call in Piwik that let it know that "now it is necessary stop sending tracking data and save it instead" and "now resume sending and send what has been stored". By the way, the "stop sending" and "resume sending" functionality is already working now.
So that part is simple. Even calling the JS tracking client (or loading it locally) is easy to solve upon detection of online/offline events. I already have this part solved. Thus the only remaining thing is "how to store the data, so that Piwik would know when those tracking event happened, so that it would be possible to reconstruct the past sequence correctly upon delayed sending". I need your or Matthieu's input on this, as you understand the existing code base and its functionality. (Thank you in advance.)
I don't understand this one. What kind of a token? The point is that the device might be offline for one week or a month. This means that we just need to keep storing the tracking data with original timestamps and simply submit it in a sequence, when the device comes online. This means that we won't have such data available for analysis immediately, but only eventually – yet it is still important and better than not having it at all. The question is to what extent the current system could support this scenario without major code changes. |
Updated the previous comment. -- |
@tsteur It is possible to write such an "offline tracking" solution as a plugin? If yes, I could look into this ASAP, if I am given a guidance on how such plugins are written. Thanks. |
With token I mean you need an authentication token in this case. It is actually hard coded that when you want to track a request that is older than 4 hours you need to authenticate see https://github.com/piwik/piwik/blob/2.17.0/core/Tracker/Request.php#L467-L474 . This is for some security reasons eg you could otherwise track into any Piwik instance data in the past etc. This token can be disabled though here https://github.com/piwik/piwik/blob/3.x-dev/config/global.ini.php#L696 I feel like we could maybe add an API to the tracker like When user becomes online, the developer could call eg a method It could be probably written as a plugin, but this API is not yet official and is undocumented and we would for sure need to add some methods to the tracker. Adding those methods to tracker could be done quickly though. I'll show you rough idea in a bit without thinking too much about it |
This could be rough idea: Developer would do tracker.setUserOffline({push: function (request) {
// eg localstorage.addItem(request);
}}) tracker.setUserOnline(localstorage.getItems()}) |
Eventually Piwik would ideally detect offline status itself and store it somewhere. The biggest problem remains the Piwik backend re the 4 hours in past only |
Thank you. |
I made in an array because an array has out of the box a I renamed the @mattab do you have any thoughts on this? |
OK, I understand your point.
Makes sense.
Absolutely. Looks very good to me. Simple and effective.
Cordova/Tizen + plain JS + HTML + CSS.
I have just cloned the Piwik repository and I am going to look into the reasoning for this limitation... |
I have just found this: It looks like the bulk submission automatically bypasses the authentification. Am I right in the assumption that it applies to our case? If yes, the whole problem will have been solved tonight. ;-) |
If not, what is the "bulk request" then? |
Oops! |
However, I have found this:
https://github.com/piwik/piwik/blob/3.x-dev/config/global.ini.php#L684-L685 Could this fit our needs? |
@tsteur |
On top of that, you are changing the version of JS Tracking Client that lacks some code related to Thus you have effectively overwritten its declaration on the line 3084. |
Also, the semicolon at the end of the line 3084 should be a comma, as the declaration of variables continues on the next line: |
Yeah the time in my virtual machine sometimes gets wrong :) The config you mentioned only applies to bulk requests in general. Not to the recording records in past. For this Regarding idpageview I think you are currently looking at Piwik 3 (not released yet), the change was made on Piwik 2 (branch 2.x-dev). |
Yes, I realised this. Therefore it looks like that 3.0-dev wasn't updated to the latest 2.17 or 2.x-dev.
Oh, that's a pity... :-\ |
This looks great @tsteur ! I'd vote for inclusion in Piwik 3 as a rather powerful new feature once tested & documented. Will help tons of people and make Piwik more resilient! |
Would also like to bump this up. What work needs to be done on the JS end? I could take a crack at it. |
I suppose you would need to store the requests eg in a local storage or so and replay them at a later time when the internet connection is back. Not sure if much else needs to be done. |
bugsnag.com (a bug/log tracking solution) does exactly this to report problems which occured when offline. |
@tsteur, @mattab, IMHO offline storage should be added using service workers (SW). This is the most logical way forward for adding any offline capability. What should be done is:
Also, the SW should forward all requests irrespective of the time passed since caching the event to the server (in the correct sequence of course). The server can then decide if it wants to accept the request and process it or reject the request and discard the data. This way, in the future, if the server code was changed to allow events older than 4h, it would be transparent to the client. Lastly, the SW approach means that the JS client itself is completely oblivious to the SW and works regardless of connectivity. The SW is API agnostic as it only saves request without manipulation and this way the offline capability is completely decoupled from the JS client and server. Its worth noting that GA also uses a similar mechanism to add offline capability to their framework. |
Thanks @saifeer very much appreciated 👍 |
hope this feature will exists someday |
When we implement this, how will we deal with the fact that currently Tracking API requests are only allowed up to 24 hours in the past?
will it be "acceptable" to drop requests made more than 1 day ago by default, and explain users they can increase the setting in the config? or we'd make this a UI general or per-site setting, or some other solution? |
Yes, the idea is that this is acceptable and it's always possible to change it. |
Learning from others, you'll want to introduce an maximum amount of held back messages so it does not flood local storage or post big payloads |
BTW did we maybe also want a new dimension whether a request was executed online vs offline? I reckon might be separate feature but to be seen. Might create issue for it later. |
FYI started some early concept using service worker in https://github.com/matomo-org/matomo/pull/15970/files Looks like this could work even for bulk requests and if send beacon is used. It's not too much tested though and haven't tweaked code much and haven't tested it in any browser but chromium. Development is quite early stages. If someone's proficient in service workers and IndexedDB feel free to leave some comments. The goal will be to cache the actual JS tracker file, and put all tracking requests in a queue should the user be offline. |
@PCSun1987 @juliusstoerrle @bdurrer @saifeer @Maqsyo Anyone maybe able to have a look at #15970 or even give it a test? I don't have any service workers in use so not sure if there's something that would cause issues with other service workers etc. Have only done some basic testing so far but if any possible be great to test it.
|
@tsteur Anything left here for Matomo 4? |
@sgiehl it's not finished (maybe not even really working yet). We'll try to get some feedback and try to get people to test it so we can see if/how it works and what adjustments it needs, etc. We'll hopefully have another beta soon and then get some feedback. Anyone seeing this issue, the feature should be included Matomo 4 beta 3 and newer. Be great to give it a try and comment here if it worked or not and what issues you ran into or what was not clear. |
FYI as this feature is available in latest Matomo 4 beta here is an FAQ on how to use it: https://matomo.org/faq/how-to/how-do-i-set-up-matomo-offline-tracking/ If you have any feedback be great to let us know |
To be honest, as someone who is first hearing about this feature, to me it sounds kind of… suspicious, let's say. My feeling is that it breaks some kind of implicit user expectation - I would never guess that a website might be tracking my every click even if I'm flying across the Atlantic and browsing a saved copy (if that's how it works, I'm not sure). It doesn't sound like a feature I would expect to find in a privacy-focused software. Just IMHO. |
If I may add, a famous quote by Jeff Goldblum from Jurassic Park about scientists comes to mind ;) |
Wait, there are people who think they are not constantly tracked everywhere? :) PWA and Electron-based web applications are quite a thing. Funny you mentioned flying, because I deploy these things to a few thousand devices of airline cabin crew. A lot of webapplications are expected to work just fine while the connectivity is bad, "offline" does not have to be a long timespan. |
Closing this issue now. Documented in https://matomo.org/faq/how-to/how-do-i-set-up-matomo-offline-tracking/ If you run into any issues with this feature, simply create an issue here in our GitHub repository. Are you a service worker or IndexedDB pro? We’d love if you could help us review our implementation. You can check out the code in the offline tracking pull request. We appreciate any feedback to help us improve this feature! You can comment directly in the PR or contact us |
Current it's not possible to send tracking event offline.
So one idea would be to extend PIWIK tracking API especially for JavaScript with additional parameter about even time-stamp (optional), so inside JS, you can keep the event locally and once connecting to internet, send all the events with the event happening time.
For basic tracking, probably can do similar way.
https://forum.piwik.org/t/does-piwik-work-even-your-offline/7295/9
The text was updated successfully, but these errors were encountered: