Offline mode missing

[ClementHard]

If the dispach of events occurs when the device is offline, the events queue is cleared before knowing if the request was a success.

[d4rken]

I'm probably going to tackle this before going into production with piwik in my Android app. Any ideas on a solution direction?

[dotsbb]

@d4rken I think there should be some FIFO container within Tracker instance so we could store here untracked views. It would be great if this container would be configurable (for example it might accept MAX_SIZE param).

[d4rken]

A few points that we should consider:

• We should store somekind of version code or error handling with the offline data. Future updates might break compatibility with previously stored but unsent offline data. Best case, still reuse it, in any othercase discard it.
• The local storage should be limited to a configurable size. We could add a setting that allows the dev to choose betwen storing in /files or /cache. Within cache files may be removed automatically if space runs low, but we might loose offline data if the user chooses to clear cache.
• The local storage should probably be a buffer such that if we exceed the limit, the oldest offline data is overwritten.
• As we are now dealing with transmission failures, there should be some kind of system that deals with "retries".
• An analytics framework should minimize it's impact on the app and device, so if we regain connectivity, and there is a lot of offline data, we shouldn't send it all at once, but gradually transmit the views. This will also limit the impact this has on the server.
• If the piwik instance is offline, this would count as transmission failure. If we have a million app users, and our piwik instance goes offline, and then goes online again, all users transmitting all offline data would probably DDOS the piwik instance.
• We could gradually transmit the offline data over a time span. Would we need to store new events if there still is offline or can we send them directly? (e.g. does the server get confused if they are totally out of order)

[dotsbb]

@d4rken Have You seen this fork? It does 0 of yours considered points at this moment but it can be something we can refactor and introduce features incrementally to it.

[d4rken]

I have now 😉 . It's one way to solve it but as you mentioned, it addresses none of the concerns. I would probably start from scratch, not refactor this.

@apc-kamezaki You might want to refactor restoreEventsFromFile . It is currently called in the constructor which means it's called from the UI thread (IO operation on the UI thread are bad). This also means if there are lots of cached events, it would block the main thread and as Piwik ususally initialises from Application.onCreate it would block the app start (huge impact on the app using Piwik).

[suzukieng]

I just had a look at both the iOS and Android tracking library codebases. The iOS tracking library uses CoreData (ultimately SQlite) to persistently queue a configurable number of events. Wouldn't it be logical to implement it in the same way for Android, i.e. using sqlite and optionally some lightweight ORM on top of it?

[d4rken]

@suzukieng That would be one good way.

But generally we don't have to focus on a specific implementation just because other piwik code bases use it (as long as it works sufficiently well). The public API should be similar and it should behave similar. "Under the hood" stuff like this is almost FFA 😁 . SQlite would be fine, but so would raw files or maybe fancy stuff like realm.io.

The primary goal is to have a low impact on the host app and as secondary goals to somehow achieve the points mentioned above.

[suzukieng]

@d4rken Completely agree with primary/secondary goals, low impact is absolutely key. What I find hard to judge is how much of a difference it makes if you store events persistently vs. buffer them in-memory, i.e. how often will the app be killed by the OS before events can be uploaded.

Android processes are in general pretty long-lived (see: http://developer.android.com/reference/android/app/Activity.html#ProcessLifecycle) and perhaps an Android Service can be persuaded to live on as long as there are events. But this would again probably go against the low-impact goal and also slightly complicates integration because you would have to add the Service to the manifest.

BTW: SharedPreferences are another light-weight mechanism for storing stuff persistently, I know the folks at count.ly use it in their Android SDK.

[orian]

Many Android devices despite having pretty decent version of system (4.3+, many 5.0.3) are short on resources. Thus I would not assume the processes are 'long lived' or the logs from the low-level devices will often be lost. Also keeping them alive just because we can sounds less user friendly than just dumping the data and sending on the next occasion.

Storage
What about using Context.getCacheDir() or if WRITE_EXTERNAL_STORAGE persmission is granted than using Context.getExternalCacheDir(). This has the advantage that if a system really needs more storage it can decide to clear the directory. The SharedPreferences happen to go OutOfMemory for some bigger data, but I've never tried to store more data there.

[d4rken]

SharedPreferences would not be very suited for this as this isn't a good case of key/value based access.

In general we have to assume the app could be killed at any moment, so unless we can somehow get a callback for when the app is killed, we shouldn't keep data in memory for long. Something like batching a few failed requests and writing them into storage based on a timer, would be my idea.

The right location to store this would indeed be a subdir of the cache dir because these files are cached and can be lost without any grave repercussions. More specifically Context.getCacheDir() (private cache, i.e. /data/data//cache) because the analytics could be considered private information (depending on what the developer tracks).

[mattab]

Hello guys, FYI: we are also implementing Offline mode for the official Piwik JS Tracker code (in matomo-org/matomo#9939). So far we make it super simple and you can see the idea here: https://github.com/piwik/piwik/compare/2.x-dev...offlinetracker?expand=1
Developer would do

tracker.setUserOffline({push: function (request) {
// eg localstorage.addItem(request);
}});

tracker.setUserOnline(localstorage.getItems());

(Just FYI in case we could maybe reuse same method name or similar. Fun to see that we are synchronised and implementing this important functionality at the same time in both projects!)

[d4rken]

Not sure yet about which approach to take.

Manually settings online/offline mode really is a simple approach, but i think the sdk should provide this on Android. Alternatively reacting only to successful/failed communication with Piwik would also allow clients to cache data in cases where the issue is on the server side. But permanent issues due to code errors would just max out the cache.... Maybe just checking connectivity and reacting to changes then...

As we don't want the authtoken to be used on Android (can't be stored securely), we should just have to cache data for up to 4 hours and discard older data. All already have the "cdt" parameter set, so sending them out of order should also not matter, right?

[mattab]

Hi @d4rken
Hope you are well :-)

To answer a few points:

• In FYI: Piwik Tracking API now allows data up to 24 hours in the past #139 we're announcing that Tracking API now accepts up to 24 hours in the past (the value is also configurable in the Piwik config.ini.php file)
• the order of requests does matter for a given device: Piwik Tracking API needs to receive requests in the right chronological order for a given user / visit, but it's OK if requests from several users/devices arrive at different times.
• +1 to SDK providing offline feature automatically on Android

[d4rken]

the order of requests does matter for a given device: Piwik Tracking API needs to receive requests in the right chronological order for a given user / visit, but it's OK if requests from several users/devices arrive at different times.

Even if the cdt parameter is set? What happens if they are out of order? Are they just discarded?