Android missing data investigation and fixes

What makes the Android app's BackgroundService stop running?

1. The app crashes. We had previously built a Crash Handler with code to restart the app if it crashed, but there was a bug in the way we were telling it to restart, so it wasn't working. We fixed the Crash Handler's restart code in commit 1a6c0e8, app version 2.3.0.

2. The user closes the app using the App Switcher. When the user closes the Beiwe app from the phone's App Switcher, the BackgroundService gets killed. In BackgroundService.java, we were already calling restartService() inside onTaskRemoved(); that appears to make the app restart.

3. The operating system kills the app. To solve this, we’ve:

   1. Started the app with the START_STICKY flag, which tells the operating system to try to restart the app if it has killed it (documentation link). We did this in commit ebe91f5, app version 2.3.3.

   2. Considered making the app a Foreground Process, but decided not to, because that would involve an extra notification appearing. Actually, can we make the app a Foreground Process? https://developer.android.com/guide/components/activities/process-lifecycle

How do we restart the app if it’s not running?

1. We added more Intent triggers to BootListener to restart the app, including SMS_RECEIVED and DATE_CHANGED. We did this in commit 334a2fd, app version 2.3.2. There are other intents we should add: https://developer.android.com/guide/components/broadcast-exceptions

2. We added a repeating timer that goes off every two minutes and tells the BackgroundService to restart if it has stopped. We did this in commit ebe91f5, app version 2.3.3. We believe there’s no problem with sending a start signal to the BackgroundService as frequently as we please. If the BackgroundService is already running, calling startService will have not stop and restart the service. According to the documentation for the startService command, "If this service is not already running, it will be instantiated and started (creating a process for it if needed); if it is running then it remains running."

   The restart alarm persists, even when the app crashes or isn’t running. While plugging the app into a debugger, we used the command adb shell dumpsys alarm | grep beiwe to see the alarms registered for the Beiwe app. Alarms remained registered through app crashes, App Not Responding (ANR) errors, and events where someone killed the app from the task switcher and we disabled the app’s automatic restart code. We weren’t able to test the event where the operating system kills the app, because when that happens is unpredictable, and it often takes several hours.

Data upload: do files ever fail to upload and get stuck on the phone?

During our testing, we did not observe any files get stuck on the phone and fail to upload to the server. However, we did find and fix several code problems that could theoretically have caused this to happen:

1. When the phone joins a WiFi network, we made it immediately start uploading files: previously, the Android app only uploaded data files when a repeating timer told it to do so. This could theoretically result in files getting stuck on the phone if the repeating timer was set to a low frequency and the phone wasn’t connected to WiFi very much. We supplemented the repeating timer by adding a listener so that when the app connects to a WiFi network, it should immediately try to upload data files. This should make file upload more reliable.

2. Synchronized the file upload function: while examining the code, we realized that the function that uploads data files can be called multiple times simultaneously. This would be likelier to happen if the file upload frequency was set to a time shorter than one hour. We don’t know what could have gone wrong if the file upload function tried to upload the same file twice, although it probably wouldn’t have resulted in any data being overwritten. Nevertheless, we thought it safest to add Java’s synchronize keyword to the file upload function, which will prevent the function from being called more than once at a time. Now, if the function is called twice, the first run must complete before the second run can start.

3. Redid file upload timeout logic: the file upload function has a built-in timeout to keep it from running for too long. Previously, this timeout was set to be the file upload frequency (which is customizable for each study) minus 2.5 seconds, the idea being to prevent one call of the file upload function from overlapping with the next call. However, if the file upload frequency was set to something very low, like 10 seconds, that would mean that each file upload function would time out after 7.5 seconds. If a large file took more than 7.5 seconds to upload, it would never actually be uploaded to the server, and it would remain stuck on the phone. After adding the Java synchronize keyword to the file upload function, we no longer needed to prevent the file upload function from being called while it was still running, since even if it’s called, it shouldn’t actually start until the previous call has finished. Therefore, we hardcoded the file upload timeout to one hour, which should be long enough for the largest data files on the slowest WiFi connections. We also added functionality to log errors in both Sentry and the Android App Log file if the app ever does hit the upload timeout.

All three of these fixes are in commit 971312b, released with version 2.3.3 of the Android app.

Other issues we fixed

We fixed several other bugs that may have either been crashing the BackgroundService or preventing the app from writing one type of data or another:

1. A permissions issue that sometimes crashed the app when trying to record the WiFi log: https://github.com/onnela-lab/beiwe-android/issues/34