Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backup APKs of apps not being backed up #65

Closed
chirayudesai opened this issue Jan 9, 2020 · 5 comments
Closed

Backup APKs of apps not being backed up #65

chirayudesai opened this issue Jan 9, 2020 · 5 comments
Labels

Comments

@chirayudesai
Copy link
Member

The system doesn't backup any apps which don't have data, which would include apps which haven't really been used.

This is with / after APK backups.

For example, I had wireguard instaleld on my phone but it didn't get backed up because I hadn't added any server entries to it (Though I had definitely launched it after install)
I would've still liked to have it backed up though, as I wanted the app present for when I do end up needing it.
Having the APK for thsi backed up and restored would be nice, as to the user it's an app they had on the older phone and don't on the newer phone after restore.

@chirayudesai chirayudesai changed the title Backup un-used / not eligible for backup user-installed apps Backup un-used user-installed apps Jan 9, 2020
@grote
Copy link
Collaborator

grote commented Jan 10, 2020

At the moment #64 does not back up APKs of apps that:

  1. have not been used, thus do not have any data to back up
  2. are over quota and thus don't get backed up
  3. opted-out of backups completely

Right now, in #64, backing up the APK and writing metadata about the backed up app are done together. First I did it early, so we included metadata about a bunch of apps we didn't actually back up, so they were shown as failures when restoring.

Then I moved it to a place where we were sure the app got backed up. This got rid of the failures, but excludes unused apps (1) or apps over-quota (2).

A possible solution could be to break up APK backup and metadata writing a bit more, maybe have two types of metadata writes: one after the APK got backed up and one after the app got backed up. We would need to remember the difference in the metadata, so that we re-install apps without data, but don't give the impression that we have a backup of the data as well.

In the restore screen, we might still show those data restores as failures to make transparent that we couldn't restore data for those apps.

So this takes care of (1) and (2), but it doesn't solve (3), because for apps that opt-out of backup, our code will not even be called. To solve this, we probably need a parallel backup mechanism that takes a list of all opt-out apps and at least backs up their APKs. Ideally this could be hooked into the regular backup schedule, but I fear we will need our own job scheduler service for this.

Alternatively, we could add all apps whose APKs didn't get backed up through the normal means to the parallel mechanism, if we need one anyway.

@chirayudesai
Copy link
Member Author

Interesting, considering this option opens up even more possibilities.
From a user perspective:

  1. Good to have in the backup, since after restore they will be the same as before.
  2. This could get tricky. Are we hitting quota with many apps, and if so, by what amount? If it's not too much maybe we could separately consider increasing the quota.
  3. Confusing, since the user would see the app but not the data

For 1, it's straightforward, and can be added straight away once we have a technical solution
For 2, let's try and get some more data
For 3, anything we do can easily get tricky and confusing for the user, so we have to tread lightly.

A parallel mechanism sounds good and a way to solve this issue, but at the same time could lead to complications.

One thing I can think of - add some more special case handling when @pm@ is being backed up, and figured out the idiosyncraises of the various apps at that time if possible.
Fire off our own service when we the system asks us to start backing up, which backs up these apks.
Our service should still run after the system backup finishes hopefully.

@grote
Copy link
Collaborator

grote commented Jan 10, 2020

  1. This could get tricky. Are we hitting quota with many apps, and if so, by what amount? If it's not too much maybe we could separately consider increasing the quota.

On my personal phone I saw this for one app, but F-Droid and Nextcloud for example had a similar issue. This normally happens when developers don't care about backup and don't take measures to stay under Google's 25 MB limit. Normally an app shouldn't get close to this.

I don't think increasing the quota will help. We already doubled Google's quota. Apps that don't care will just amass data until they are above whatever quota.

What we could do though is add a field to the metadata reflecting the backup status of each app, so it becomes transparent to the user and the app could eventually fix it. These could be for example:

  • APK_OPT_OUT when an app opts out of backup, but we got its APK
  • APK_ABOVE_QUOTA when an app is above the quota, but we got its APK
  • APK_ERROR when an app has a backup error, but got its APK
  • APK_NO_DATA when an app has no data to back up, but we got its APK
  • APK_AND_DATA we got APK and data for the app, should be for most apps
  • SYSTEM_ONLY_DATA a system app whose data we got

We could then show this in a meaningful way to users to hopefully make things less confusing.

One thing I can think of - add some more special case handling when @pm@ is being backed up, and figured out the idiosyncraises of the various apps at that time if possible.
Fire off our own service when we the system asks us to start backing up, which backs up these apks.
Our service should still run after the system backup finishes hopefully.

We need to be careful here as the AOSP backup code is very under documented and can have surprises. For example, apps can notify the system that data changed and then only a backup of that app is triggered. I've also seen @pm@ getting backed up many times with just a few other packages. Since @pm@ includes many small keys, it already takes a long time, but it is a possible candidate for hooking this into.

@chirayudesai chirayudesai changed the title Backup un-used user-installed apps Backup APKs of apps not being backed up Jan 10, 2020
@chirayudesai
Copy link
Member Author

chirayudesai commented Jan 10, 2020

  1. This could get tricky. Are we hitting quota with many apps, and if so, by what amount? If it's not too much maybe we could separately consider increasing the quota.

On my personal phone I saw this for one app, but F-Droid and Nextcloud for example had a similar issue. This normally happens when developers don't care about backup and don't take measures to stay under Google's 25 MB limit. Normally an app shouldn't get close to this.

I don't think increasing the quota will help. We already doubled Google's quota. Apps that don't care will just amass data until they are above whatever quota.

Makes sense.

What we could do though is add a field to the metadata reflecting the backup status of each app, so it becomes transparent to the user and the app could eventually fix it. These could be for example:

  • APK_OPT_OUT when an app opts out of backup, but we got its APK
  • APK_ABOVE_QUOTA when an app is above the quota, but we got its APK
  • APK_ERROR when an app has a backup error, but got its APK
  • APK_NO_DATA when an app has no data to back up, but we got its APK
  • APK_AND_DATA we got APK and data for the app, should be for most apps
  • SYSTEM_ONLY_DATA a system app whose data we got

Looks good. I think that pretty much covers all cases, and if not, more could always be added I guess.

We could then show this in a meaningful way to users to hopefully make things less confusing.

One thing that might help is having different failed icons / colors.
Red for complete failure / data loss, yellow for something intermediate, green for good.

grote added a commit to grote/seedvault that referenced this issue Jan 13, 2020
This should also affect apps that have other errors during the backup
process, but it does not affect apps that opt-out of backup completely.

First part of seedvault-app#65
grote added a commit to grote/seedvault that referenced this issue Jan 14, 2020
This should also affect apps that have other errors during the backup
process, but it does not affect apps that opt-out of backup completely.

First part of seedvault-app#65
@grote
Copy link
Collaborator

grote commented Jan 17, 2020

This has been fixed in the develop branch, so can probably be closed now or when merged into master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants