NEW: Automated cloud backups and OAuth support!

[dupontgu]

This was a little bit of a side quest, but a few weeks ago we realized that the Tool Manager provided a path towards implementing two highly-requested features:

Cloud Backups and OAuth

Backups: LightOS will now be able to automatically back user data up to a cloud provider of their choice. This includes internal LightOS stuff like photos and messages, and also data owned by community tools! At backup time, each tool will be passed the time of last successful backup and will have to opportunity to provide a list of files that are due to be backed up. We built this on top of the existing Tool Manager I/O so we hope you'll find it relatively easy to integrate.

OAuth: This one has haunted us forever. There are so many great third-party services that use OAuth to log users in! But if you're familiar with how it works, you know it generally relies heavily on the usage of a web browser. We at Light remain committed to NOT putting a browser at our users' fingertips. However, the Tool Manager's embedded web server and browser-based UX (which is accessible from a different device) gives users a convenient way to log in to their favorite services that's decoupled from Light's cloud backend. (The one little piece of cloud infra required here is a little Cloudflare relay that third-party OAuth can call back to, which forwards the credentials back to a Light Phone via websocket. Browsers won't let you redirect to a local IP address directly. We'll have a publicly available instance, but tools will not need to use ours! It's open source and can be hosted on any Cloudflare account.)

These two features are slightly intertwined at the moment - the first cloud providers we're going to support for backups are Google Drive, Dropbox, and Microsoft OneDrive, all of which require OAuth to log in. We know that many of you want to move away from these organizations if you haven't done so already, so are working towards supporting WebDAV as well. Please let us know if you'd be excited about that and/or if you'd be able to help test it.

We are also considering spinning up our own encrypted cloud backup service. Not a high priority at the moment, but again, let us know if you'd be excited to use that instead. It would take considerable effort, and will likely incur a small monthly fee from users, but we like the idea.

What happens next:

1. Backup repo and OAuth relay are public today! take a gander.
2. This week, I'll open up a stack of PRs in the SDK repo to wire up backups in the server and allow for testing within the emulator. The Tool Manager Demo example tool will be updated to demonstrate how SDK-built tools can provide backups. The meat of the OAuth stuff will be in these PRs.
3. That code will get pulled into LightOS once merged into the SDK server and included in an upcoming release.
4. For those devs interested in OAuth for your own tools - I'm going to need to rearrange a few things to simplify this. Feel free to follow the pattern used in the SDK server in your own tool, but I'd recommend hanging tight for a week or two

Anticipated questions:

• Can users restore backups? We are currently prioritizing getting data copied off the device, but we absolutely plan to add restoration as well. Backups of LightOS will be human viewable/readable once they land on the cloud and that's the most important piece from our point of view - we don't want users to lose valuable data. If you're going to add backup support to your tool, we highly encourage making it easily accessible in it's backed-up form!
• Are existing tools going to get OAuth upgrades? (looking at you, Calendar) Eventually!
• Can you support [other cloud provider]? Feel free to ask this, but we'll likely only add providers if we see a very large number of requests. We feel we're covering a good amount of ground with the 3 most popular (US, at least) + WebDAV

[charliemacmillan]

@dupontgu I would love to see iCloud Drive added as a provider if possible, and assuming there are enough requests for it to be worthwhile!

[dupontgu]

Would love that too personally but I don't think Apple has an SDK that will run on Android. Seems like there are some unofficial things but I am hesitant to build on top of anything reverse engineered.

[wsturgiss]

Might be able to spin up a proxy service that can take in WebDAV and send files off to icloud.

[wsturgiss]

If it is acceptable, I'd like to port some of my existing Immich backup tool to the official light backup repo. Are you open to self-hosted options? I know this could lead to fragmentation and maintenance overhead concerns so I'd also like to propose a non-giant tech company alternative as well: Proton Drive.

[wsturgiss]

Perhaps S3 as well, thats fairly universal among developers.

[wsturgiss]

Did a bit of research, looks like Proton is developing an official SDK but its still in development and so may change. There are also 3rd party clients that support a legacy API that Proton maintains...presumably until the SDK is officially released.

[dupontgu]

Yeah I just found the same info. I think Proton would be a great fit! Will keep an eye on that. Immich is awesome but I don't think it fits in the current paradigm. Cloud provider needs to be file type agnostic, since a user's entire backup will end up there.

[wsturgiss]

Understood. That makes sense. I'd suggest rsync or syncthing but I think that might be too technical.

[wsturgiss]

I have a working webDAV server spun up so I can begin testing. I can wait till it hits main or if you guys want I can try some stuff on your feature branch.

[wsturgiss]

Oh, looks like WebDAV isn't in the branch yet.

[dupontgu]

PR for webdav is up here! Admittedly have only had a chance to test with a local throwaway server:
#291