[Feature Request] Drive/Dropbox sync

[MatissJanis]

This might be controversial, but bare with me.

Actual is a local-first app. The frontend is deployable to a static website. Eventually we might expose it nicely in actualbudget.com domain for everyone to use.

However, the server is dynamic. It requires some technical knowledge to deploy it. Which means users with low technical skills won't be able to use the sync capabilities. For many this is a deal-breaker.

So here's the feature request: what if we allowed to do sync to drive/dropbox or other 3rd party?

--

Technical details (just an initial proposal to get the ball rolling)

The current sync implementation would not work with these 3rd parties. Theoretically we could set-up another sync mechanism that syncs the full database, however that will have performance issues for large budgets. So another alternative is to sync with multiple files. If the current sync logic sends small protobufs, we could also send those same protobufs to a 3rd party and store them there. And once we have ~50 individual protobuf files on the 3rd party we "batch them" into a larger file. This way we keep the sync functionality fast whilst having a full backup sitting on a 3rd party server.

But again: this is just an early proposal. You understand the feature request better now, so please leave some better ideas in the comments below (or submit a pr).

[jlongster]

It's a great idea! I'll drop some opinions here. Dealing with a sick kid throwing up so this won't be as detailed as it could be. Obviously my opinions are biased here as someone who built the current system.

I've looked into something like this before. It would be great if we could leverage the local system to handle moving data back and forth, but unfortunately that kind of robust system just doesn't exist. There's a lot of dragons when it comes to distributed systems: you need to make sure everything is written to disk atomically and that the checks and balances are done at the right time.

A filesystem is a poor platform for that kind of work, especially a networked filesystem. It would be a failure if you started reading a file that was being written to, for example. There are ways around this, but it really requires a new syncing format entirely to ensure that data is read and written in the way that you expect 100% of the time to avoid corrupt sync states.

The most critical flaw: the merkle trie. That's how you know your current data is actually accurate and that you are up-to-date with everyone else correctly. We apply sync messages and then update the merkle trie to update the hash: this has to be persisted to the filesystem atomically. Your data and the merkle trie must always 100% be in sync. You cannot do this kind of deterministic writes with google drive/dropbox.

The result of these failures will be really obscure things, like the app saying you are currently up-to-date but the data actually isn't. It's never acceptable for this to happen imo.

Personally, I just simply don't like these kinds of services for app state. I use logseq with an iCloud backend, and tried dropbox before. When I open the mobile app sometimes it takes minutes for data to show up. And sometimes data doesn't show up at all! My experience with storing this data as an app has never been good. And apps like logseq (ynab 4, an old version, even used something like a file sync) are all moving away from this and building there own syncing strategies.

I think there is something here; you could do some research on CRDTs stored as filesystems. It could be really exciting. But you'd have to reinvent large parts of the syncing layer for it to be robust imo.

[jlongster]

Also I worried that you all, as maintainers, would have to deal will a lot of users complaining about something going wrong with syncing when it isn't your fault at all!

[Kidglove57]

A non technical user view point by way of illustration.

An old Mac budgeting app called Moneywell had its sync broken by Dropbox. The new custodians have been working for 3 years to restore sync. In the end they accepted that iCloud was the lesser of the evils from amongst their possible solutions and it has been in beta for a year. However, iCloud sync still gives quite a few problems even after all that effort and in particular in how slow it can be to catch up on the second device.

As James intimates, I do fear that this and the much talked about bank feeds could cause no end of “help!” Issues for the volunteer team!

[biohzrddd]

Great idea. When I was thinking of writing my own app from scratch, using Google for access control and syncing to Google Drive was my #1 feature to implement. Thankfully, I found Actual Budget :).

Personally, I like how YNAB4 on Windows did it. You save the file for each device that accesses the budget to the dropbox/ynab/device folder, dropbox deals with syncing. it is up to the device to merge changes from other devices. In my 10+ years using YNAB4 this way across 1-3 devices and two users, I've maybe once had a conflict, but it only showed up as a duplicate transaction.

But with a web app on a server, you need to access the dropbox/google drive/onedrive/etc API and sync files from there.

What's the requirements for a server to add syncing to each service? Is it per app or per server? That could be another hurdle for new users, registering for and getting an API key.

[MatissJanis]

I'll close off this idea in order to clean up our backlog. If someone still wishes to work on this - please open a new issue and lets continue discussing!