background sync on mobile devices

[alexanderadam]

There are various issues (1, 2, 3, 4, 5, …) and threads (1, 2, 3) on it but it's still not clear to me what exactly is technically not possible to implement.

This comment from September 2018 says:

Cannot currently be fixed due a limitation of the underlying framework

Which might be true but that's what libraries are for, right? Therefore this is a very misleading comment.

And this comment from October 2019 claims:

we rely on existing libs, which for now don’t exist on React Native

But it doesn't mention which exact libraries would be needed. The logic for syncing is obviously already there. So in theory only the support for doing exactly this sync in a background task would be needed.
And there are libraries for react-native that are doing exactly this since a few years. For example

• react-native-background-fetch for Android & iOS (first version from August 2016)
• react-native-sync-adapter using the Android SyncAdapter (first version from April 2017)
• react-native-background-task again for Android & iOS (first version from February 2017)
• react-native-background-job for Android (available since December 2016)

As you can see, these libraries are far older than these comments. So again this would be wrong and a false statement in case this comment refers to syncing libraries.

Also there's this comment from June 2019

Most likely not possible. Android phones these days have services that kill off apps that aren't used, whether you want to keep them running or not. https://dontkillmyapp.com/

Well,

1. it only affects certain vendors in an extreme way. Not every phone is affected.
2. The exact same website you mentioned shows some workarounds for this very issue.
3. Apps are usually killed after a while — not immediately. So Joplin could probably still sync for a while until it would be killed.
4. Some FOSS apps which are showing notifications while they are running doesn't seem to be affected (or at least doesn't be to seem affected so often). For instance DNS66 or personalDNSfilter.

So also this comment is false / misleading since the already mentioned website also explicitly mentions workarounds.

I would obviously understand if the issue would only be the lack of people working on this. In that case this issue could be kept open for tracking, avoiding duplicates and having the possibility to show this issue for possible contributors, so that they know what's still open.

But currently the given comment are at least somehow lacking some information (or even not relevant / true).

Thus it would be great if someone could clarify what's really missing here.

PS: Thank you for your work!

[stale]

Hey there, it looks like there has been no activity on this issue recently. Has the issue been fixed, or does it still require the community's attention? This issue may be closed if no further activity occurs. You may comment on the issue and I will leave it open. Thank you for your contributions.

[jaanli]

Agreed, this is a major issue on iOS!

[skylize]

The pain of missing a background sync feature is greatly accentuated by the way in-app sync works:

Joplin fetches 100% of the available updates in a single transaction. If you switch to another app before sync completes, you risk power management shutting down Joplin's network connection(s) and/or Joplin itself, thus breaking the transaction.

Joplin only tells you anything went wrong if the OS decided not to completely kill Joplin between the failure and reopening the app. In the case Joplin alerts you, it does so with a brief red-italic message underneath the Sync/Cancel button. Despite the coloring, this message demands very little attention because it is normal weight, a smaller font than anything else in the Sidebar, and stuck against the bottom edge of the screen. To even have a chance of spotting it, you need to explicitly open the Sidebar, which is hidden by default.

The problem is self-perpetuating: A larger update list means a transaction is more likely to fail to complete during the time Joplin is both open and in focus. That increases the likelihood the next attempt to sync will include yet even more updates appended, and an even greater chance of failure.

---

I use Joplin on both my phone and laptop several times per day. Yet, as I type this, I am waiting on exactly this kind of gigantic sync on my phone. At some point (I'm guessing just over a week ago?) an initially-slightly-oversized update spiraled out of control into a monster.

After finally working out why a bunch of my notes were missing, I committed to keep the app open until completion (with Stay Awake activated in Developer Options). Unfortunately, after 1.5 hours and 68,000 updates fetched, I had an urgent need to use my phone for another task. When I came back to Joplin, all that progress had been just dumped in the trash 😠. ...Now waiting again; 36k fetched so far; fingers crossed the job will complete this time. 🤞

---

To attack this problem directly, sync would need to be changed to chunk update lists to a reasonable size; and failure notifications would need to be made more aggressive.

On the other hand, even if you completely put aside the obvious UX improvement that would come from having notes already-in-sync when you open the app, Background Sync would transform this absurd problem of unbounded sync transactions into a total non-issue.

Consistent periodic syncing naturally leads to smaller update sizes than occasional random syncing. But even if you throw some extra-large updates into the mix, finalizing and retrying would both be way less fragile when freed from coupling to UI uptime.

So with Background Sync, this issue simply up-and-disappears.

[jaanli]

Thank you @skylize - you nailed it, much more eloquently than I could have.

This is the only thing keeping me on 1writerapp.com which also syncs markdown files to dropbox, but changes appear within a second (really!) on either device, so I don't have to worry about losing info like @skylize did.

Would be curious to know what the missing pieces are to get this kind of performance, given that 1writerapp is so similar and nails the UX in this respect.

[alexanderadam]

Would be curious to know what the missing pieces

I wanna know this as well. The previous statements claimed that some things are missing but it wasn't explained what the missing things are.
It's certainly not the background functionality, since there are libraries for that, which are much older than the comments.
Thus it must be something else. 🤷

[roman-r-m]

It's just not been implemented, that's all.

[alexanderadam]

It's just not been implemented, that's all.

But this comment from September 2018 says:

Cannot currently be fixed due a limitation of the underlying framework

And this comment from October 2019 claims:

we rely on existing libs, which for now don’t exist on React Native

[jaanli]

Fantastic, thanks @alexanderadam!

How many hours of dev time would this take, or is this simply infeasible on React Native?

If so, I’d imagine this is a decisive factor of whether or not someone would use/switch to Joplin.

I donated 20 euros recently and happy to donate more.

Would be good to get a sense of how badly people need this feature (I certainly do), not sure how budgets work in open source but we can help fundraising on social! Open to any ideas/advice.

E.g. is this React Native iOS/Android library for background sync incompatible for some reason?

https://github.com/transistorsoft/react-native-background-fetch

[alexanderadam]

E.g. is this React Native iOS/Android library for background sync incompatible for some reason?

https://github.com/transistorsoft/react-native-background-fetch

I don't know but it's the first example library that's mentioned in the first comment.

[jaanli]

Sounds good, curious to know what libraries don’t exist on React Native then!

[jaanli]

The RescueTime.com app uses push notifications to trigger a background sync.

Could that be a quick and dirty workaround while the library issue is sorted out? The push notification could just trigger a background sync with Dropbox it seems. Maybe I’m missing something! Trying to think of quick fixes.

[tangodev]

This is the main reason why I dislike using Joplin on my phone. I even have a separate folder for taking notes on the phone as to avoid conflicts with desktop instead of editing stuff directly.

I'm supporting Joplin financially and I'd like to know when can I expect this to be implemented? I'm not angry, just disappointed.

"Babysitting" a sync is just sad.

[jaanli]

+1, same issue as @tangodev

[danepowell]

Note that someone implemented background sync in #4925 but it was not merged due to perceived reliability issues.

[roman-r-m]

Note that someone implemented background sync in #4925 but it was not merged due to perceived reliability issues.

It's wasn't actually completely implemented, see #4925 (comment)

[laurent22]

Ok let's stop here because nothing new is being said and it's getting heated. We need technical expertise, which takes time and costs money. Rants are cheap, but we get nothing out of them.