database sync (Offline Support)

[natgross]

Does AW db support syncing devices? So devices can work offline, something like MongoDB Atlas, or [ahem] Firebase?

[eldadfux]

We had initial thought on adding a separate library that will act as a proxy in front of Appwrite and sync any changes and provide offline cache depending on the user network status. The main idea was to handle it in separate layers to avoid added complexity to the API and avoid coupling.

[natgross]

Since Appwrite is being billed as a Firebase replacement, I was hoping that it also includes database, which then means offline syncing like Firebase.

[eldadfux]

I'll leave this issue open, and we'll update when we make progress in this front.

[1cedsoda]

I would love to see offline support for appwrite

Lately, I have been looking into offline document databases and offline support in firestore and noticed a few things

Firestore offline support works after an online-first paradigm.

• Firestore allows you to specify a location to fetch from, either server or cache or automatic (default) if you are online every document is fetched from the database.
• Out-of-the-Box firestore allows to create/edit/delete objects when the device is offline and this mechanism is very good and also well explained on YouTube.

I tried to implement a workaround for offline-first support for firestore with lastEdited timestamps, where it fetches "old" data from the cache and "new" data from the server. This works with a major flaw: The client won't detect if an object was deleted or query results changes that affects the cached documents.

An offline-first paradigm document database would be a firebase-killer for some people, like me in a current flutter project.
Users outdoor sometimes don't have an internet connection / mobile data flat at all, or a bad one.

I think of a client SDK with an offline-first/client sync document database that automatically

• cache fetched data,
• only pulls changed data
• and query results that affects data in cache
• There would also be a mechanism to tell the client which documents were deleted
• There is a system to add/edit/delete locally and push changes later when the device is online
• Run queries on the cache with indices

The only document database ,I know, that currently works on this functionality is MongoDB with its product "MongoDB Realm Sync". They aquired Realm some time ago who developed on-device databases. Sadly they haven't released an flutter SDK yet, because they are currently working with the dart developers to develop some new features for dart. Also as far as I remember MongoDB provides offline first support only in connection with their own priced hosting "MongoDB Atlas". There is no option to self-host a MongoDB with offline-first support.

[natgross]

There are two competing Couchbase Lite Flutter implementations. They are both offline-first. Although incomplete api's, they function pretty much ok. https://pub.dev/packages?q=couchbase

…

On Fri, Jul 2, 2021 at 5:21 PM Phyyyl ***@***.***> wrote: I would love to see offline support for appwrite Lately, I have been looking into offline document databases and offline support in firestore and noticed a few things Firestore offline support works after an *online-first* paradigm. - Firestore allows you to specify a location to fetch from, either server or cache or automatic (default) if you are online every document is fetched from the database. - Out-of-the-Box firestore allows to create/edit/delete objects when the device is offline and this mechanism is very good and also well explained on YouTube. I tried to implement a workaround for offline-first support for firestore with lastEdited timestamps, where it fetches "old" data from the cache and "new" data from the server. This works with a major flaw: The client won't detect if an object was deleted or query results changes that affects the cached documents. An *offline-first* paradigm document database would be a firebase-killer for some people, like me in a current flutter project. Users outdoor sometimes don't have an internet connection / mobile data flat at all, or a bad one. I think of a client SDK with an offline-first/client sync document database that automatically - cache fetched data, - only pulls changed data - and query results that affects data in cache - There would also be a mechanism to tell the client which documents were deleted - There is a system to add/edit/delete locally and push changes later when the device is online - Run queries on the cache with indices The only document database ,I know, that currently works on this functionality is MongoDB with its product "MongoDB Realm Sync". They aquired Realm some time ago who developed on-device databases. Sadly they haven't released an flutter SDK yet, because they are currently working with the dart developers to develop some new features for dart. Also as far as I remember MongoDB provides offline first support only in connection with their own priced hosting "MongoDB Atlas". There is no option to self-host a MongoDB with offline-first support. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACDECK4JPFMJGUPHV3VZVYTTVYUVXANCNFSM45HSHKKQ> .

[de1mat]

It would be great if the offline data could be encrypted at rest as well.

[ccfiel]

I think this is one major feature that lacking to compete with firebase or aws amplify to have offline first feature

[echosalik]

There are two competing Couchbase Lite Flutter implementations. They are both offline-first. Although incomplete api's, they function pretty much ok. https://pub.dev/packages?q=couchbase

I would suggest use something like SQLITE with JSON1 and Cipher APIs. It can act as a proxy, all data pull gets saved to sqlite too and any data pushed to server gets pushed to sqlite as well. That way even if network is not present or has poor performance, it can still work offline.

Now that proxy will be the determining factor to either load from sqlite or use live data.

[jogehl]

This would really be a nice feature. Is there any plan to implement it?

[fivestones]

I agree that this would be a great feature. Is anything in the works to add it?

[meepeek]

WatermelonDB and RxDB seems to be the best starting point.

• WatermelonDB is on client side. It comes with sync info and sync adapter for customize sync function to match any remote backend. It is also based on SQLite so appwrite team may share existing code since appwrite core is on mariadb. It also support RxJS API. See WatermelonDB
• RxDB can sync with graphql server, this is an option if appwrite plans to support graphql. See RxDB

I would like to propose the idea as well as taking note for myself.

[bgervan]

WatermelonDB and RxDB seems to be the best starting point.

• WatermelonDB is on client side. It comes with sync info and sync adapter for customize sync function to match any remote backend. It is also based on SQLite so appwrite team may share existing code since appwrite core is on mariadb. It also support RxJS API. See WatermelonDB
• RxDB can sync with graphql server, this is an option if appwrite plans to support graphql. See RxDB

I would like to propose the idea as well as taking note for myself.

Both of it Js only or react only solutions. Appwrite provides SDK in other languages like flutter, which would be hard to implement that mechanism, and maintain different solution may not the best option. It's easier to come up with a self made solution which can be adapted to all client SDKs. If they implement CRDT on the client sides, then it's not that hard and it requires a few field only, like hibrid logical clock.

[meepeek]

@p4-k4 I too require that feature but I think it is a very complicate one. FOSS man power is limited so I think it could be a while before it ready. Currently, the most I can do for them is testing, propose ideas and report issues I came across as I use.

For now, I think the team has good progress that now I can use appwrite for my own user, data, api key management and also serverless approach to some extent.

Hopefully, this feature could come along with Apple's PWA support for push notification that might available next year. Once appwrite is solid enough, we'll all have the option to escape the big companies' jails.

[hey-nicolasklein]

Just checking in ahem 1 year later ;) Offline support is a dealbreaker for many cases, so much that it is the sole reason people look to Firebase (and as of recently, MongoDB Realm Flutter SDK beta was released with offline sync).

There's been another similar discussion for supbase but looks like discussion is being ignored regardless of the demand.

Currently seems Firebase and Realm are the only solutions to look to and the worst thing of it, they're all paid solutions and your data is held with someone else!

I would also add AWS Amplify as another similar solution with offline support. But like you've said, absolute vendor lock in.

[scriptsman]

Is there any other solution for web and flutter working fine with appwrite before official offline support come out?

[mmv-dev]

Did any tried WatermelonDB with appwrite. Or have any idea on how to connect. Please help me.
Thank your for your valuable time.

[nmdias]

This is a cherry on top feature. Giving a user instant access to a collection that is already locally cached, while it goes in the background to fetch for possible updates to that collection, is a killer feature.

Top of the head benefits:

• Major user experience and usability upgrade for end users 🙋‍♂️
  • Offline usage
  • Handles unstable data connections, no blocking user flows
• Developer joy 🥳
• Instant access to cached documents 🚀
• No unnecessary usage of cpu, memory, bandwidth for unchanged collections 🤤

I've read it was planned for after v1.0? Do we know if 1.0 is close, anyone?

[pchasco]

I don't consider this a "cherry on top" feature. It is a requirement for mobile apps whose network connectivity will always be unreliable.

[thirstycoda]

I don't consider this a "cherry on top" feature. It is a requirement for mobile apps whose network connectivity will always be unreliable.

I agree. Not having this feature is a deal breaker for me

[hey-nicolasklein]

I also think that Appwrite is not fully a self-hosted firebase without that feature. One of the main aspects of a firebase realtime database is the offline capabilities. As mobile devs we need this feature to fully replace firebase.

[mmv-dev]

Sincere request for the developers to implement this feature on priority. You will see a explosion in the count of users using Appwrite.

[iamshivkumar]

Waiting for this feature

[bazrafkan]

Any update for this feature?

[algoflows]

It's mental to me that no one is solving this issue with the only two players on the market being AWS and GCP. Every single metric shows increased mobile usage year over year, every one knows offline first is critical and necessary for serous multiplatform/mobile apps.

The only real viable options on the market right now for any product that includes mobile is Amplify, Firebase or Couchbase. You either need to roll your own offline online sync or switch back to one of the three listed above.

Developers expect and want clients that can sync with multiple backends. Mark my words the first company to bring a unified Graph data layer and offline sync wins the PAAS race. Developers want flexibility to move between different data solutions but also sync client server state with global consistency. Also the ability to access multiple data endpoints from a single graph client...

Imagine a world when choosing a globally distributed strongly consistent database for example CockroachDB were unified into a single Graph with any other db you wanted using something like (Hasura/Wundergraph) but then double that up with offline sync capabilities for offline first experiences.

Unstoppable.

[laurentpayot]

In a recent Meetup, a guy from a relatively big financial startup told me offline stuff is so complex that they were trying to reverse-engineer Firebase to see how they use IndexedDb in a reliable way.
Maybe appwrite offline support development effort was underestimated?

[TheRedSpy15]

In a recent Meetup, a guy from a relatively big financial startup told me offline stuff is so complex that they were trying to reverse-engineer Firebase to see how they use IndexedDb in a reliable way. Maybe appwrite offline support development effort was underestimated?

Depends on the project and how it's structured. Firebase's offline implementation is really nothing more than a cache and compares timestamps when communicating with the server. Obviously I'm simplifying it, but that is the logic behind it

[pchasco]

In a recent Meetup, a guy from a relatively big financial startup told me offline stuff is so complex that they were trying to reverse-engineer Firebase to see how they use IndexedDb in a reliable way. Maybe appwrite offline support development effort was underestimated?

Depends on the project and how it's structured. Firebase's offline implementation is really nothing more than a cache and compares timestamps when communicating with the server. Obviously I'm simplifying it, but that is the logic behind it

It is quite complicated once you get into conflict resolution and things of that nature. Personally, I would have used CouchDB and Couchbase Sync Gateway for the backend. There is already Couchbase Lite (native) and PouchDB (JavaScript and IndexDB) for the client which enable offline-first with sync. The community editions are open source and free of charge, although I haven’t read the licenses to know whether there is some language that would prevent their usage in appwrite.

[eldadfux]

I can confirm this feature is in work. We’re planing to release a beta version of the Flutter SDK with full offline support. Once stable, we’ll replicate the design to the web SDK and then to both Android and iOS

[fivestones]

I can confirm this feature is in work.

This is awesome! It's the main thing I've been hoping for in appwrite all along...especially excited about that last one: iOS.

How are you going about this? Is it a form of CRDT? How are you dealing with conflict resolution? Just curious.

[laurentpayot]

@eldadfux Will offline persistence also work in web workers like Firestore?

[hortigado]

Great, it is important for me..

[mdajminpradhan]

I was almost sure that i'm gonna switch to appwrite. but suddenly searched whether appwrite has offline support or not. Now I see i have to wait for it.

Appwrite we need it ☺️♥️

[stnguyen90]

For a sneak peek at our implementation of offline support, look at this discussion: #5326! We'd love to get your feedback!

[mghase]

We had initial thought on adding a separate library that will act as a proxy in front of Appwrite and sync any changes and provide offline cache depending on the user network status. The main idea was to handle it in separate layers to avoid added complexity to the API and avoid coupling.

Where did this endup

[hortigado]

@mghase I recommend this package, I did some tests and I was able to implement it 100% with appwrite. Now I am deploying to a real project. It works very well and you can choose how you want the data to be sent and received from the server to be saved locally.
I'll try to make a tutorial when I finish the current project I'm working on
https://pub.dev/packages/brick_offline_first

[TechComet]

Hive Flutter
https://pub.dev/packages/hive_flutter
https://pub.dev/packages/hive

[ccfiel]

I notice RFC is using a cache, not a local database? Can this handle lot of data? https://github.com/appwrite/rfc/tree/feat-offline-support/021-offline-support

[aliazimoshan]

This would really be a nice feature. Is there any update?

[rohankm]

I wrote a custom sync function for watermelondb and my app works offline in React native

[SMARTMICROWORLD]

Is there any update @stnguyen90 ?

Also add offline feature for logged in users like we get logged in user data even after if device is offline like in firebase auth().currentuser. In react native

[Sun3]

I am also looking for an update. Please give us at least an approximate date range and if this is even a priority. This is a crucial feature for any mobile app and without it, how can we switch from Firebase to AppWrite?

Thank you.

[hortigado]

@Sun3 You can use FlutterData or https://github.com/GetDutchie/brick

[nathanael540]

Hello, does anyone have an example of how to use Appwrite with brick?

[xwchris]

I found this https://signaldb.js.org/examples/appwrite/