Connector for browser indexedDB

[bajtos]

At the moment, loopback's memory connector can be configured to persist data in browser's local storage. This has a significant problem as the local storage is size limited to about 5MB.

We should support storing the data in indexdb too, possibly via https://github.com/google/lovefield

A community-run implementation: https://github.com/r3dm/loopback-connector-indexedDB

[vigyanhoon]

5MB is practically too less for a major application housing all data offline.

Please add IndexedDB support. We can't think of an application to survive only on 5 MB where most part of the app is going to be offline.

[ffflabs]

How is localstorage support provided? I thought memory connector was persisted server side.

[bajtos]

See Memory.prototype.loadFromFile and Memory.prototype.saveToFile.

We had extended memory connector to support both fs files and browser's localStorage.

It is possible to extend the memory connector to load/save data in indexeddb, but such solution would be be very memory hungry, as the page will load whole indexeddb into javavascript memory (heap). A better solution is to implement a new connector that will fetch the data from the indexeddb on per-record basis as needed.

[jrschumacher]

What about using Mozilla's localForge solution?

[bajtos]

A powerful solution would be to implement a levelup connector, which could be then configured to use either LevelDB on the server or IndexedDB in the browser (via level.js).

However, both modules are quite large and thus the final client bundle would be even bigger than it is now.

In the light of that, I am inclining towards writing a connector directly on top of the IndexedDB API, so that the module size is as small as reasonably possible.

[jrschumacher]

Ah, that makes sense. Maybe it could be a wrapper which try's to detect leveldb or local forge and if found then resorts to the included library?

[bajtos]

Maybe it could be a wrapper which try's to detect leveldb or local forge and if found then resorts to the included library?

API wise, we could use localForge instead of IndexedDB. However, [https://mozilla.github.io/localForage/localforage.min.js] is almost 24kb, which is a problem.

Why are you pushing for localForage, what are the advantages compared to pure IndexedDB?

[jrschumacher]

I'm not. Just what I know.

[bajtos]

I'm not. Just what I know.

Fair enough. I was wondering if there is something important that I am missing.

[vigyanhoon]

Hi,
Any update on when this is going to be implemented. We would love to have idb implemented!
Thanks.

[bajtos]

@vigyanhoon This is not the top priority for us ATM, it's almost certain that the feature won't be implemented in the next 4 weeks. Would you mind contributing the connector yourself?

[vigyanhoon]

I will try. Please provide resources to look upto.

[bajtos]

@vigyanhoon I am afraid this is the only documentation we have so far: http://docs.strongloop.com/display/LB/Building+a+connector

The implementation of the filtering function is at the moment hidden inside the memory connector: loopback-datasource/lib/connectors/memory.js#L367-L527. We would like to expose it via a public API at some point, either as a public API of loopback-datasource-juggler, or even better as a standalone module.

[jrschumacher]

@bajtos how does loopback memory connector connect to the browser's localstorage?

[raymondfeng]

The implementation of the filtering function is at the moment hidden inside the memory connector: loopback-datasource/lib/connectors/memory.js#L367-L527. We would like to expose it via a public API at some point, either as a public API of loopback-datasource-juggler, or even better as a standalone module.

+1 to expose it. I want to use it as part of the rules engine.

[bajtos]

@jrschumacher search for "localStorage" in loopback-datasource-juggler/lib/connectors/memory.js.

Note that the approach used by memory connector, where it builds/loads a giant JSON containing all model instances, is not the right approach to use for the IndexedDB connector. The IndexedDB connector should store each model instance as a standalone IndexDB entity (row) and then use IndexDB API to access a single model instance or iterate over all model instances.

[gruzilla]

i just found https://github.com/r3dm/loopback-connector-indexedDB by @morenoh149 and @BerkeleyTrue

what about that (havn't tried it yet)?

[boscodsouza82]

I have tried this and it is not working. From what I can recall, it doesn't have the 'all' method implemented.
Also, the lack of solid indexedDB support (even) in iOS 8 is what made me change my track with regards to an indexedDB connector.
Note though: For me, it was a good starting point while learning to write a connector for loopback.

I'm currently working to put together a sqlite connector for loopback (iOS and Android). Basing my work off the existing loopback-connector-sqlite for node, in combination with Cordova SqlitePlugin.
Sqlite support on iOS and Android is solid.

[BerkeleyTrue]

Sorry y'all. That plugin is a work in progress.

[catogonzalez]

I'm currently working to put together a sqlite connector for loopback (iOS and Android). Basing my work off the existing loopback-connector-sqlite for node, in combination with Cordova SqlitePlugin.
Sqlite support on iOS and Android is solid.

This sounds great @boscodsouza82 +1

[boscodsouza82]

Had put this on hold due to being busy with other parts of my app. However, I have now resumed work on loopback (connector and sync), and will be posting a solution as soon as I can, within the month at worst.

For those that did, don't lose hope. :)

[gruzilla]

Had put this on hold due to being busy with other parts of my app. However, I have now resumed work on loopback (connector and sync), and will be posting a solution as soon as I can, within the month at worst.

would be awesome! @boscodsouza82 +1

[boscodsouza82]

Just to clear up some confusion (for myself and others).

Here's my approach for an alternative to LocalStorage. For now, this approach works for a Mobile App, not for a Mobile Website.

For the sake of discussion, let's use the Official Sync Example, https://github.com/strongloop/loopback-example-offline-sync

1. I first got the above example working as a Mobile App, via Cordova.
2. The app now used LocalStorage on the Client and MongoDB on the Server. Quick Note (to self): Both LocalStorage and MongoDB are not relational databases.
3. I next got the Server to use MySQL. I did this to ensure sync would work with a relational database (e.g. MySQL). My plan was to make it work with SQLite which, too, is a relational database.
4. The (app) Client continued using LocalStorage.
5. The additional connector I used on the Server was loopback-connector-mysql, (https://github.com/strongloop/loopback-connector-mysql).
6. To get SQLite working with Loopback, I figured I needed some SQLite connector.
7. So luckily and happily, I got a hold of this, (https://github.com/Synerzip/loopback-connector-sqlite).
8. However, it was a connector for the Server. I need a similar connector for the Client.
9. So I hacked out parts of the above Server SQLite connector and repurposed it to work on the Client side. I also hacked Loopback to add SQLite to the existing list of Client connectors. Further I also needed to use Cordova SQLite Plugin (https://github.com/litehelpers/Cordova-sqlite-storage).
10. I am, at the moment, testing out the Client SQLite connector and should be able to share results by tomorrow.

Note once again, that I based all my work off the Official Sync Example. We can keep our discussions focused around that example.

I am making this elaborate announcement for the following reasons...

1. To have people cross-question this Client Native SQLite connector/solution, so that it is implemented in the most robust and efficient way possible, for a Mobile App.
2. To state that this solution will not solve the LocalStorage problem for a Mobile Website. I understand now (at this late stage in the life of this thread), that IndexedDB is the only way to solve the problem for a MobileWebsite. Native SQLite is not an option there.
3. To be able to help out if anyone else is trying to put together similar alternatives to LocalStorage whether for Mobile Website or App.

Best regards,
Bosco.

[gruzilla]

Thanks @boscodsouza82 for your findings!

We tried to use replication as shown in the loopback-example with LocalStorage and PostgreSQL.

LocalStorage unfortunately is no solution for the future as it is limited to 5MB. Concerning robustness a sophisticated cleanup, archiving or some store-when-required methodology would have to be implemented following this approach.

As loopback client is still missing a well tested and maintained indexddb-connector, we are currently looking into removing the loopback-dependency in our client code testing http://dev.yathit.com/ydn-db as it already supports indexddb and syncs with rest-apis. Syncing via Rest makes the code independent of the underlying server side database like mysql, postgres or others. And loopback is strong in building awesome rest apis.

[boscodsouza82]

Welcome @gruzilla. :)

Side note: Unable to access http://dev.yathit.com/ydn-db

Also quickly read about ydn-db sync, https://dev.yathit.com/ydn-db/doc/sync/index.html

What are your thoughts about ydn-db sync vs loopback sync? No hurry, whenever you get the time. :)

[nikvaliris]

@boscodsouza82 - would it be possible to share your Cordova client version of loopback sync example? I tend to use the Ionic and Cordova client frameworks for Android so would be good to see how you went about it.

[boscodsouza82]

I am still testing it out on Android and iOS. Had hoped to finish y'day but still on it. Will share as soon as I am done.

[nikvaliris]

Brilliant thanks @boscodsouza82, this will help me save a lot of time.

[gruzilla]

sorry @boscodsouza82 for the link copy pasting mistake..

after looking a little closer at the code we saw that the sync-module is in a private bitbucket repository and we are unhappy with the whole project-setup. therefore we decided against using it.

again @boscodsouza82 are you interested in joining forces developing the connector? do you have a github repo for it?

[boscodsouza82]

@gruzilla :)

Half happy you decided against ydn-db since now we will have atleast 3 more people looking at this connector. me, @gruzilla and @nikvaliris. :)

For ydn-db though, I think they have a different approach than the one Loopback uses. However, I am not versed enough to comment on either. Maybe someone from the Loopback team can chime in, no hurry :).

I intend to put all this up on github soon, but I am a bit caught in device related troubles (on iOS and Android) and am ironing those out.

The project does appear a bit different from the original, https://github.com/strongloop/loopback-example-offline-sync. This is primarily because I have to tweak it to work as a mobile app.

@nikvaliris, it's already largely Ionic. However, I am not using Ionic lib, rather am using Angular directly, as per the official version. Will replace it in a few days.

2 Projects

1. loopback-example-offline-sync-multi, a modified version of the Official Sync Example.
   I intend to name it MULTI, because I eventually intend to somehow use SQLite in the desktop browser as well, atleast for Chrome. Not sure who would want such a thing. However, for starters it will only be tested on iOS and Android. For now though, the desktop version could rely on localStorage just like the official version.
2. loopback-connector-sqlite-cordova, named as such to distinguish it from the server side connector, https://github.com/Synerzip/loopback-connector-sqlite.

I am interested in joining forces. 👍

[boscodsouza82]

1. Here's the project for collaboration, https://github.com/boscodsouza82/loopback-example-offline-sync
   It exposes the 'multi' branch by default, let's use that for collaboration. The 'master' branch is forked from the Official Sync Example.
2. The name of the sqlite connector is loopback-connector-cordova-sqlite (as oppposed to loopback-connector-sqlite-cordova as I had initially thought).
   I have not yet created a github repo for it. Will do that as we proceed. It's usable though.

Functionally, the connector creates a SQLite DB, and it's table on the devices. I am facing some issues with some failing SQL queries so debugging those at the moment.

Please feel free to use the code and provide solutions if possible.