Skip to content

Make an in-browser version #23

Closed
louischatriot opened this Issue Jun 17, 2013 · 31 comments
@louischatriot
Owner

After seeing how much faster NeDB is than TaffyDB, I'm considering porting NeDB to the browser, with persistence support using LocalStorage. Please comment if you think this would be useful.

@louischatriot
Owner

Clearly localStorage is not ideal, considering its limitations, but at least it is supported by more browsers than IndexedDB. What I could do is to use IndexedDB when it is available and fall back to localStorage when it's not.

@superqd
superqd commented Jun 20, 2013

I think it would be worthwhile to create a browser version. Local client side storage is nice, but sometimes it's very useful to have a better in memory data structure for use as a data cache, etc (especially if you might exceed the HTML 5 storage limits, and can't rely on indexeddb support).

@superqd
superqd commented Jun 20, 2013

I just looked at your benchmarks against TaffyDB. Wow. NeDB would be an awesome alternative to TaffyDB. I know for some of the uses I was looking for, the find function alone would make NeDB the winner. If it was available in the browser.

@carlsverre

The browser side world would benefit from having a faster datastore. With the advent of faster JS interpreters, more and more data processing happens on the client. In terms of persistence, I am with the others on focusing on localstorage. It may also be interesting to look at web workers to perform tasks such as async-durability (write data to localstorage in the background, rather than sync with the inserts).

@louischatriot
Owner

Thanks for the comments, that confirms the need to port NeDB to browsers so I'll do it. The web workers idea is interesting, I'll do the porting first and then see how to do background persistence. I am experimenting on a similar idea for the current NeDB (I call it pipelining), but this is still experimental (queued saves can actually slow current operations).

@maxogden
@louischatriot
Owner

@maxogden Yes as I said I'll probably use local storage only as fallback. But even as just an in-memory store, I think NeDB will be useful, when I see the number of people who use the much slower TaffyDB.

@utunga
utunga commented Jun 20, 2013

I was looking for a good way to store data locally in a (phonegap) type mobile app and then have it sync with MongoDB. There may be other solutions but if this was a way to do that it would be awesome too.

The other angle where I'd like to see this in the client is to make easier fakes for web browser based unit tests

@louischatriot
Owner

@utunga I made this utility that you can use to transfer all your data from a NeDB database to a MongoDB one (for people who used NeDB for rapid prototyping and outgrew it).

For now you need to give it a NeDB datafile but I will adapt it when I port NeDB to the browser.

@FREEZX
FREEZX commented Jun 22, 2013

I would definitely say this feature would be very useful.

@naturalethic

@louischatriot I'd very much like to have this library available for IE 9+, which would require LocalStorage support.

@maxogden PouchDB only uses IndexedDB, perhaps that's more for you. There is value for some of us in supporting older browsers. WebSQL is dead.

@pspeter3

I think it would be cool to make the browser storage engine as plugin or adding events emitters to the database so you could have it sync to Dropbox or Google Drive as well.

Example

var tasks =  new NeDB({localStorage: 'tasks'});
tasks.on('insert', function(task) {
  dropbox.writeFile('tasks.db', tasks.localStorage, function(err) {
     if (!err) { console.log('synced');
  });
});

This would require creation by specifying the id and access to the localStorage value. The reference for the dropbox sdk is here https://tech.dropbox.com/2012/08/some-love-for-javascript-applications-2/

@nevf
nevf commented Jun 28, 2013

@louischatriot I am excited to see Nedb come along, especially in seeing it's similarities to MongoDB and would love to see it working in the Browser.

I agree with @maxogden etc, that you should avoid localstorage, as it is too limiting for any serious use. Either WebSQL or IndexedDB are available for all browsers and there are front end wrappers that convert the IndexedDB API into WebSQL as required. This will enable you to just write a DB plugin for IndexedDB that will then run on all Browsers. See: http://nparashuram.com/IndexedDBShim/

@naturalethic

I am mystified at why you guys are trying to persuade Louis to not implement a feature that he seems to want to do, and that I absolutely require, when doing so has zero effect on anyone who prefers IndexedDB as that will also be supported.

@nevf
nevf commented Jun 29, 2013

@naturalethic If your localstorage meets your requirements then that's fine, but I would have thought it less work for Louis to support a single API, that being IndexedDB, which with the shim I mentioned will work on all browsers. In that case why even bother supporting the limited capabilities of localStorage. To quote: http://stackoverflow.com/questions/2989284/what-is-the-max-size-of-localstorage-values

Web storage can be viewed simplistically as an improvement on cookies, 
providing much greater storage capacity (2.5 MB per origin in Google Chrome; 
5 MB per origin in Mozilla Firefox, and Opera; 10 MB per storage area in Internet Explorer) 
and better programmatic interfaces.

And you may want to see the results on this page: http://arty.name/localstorage.html

Further localStorage only supports strings and has no index capabilities. And the user can delete the contents of localStorage, which could be a disaster waiting to happen.

@mpj
mpj commented Jun 29, 2013

God god yes! I'd love to see this.

@mainlove
mainlove commented Jul 6, 2013

Does nedb having a Browser based Console to see/edit/config the data .
Sometimes we need to watch the data in Memory when developing or testing.

The best embed database in JAVA--h2, having a Browser based Console application, which is very convenient.

@louischatriot
Owner

There is no such thing yet. What you could do however is use Mongo Edit (another project of mine, a web based Mongo GUI) and modify it to make it work with nedb. That should not be too much work since both APIs are very close. I unfortunately don't have enough time to do it.

Here is the link : https://github.com/louischatriot/mongo-edit

@superqd
superqd commented Jul 10, 2013

Will the in browser port be available as a different project, or will the existing project be modified to work in both environments?

@louischatriot
Owner

My goal is to use the same project since most of the code will be common.

@louischatriot
Owner

I finally found the time to do it, the browser version is ready! Check the readme out: https://github.com/louischatriot/nedb#browser-version

It's still young, for now it doesn't support persistence (working on it) and I haven't yet tested it on all browsers (working on it) but at least you can test this young version :)

@naturalethic

Dependencies 'underscore' and 'binary-search-tree' are missing from the dependency check.

@louischatriot
Owner
@naturalethic
@louischatriot
Owner
@naturalethic

Yes that's clear. I had just chosen to build it myself, this console output should make clear what I meant:

joshua@zeta nedb § node browser-version/build.js
Installing build dependencies
Removing contents of the src directory
Copying source files
Copying browser specific files to replace their server-specific counterparts
Browserifying the code
Error during build
[Error: module "underscore" not found from "/Users/joshua/Workspace/nedb/browser-version/src/lib/datastore.js"]
joshua@zeta nedb § ls node_modules/
async       browserify  fs-extra    uglify-js
joshua@zeta nedb § npm install underscore
npm http GET https://registry.npmjs.org/underscore
npm http 200 https://registry.npmjs.org/underscore
npm WARN package.json uglify-js@2.2.5 'repositories' (plural) Not supported.
npm WARN package.json Please pick one as the 'repository' field
npm WARN package.json callsite@1.0.0 No repository field.
underscore@1.4.4 node_modules/underscore
joshua@zeta nedb § node browser-version/build.js
Installing build dependencies
Removing contents of the src directory
Copying source files
Copying browser specific files to replace their server-specific counterparts
Browserifying the code
Error during build
[Error: module "binary-search-tree" not found from "/Users/joshua/Workspace/nedb/browser-version/src/lib/indexes.js"]
joshua@zeta nedb § npm install binary-search-tree
npm http GET https://registry.npmjs.org/binary-search-tree/0.2.3
npm http 200 https://registry.npmjs.org/binary-search-tree/0.2.3
npm http GET https://registry.npmjs.org/binary-search-tree/-/binary-search-tree-0.2.3.tgz
npm http 200 https://registry.npmjs.org/binary-search-tree/-/binary-search-tree-0.2.3.tgz
npm WARN package.json uglify-js@2.2.5 'repositories' (plural) Not supported.
npm WARN package.json Please pick one as the 'repository' field
npm WARN package.json callsite@1.0.0 No repository field.
binary-search-tree@0.2.3 node_modules/binary-search-tree
joshua@zeta nedb § node browser-version/build.js
Installing build dependencies
Removing contents of the src directory
Copying source files
Copying browser specific files to replace their server-specific counterparts
Browserifying the code
Creating the minified version
Build finished with success
@louischatriot
Owner

The issue was that the source dependencies such as underscore and binary-search-tree are not checked in to the Git repo so you needed to reinstall them. This is now automatic.

@Amii23
Amii23 commented Aug 22, 2014

Is there any way to integrate NeDB with Phonegab, so we can use it for mobile Devices(android, ios etc).

@louischatriot
Owner

That would definitely be valuable and interesting, yes. I have no experience with Phonegap, could you open a new issue with the requirements ?

@louischatriot
Owner

As of v1.4, NeDB automatically selects the best method to persist data on the browser (IndexDB then WebSQL then localStorage), so the limit is removed. WARNING: it's not back compatible with v1.3, but that should not be a major issue as user apps can resync entirely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.