support for protocol buffers?

[mspanish]

Hello this looks great, thanks very much. I'm using the levelDB setting - as I'd like to save a bunch of records via Node.js - then copy it all over to my server for read only access from my serverless single page javascript app.

Look like I'll be able to do that nicely with this - but I'm also planning to use protocol buffers instead of json, as I have a lot of data - I've seen some other Node.js libraries that save to leveldb with this - and I think I can just decode/encode with yours - but have you thought of adding support for this?

Thanks again.

[only-cliches]

Hello Mspanish,

I actually hadn't heard of protocol buffers before you opened this issue, just learned a few things. 👍

I'm honestly not sure how you would go about writing protocol buffer support, from my (very basic) understanding it's not a standard but a set of concepts like Flux.

I'd guess that the protocol buffer feature would provide an alternative to the data model architecture currently being used, a different way to describe how you're storing the data.

Let me know what you're thinking.

Best,
Scott

[mspanish]

I just heard about it yesterday :) I'm just trying to get the leanest, fastest datastore possible for a static website. I started by Googling "alternative to JSON" - and quickly came up with MessagePack, BSON, and then Protocol Buffers as well as a couple of similar ones. Seems binary stores and loads quicker, which would be great!

I'll keep digging as it looks quite promising - -but one question I have is about NanoSql - I haven't tested client side yet - but using LevelDB that I just copy over (prepopulated with static data) - the user can access records 1 at a time, right? As in, when they search for a specific record - they don't have to download all of the JSON in order to access it? For example right now I have maybe 10MB of data that feeds my graphics app - but the chunks are actually very small - not even 1 KB per record. So imagine what a waste if each user had to download all that JSON just to get a few graphic definitions! I tried Oboe.js for streaming regular json - but it had no effect on the quantity of data they had to do - even though I set it to "hang up" when they found their item, as some items were located closer to the bottom of the index - the user still ends up with a big payload just to access a few things.

The old way to do this is to just use a server for the data - and I was going to do that - until I began looking into all of this other stuff, and thought there just may be a way to do this.

Thanks for your lib, it shows a lot of initiative, and looks like a fantastic db.

[only-cliches]

I've been using NanoSQL as a MySQL replacement on quite a few projects with great success.

With NanoSQL and the LevelDB backend a full table scan of 100,000 records takes around a second on a low end system. So even if you use a complex .where() statement with a large number of records, you can expect the result back very quickly.

If you can find a way to keep your search queries limited to trie types, primary keys, or secondary indexes you can expect queries to return within 10s of milliseconds.

I've got a little bit of documentation optimizing queries:
https://github.com/ClickSimply/Nano-SQL/wiki/4.-Default-Store#fast-queries

You can use .where() on the server to select only the records the user is looking for.

// A simple function with express to return only the desired results
app.get("/items", (req, res) => {
    if (!req.query.title) {
        res.status(500).send("Invalid query!").end();
        return false;
    }
    nSQL("records").query("select").where(["title","LIKE",req.query.title]).exec().then((rows) => {
        res.status(200).send(rows).end()
    });
});

Someone can send a request like website.com/items?title=world and they'll get back a json of all matching records.

This doesn't require the client to download any records they don't need, only getting the ones they're seeking.

Hope that makes sense!

Best,
Scott

[only-cliches]

Another note... you could easily save the records pulled from your server into client side nanoSQL instance with the same data models. With persistence on the client enabled this will give you the offline capability you're after.

[mspanish]

Hello and thanks again! How do I get the client side to know where the DB is? So if I have a bunch of saved data that I created in Node.js, if I copy the db folder over to the client - how would I get NanoSQL to know where to look for it?

[mspanish]

I'm not sure if I made it clear enough that I don't plan to serve this app from a server at all, it's a purely serverless environment - where the backend is being built offline.

[only-cliches]

Are you running an Electron app then? Where is the LevelDB running from and where is the client system in relationship to it?

[mspanish]

No - just plan html5 (Backbone). Right now i'm running on localhost - but the app will deploy to Netlify - a static web host.

I'm building everything locally using Node.js - I am just looking for a chunkable, portable data format so that users can access chunks of data from the Backbone app, with no need for a server. The alternative is to serve the graphic definitions from a regular database - I've uploaded some data to Appbase.io, and was going to serve my graphic definitions that way when I began reading about Protocol Buffers and other stuff - so I got the idea that I could build the DB using my local server and Node.js, then port all of the data to my web server, where the user could access this data through the web browser/app.

[mspanish]

Does levelDB require the user to use only data created BY the particular instance? So you cannnot use prepopulated or portable databases? Sorry about my lack of understanding in this area!

[only-cliches]

No problem at all. :)

Level DB stores it's actual data in files that aren't portable in the way you're describing, they must opened by software on your computer/server designed to work with Level DB stores.

For the use case you're after, LevelDB isn't a good solution since it MUST have a nodeJS instance running to serve it's data.

If you're using a static web host your options are very limited. You have two routes you can take:

1. You serve the data from somewhere like appbase.io or some other database as service like you suggested.
2. Chunk the data down into a bunch of smaller json files, host them on the static web host and ajax them in as desired.

At this point you could use anything to store the data locally in the browser. If you plan to allow the data to stick around after a page reload then using nanoSQL in your app with {persistent:true} will cause nanoSQL to save all the data you put into it to IndexedDB. Meaning the store will be in the exact same state before and after each page reload.

Hope that's helpful!

[mspanish]

that's very helpful - wading through all of the leveldb stuff I couldn't really tell if what I wanted to do was possible (it did seem unlikely to have access to any kind of DB client side without reading the whole thing, unless the data was stored in chunks to begin with! Thanks a lot, I'll see if I can use NanoSql on the other end to help with caching data and user stuff.