Permalink
Browse files

sync api spec

  • Loading branch information...
youurayy committed Mar 14, 2012
1 parent 2271a44 commit 4431bb78c4ffe4a338d023dd42890fa57d173e65
Showing with 331 additions and 1 deletion.
  1. +7 −1 README.md
  2. +292 −0 SyncAPI.md
  3. +32 −0 logo.fxg
View
@@ -48,7 +48,7 @@ So the idea here is to bring datastore functionality and scripting into the same
- [awesome](https://github.com/janl/awesome) - "A Redis implementation in node.js"
- [nedis](https://github.com/visionmedia/nedis) - "Redis server implementation written with nodejs"
- [EventVat](https://github.com/hij1nx/eventvat) - "evented in-process key/value store with an API like that of Redis"
-
+- [PouchDB](https://github.com/mikeal/pouchdb) - "Portable CouchDB JavaScript implementation"
### Scratchpad
@@ -81,6 +81,12 @@ So the idea here is to bring datastore functionality and scripting into the same
- single-process - fast, but multiple cores and multiple Nodes cannot work with the same data, clustering must be applied
- shared-memory implementation - certain overhead and latency but higher total performance up from a certain number of cores (atomic ops and async API necessary at this point)
+
+
+### API
+
+Proposed base API is [here](https://github.com/ypocat/nodejsdb/blob/master/SyncAPI.md).
+
### Notes
View
@@ -0,0 +1,292 @@
+## nodejsdb
+# Synchronous API Specification
+
+The base structure that should be sufficient for most use cases is a key-value map with ordered key elements. In the context of synchronous API, this structure is fully contained in fast memory (RAM).
+
+A properly implemented tree structure should not be slower (e.g. see "Performance" [here](https://github.com/pconstr/rawhash)) than an un-ordered (hash) map, and thus the necessity of multiple types of maps can be avoided.
+
+The need for a pure list structure is currently unproven, as in many cases the FIFO/LIFO structures can be implemented using an ordered map with unique or weighted keys, thus for now a list structure is not included, although if real-life experience proves to require one, it will be added.
+
+The fact that this version of the API is synchronous is allowing to make it very simple, e.g. atomic operations are not needed (because all DB calls are atomic), callback result functions are not necessary, and command queuing is not needed (because there is virtually zero latency to be avoided). In fact, the sheer simplicity of the synchronous API and its programming model may in the end prove more efficient than a multi-threaded, (possibly off-process), asynchronous version (both development and runtime efficiency considered). A supporting rationale for that case would be: *"At some point, you are going to have to shard anyway, so do it early and do it properly"*.
+
+To maintain maximum implementation and usage flexibility, the API doesn't specify (external) persistence. (Intrinsic persistence is not considered to be feasible with a synchronous API). External persistence can easily be added via the 'set' event (see below). This allows for implementations which e.g. don't store data to hard-drive at all, but rather commit them in batches (where the buffered batch can be protected by multi-machine redundancy; /contrast this with full-data redundancy a.k.a mirroring/) to an external store, like S3.
+
+
+
+## Usage
+
+```js
+// request the DB
+var db = require('the-impl');
+
+db.set('users', 1234, { fname: 'abc', lname: 'def' });
+
+var user1234 = db.get('users', 1234);
+
+var subset = db.range('users', true, 10, 20);
+
+var sz = db.size();
+
+```
+
+## Operations
+
+### Set
+
+```js
+db.set(name, key, value, event);
+
+/*
+ Parameters:
+ name: name of the ordered map
+ key: the key
+ value: the value to set or overwrite
+ event: optional; override the name of the callback; see Events below
+
+ Spec:
+ 1. both key and value support (and store the type of): String, Number, Boolean, Buffer, Object, Array
+ 2. Object and Array are serialized without checking for cyclic dependencies
+ 3. everything except Number is compared (and stored) as binary string
+ 4. Number compared with binary string always compares lower (is ordered in front of it)
+ 5. setting `null` as value deletes the key (thus no explicit delete operation)
+ 6. `null` keys are not supported (implementation dependent behavior)
+ 7. on insufficient memory, an exception is thrown with message 'out of memory'
+ 8. `true` is returned if the DB was changed:
+ 8a. on delete, the key was existing and deleted
+ 8b. on set, the key did not exist
+ 8c. on set, the key did exist but had a different value
+
+ Examples:
+*/
+
+db.set('users', 'mykey', { fname: 'abc', lname: 'def', created: Date.now() });
+
+db.set('usersByScore', 123.456, 123);
+```
+
+### Set Event
+
+```js
+db.on.mycollection = function(name, key, value, previous) {
+ // your handler code here
+};
+
+/*
+ Parameters:
+ name: name of the collection that was changed
+ key: the key
+ value: the value that was set, `null` for deleted
+ previous: the previous value that was replaced
+
+ Spec:
+ 1. after each `.set()` which changes the database state, an event is triggered if a handler is registered
+ 2. by default, handlers are registered under the collection name
+ 3. this collection name registration can be overriden with the `event` parameter of `.set()`
+ 4. since this is a synchronous API, you have to return from the handler synchronously (and fast)
+
+ Examples:
+*/
+
+// classic example:
+db.on.myCollection = function(name, key, value, previous) {
+ // `name` is 'myCollection'
+};
+db.set('myCollection', 'somekey', 'somevalue');
+
+// example of event type overriding:
+db.on.myDynamicCollection = function(name, key, value, previous) {
+ // `name` will be 'myDynamicCollection-12456' for the example `set` below
+};
+db.set('myDynamicCollection-12456', 'somekey', 'somevalue', 'myDynamicCollection');
+
+```
+
+### Get
+
+```js
+db.get(name, key);
+
+/*
+ Parameters:
+ name: name of the ordered map
+ key: the key
+
+ Spec:
+ 1. data is returned in the type it was stored under
+ 2. the key doesn't have to match the original type, but it has to match when binary-serialized
+ 3. null is returned for non-existent keys
+
+ Examples:
+*/
+
+var val = db.get('users', 'mykey');
+```
+
+### Range
+
+```js
+db.range(name, descending, from, to, limit);
+
+/*
+ Parameters:
+ name: name of the ordered map
+ descending: optional; true if the traversal order is in descending order, false for ascending
+ from: optional; what key to start at, inclusive; use `null` for start at the edge
+ to: optional; what key to end at, inclusive; use `null` to end at the edge
+ limit: optional; max number of results to return
+
+ Spec:
+ 1. result is aray of objects with properties key: and value:
+ 2. keys and values are decoded to their original types
+ 3. if the map is non-existent, returns an empty array
+
+ Examples:
+*/
+
+var res = db.range('usersByName', true, 'A', 'Z', 10); // res = [ { key: , value: }, ... ]
+
+res.forEach(function(v) {
+ myusers.push(db.get('users', v.value));
+});
+```
+
+### Size
+
+```js
+db.size(name);
+
+/*
+ Parameters:
+ name: name of the ordered map
+
+ Spec:
+ 1. returns the number of keys in the map
+
+ Examples:
+*/
+
+var sz = db.size('users');
+
+```
+
+
+## Example
+
+A simple schema with 2 models: __User__ and __Message__, where one sender can message multiple recipients. The storage API is hypothetical. This example uses `Array.forEach` for brevity, but in a high-performant code you probably want to use [something else](http://jsperf.com/fore-vs-for/2).
+
+```js
+
+// User: { login: }
+// Message: { sender:, recipients:, text:, date: }
+
+/*
+ +----------------+ +------------------+
+ | | 1 Sender | |
+ | |<-------------+ |
+ | User | | Message |
+ | |<-------------+ |
+ | | * Recipients | |
+ +----------------+ +------------------+
+*/
+
+var db = require('nodejsdb-impl');
+var storage = require('storage-impl');
+
+db.on.users = function(name, key, value, previous) {
+ if(!value) {
+
+ // handle deletion; cascade to Message.sender
+ db.range('messagesSentBy-' + key).forEach(function(v) {
+ db.set('messages', v.value, null);
+ });
+ }
+ if(!storage.loading)
+ storage.store(name, key, value);
+};
+
+db.on.messages = function(name, key, value, previous) {
+
+ if(!value) {
+
+ // handle Message deletion, remove from senders
+ db.set('messagesSentBy-' + previous.sender, key, null);
+
+ // remove from recipients
+ previous.recipients.forEach(function(v) {
+ db.set('messagesReceivedBy-' + v, key, null);
+ });
+ }
+
+ var r = value.recipients;
+ for(var i = 0, len = r.length, x = rnd(); i < len; i++)
+ db.set('messagesReceivedBy-' + r[i], value.date + '.' + (x + i), key);
+
+ db.set('messagesSentBy-' + value.sender, value.date + '.' + rnd(), key);
+
+ if(!storage.loading)
+ storage.store(name, key, value);
+};
+
+db.on.messagesReceivedBy = function(name, key, value, previous) {
+
+ // do something generic for the 'messagesReceivedBy-<userID>' collection class
+
+ var userID = parseInt(/-(\d+)^/.match(name)[1], 10);
+};
+
+// get incoming messages for a user, ordered from newest to oldest
+function getInboxForUser(userId) {
+ var messages = [];
+ db.range('messagesReceivedBy-' + userId, false).forEach(function(v) {
+ messages.push(db.get('messages', v.value));
+ });
+ return messages;
+}
+
+// get sent items for a user, ordered from newest to oldest
+function getSentItemsForUser(userId) {
+ var messages = [];
+ db.range('messagesSentBy-' + userId, false).forEach(function(v) {
+ messages.push(db.get('messages', v.value));
+ });
+ return messages;
+}
+
+function rnd() {
+ return Math.round(Math.random() * 10000);
+}
+
+storage.loading = true;
+storage.read('./db').each(function(col, key, val) {
+ db.set(col, key, val);
+});
+storage.loading = false;
+
+
+// DB is ready here.
+
+
+if(!db.size('users')) {
+ console.log('Creating data...');
+
+ db.set('user', 'user1', { login: 'user@one' });
+ db.set('user', 'user2', { login: 'user@two' });
+ db.set('user', 'user3', { login: 'user@three' });
+
+ db.set('message', db.size('messages') + 1, { text: 'hello 1', sender: 'user1', recipients: [ 'user2', 'user3' ]});
+ db.set('message', db.size('messages') + 1, { text: 'hello 2', sender: 'user2', recipients: [ 'user1', 'user3' ]});
+}
+
+console.log('Inbox for user1');
+getInboxForUser('user1').forEach(function(v) {
+ console.log(v.value);
+});
+
+console.log('Sent Items for user1');
+getSentItemsForUser('user1').forEach(function(v) {
+ console.log(v.value);
+});
+
+// yup, that's it!
+
+```
View
@@ -0,0 +1,32 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<Graphic version="2.0" viewHeight="727.141" viewWidth="784.836" ai:appVersion="15.0.0.399" ATE:version="1.0.0" flm:version="1.0.0" d:using="" xmlns="http://ns.adobe.com/fxg/2008" xmlns:ATE="http://ns.adobe.com/ate/2009" xmlns:ai="http://ns.adobe.com/ai/2009" xmlns:d="http://ns.adobe.com/fxg/2008/dt" xmlns:flm="http://ns.adobe.com/flame/2008">
+ <Library/>
+ <Group ai:seqID="1" d:layerType="page" d:pageHeight="727.141" d:pageWidth="784.836" d:type="layer" d:userLabel="Artboard 1">
+ <Group ai:seqID="2" d:type="layer" d:userLabel="Layer 1">
+ <Rect x="-2.39844" width="787.234" height="727.141" ai:seqID="3">
+ <fill>
+ <SolidColor color="#46483E"/>
+ </fill>
+ </Rect>
+ <Path x="76.6328" y="121.162" ai:seqID="4" data="M190.047 0.916458C187.805 -0.333542 185.07 -0.302292 182.844 0.994583 180.641 2.29927 179.273 4.67427 179.273 7.23677L179.273 191.229C179.273 193.041 178.312 194.713 176.75 195.612 175.18 196.526 173.25 196.526 171.672 195.612L141.648 178.315
+C137.164 175.737 131.648 175.745 127.18 178.315L7.23438 247.534C2.75781 250.127 0 254.901 0 260.081L0 398.541C0 403.713 2.75781 408.487 7.23438 411.088L127.164 480.354C131.648 482.94 137.172 482.94 141.648 480.354L261.609 411.088C266.086 408.487
+ 268.844 403.713 268.844 398.541L268.844 53.393C268.844 48.143 266.008 43.2993 261.414 40.7446L190.047 0.916458ZM178.922 353.041C178.922 354.338 178.227 355.526 177.109 356.174L135.922 379.916C134.805 380.565 133.438 380.565 132.32 379.916L91.1328
+ 356.174C90.0156 355.526 89.3203 354.338 89.3203 353.041L89.3203 305.502C89.3203 304.213 90.0156 303.01 91.125 302.362L132.312 278.581C133.438 277.932 134.805 277.932 135.93 278.581L177.109 302.362C178.227 303.01 178.922 304.213 178.922 305.502
+L178.922 353.041Z">
+ <fill>
+ <SolidColor color="#FFFFFF"/>
+ </fill>
+ </Path>
+ <Path x="440.805" y="121.162" ai:seqID="5" data="M7.42188 40.7446C2.83594 43.2993 0 48.143 0 53.393L0 398.541C0 403.713 2.75781 408.487 7.23438 411.088L127.188 480.354C131.672 482.94 137.188 482.94 141.68 480.354L261.602 411.088C266.086 408.487 268.844 403.713 268.844 398.541L268.844 260.081
+C268.844 254.901 266.086 250.127 261.602 247.534L141.664 178.315C137.188 175.745 131.68 175.737 127.188 178.315L97.1719 195.612C95.5938 196.526 93.6641 196.526 92.0938 195.612 90.5312 194.713 89.5703 193.041 89.5703 191.229L89.5703 7.23677C89.5703
+ 4.67427 88.2031 2.29927 86 0.994583 83.7734 -0.302292 81.0391 -0.333542 78.7969 0.916458L7.42188 40.7446ZM89.9219 305.502C89.9219 304.213 90.6172 303.01 91.7344 302.362L132.906 278.581C134.039 277.932 135.406 277.932 136.531 278.581L177.719 302.362
+C178.828 303.01 179.523 304.213 179.523 305.502L179.523 353.041C179.523 354.338 178.828 355.526 177.711 356.174L136.523 379.916C135.406 380.565 134.039 380.565 132.922 379.916L91.7344 356.174C90.6172 355.526 89.9219 354.338 89.9219 353.041L89.9219
+ 305.502Z">
+ <fill>
+ <SolidColor color="#8CC84B"/>
+ </fill>
+ </Path>
+ </Group>
+ </Group>
+ <Private/>
+</Graphic>

0 comments on commit 4431bb7

Please sign in to comment.