Skip to content

A fast, low memory, transactional, index & query enabled NoSQL database engine and server for node.js and browser with realtime data change notifications

License

Notifications You must be signed in to change notification settings

appy-one/acebase

Repository files navigation

AceBase realtime database

AceBase realtime database

A fast, low memory, transactional, index & query enabled NoSQL database engine and server for node.js and browser with realtime data change notifications. Supports storing of JSON objects, arrays, numbers, strings, booleans, dates, bigints and binary (ArrayBuffer) data.

Inspired by (and largely compatible with) the Firebase realtime database, with additional functionality and less data sharding/duplication. Capable of storing up to 2^48 (281 trillion) object nodes in a binary database file that can theoretically grow to a max filesize of 8 petabytes.

AceBase is easy to set up and runs anywhere: in the cloud, NAS, local server, your PC/Mac, Raspberry Pi, the browser, wherever you want.

🔥 Check out the new live data proxy feature that allows your app to use and update live database values using in-memory objects and no additional db coding!

Table of contents

Getting Started

AceBase is split up into multiple packages:

  • acebase: local AceBase database engine (github, npm)
  • acebase-server: AceBase server endpoint to enable remote connections. Includes built-in user authentication and authorization, supports using external OAuth providers such as Facebook and Google (github, npm).
  • acebase-client: client to connect to an external AceBase server (github, npm)
  • acebase-core: shared functionality, dependency of above packages (github, npm)

AceBase uses semver versioning to prevent breaking changes to impact older code. Please report any errors / unexpected behaviour you encounter by creating an issue on Github.

Prerequisites

AceBase is designed to run in a Node.js environment, as it (by default) requires the 'fs' filesystem to store its data and indexes. However, since v0.9.0 it is now also possible to use AceBase databases in the browser! To run AceBase in the browser, simply include 1 script file and you're good to go! See AceBase in the browser for more info and code samples!

Installing

All AceBase repositories are available through npm. You only have to install one of them, depending on your needs:

Create a local database

If you want to use a local AceBase database in your project, install the acebase package.

npm install acebase

Then, create (open) your database:

const { AceBase } = require('acebase');
const db = new AceBase('my_db'); // nodejs
// OR: const db = AceBase.WithIndexedDB('my_db'); // browser
db.ready(() => {
    // Do stuff
});

Try AceBase in your browser

If you want to try out AceBase running in Node.js, simply open it in RunKit and follow along with the examples. If you want to try out the browser version of AceBase, open google.com in a new tab (GitHub doesn't allow cross-site scripts to be loaded) and run the code snippet below to use it in your browser console immediately.

To try AceBase in RunKit:

const { AceBase } = require('acebase');
const db = new AceBase('mydb');

await db.ref('test').set({ text: 'This is my first AceBase test in RunKit' });

const snap = await db.ref('test/text').get();
console.log(`value of "test/text": ` + snap.val());

To try AceBase in the browser console:

await fetch('https://cdn.jsdelivr.net/npm/acebase@latest/dist/browser.min.js')
    .then(response => response.text())
    .then(text => eval(text));
if (!AceBase) { throw 'AceBase not loaded!'; }

var db = AceBase.WithIndexedDB('mydb');
await db.ref('test').set({ text: 'This is my first AceBase test in the browser' });

const snap = await db.ref('test/text').get();
console.log(`value of "test/text": ` + snap.val());

Setup a database server

If you want to setup an AceBase server, install acebase-server.

npm install acebase-server

Then, start your server (server.js):

const { AceBaseServer } = require('acebase-server');
const server = new AceBaseServer('my_server_db', { /* server config */ });
server.ready(() => {
    // Server running
});

Connect to a remote database

If you want to connect to a remote (or local) AceBase server, install acebase-client.

npm install acebase-client

Then, connect to your AceBase server:

const { AceBaseClient } = require('acebase-client');
const db = new AceBaseClient({ /* connection config */ });
db.ready(() => {
    // Connected!
});

Example usage

The API is similar to that of the Firebase realtime database, with additions.

Creating a database

Creating a new database is as simple as opening it. If the database file doesn't exists, it will be created automatically.

const { AceBase } = require('acebase');
const options = { logLevel: 'log', storage: { path: '.' } }; // optional settings
const db = new AceBase('mydb', options);  // Creates or opens a database with name "mydb"

db.ready(() => {
    // database is ready to use!
})

NOTE: The logLevel option specifies how much info should be written to the console logs. Possible values are: 'verbose', 'log' (default), 'warn' and 'error' (only errors are logged)

Loading data

Run .get on a reference to get the currently stored value. This is short for the Firebase syntax of .once("value").

const snapshot = await db.ref('game/config').get();
if (snapshot.exists()) {
    config = snapshot.val();
}
else {
    config = defaultGameConfig; // use defaults
}

Note: When loading data, the currently stored value will be wrapped and returned in a DataSnapshot object. Use snapshot.exists() to determine if the node exists, snapshot.val() to get the value.

Storing data

Setting the value of a node, overwriting if it exists:

const ref = await db.ref('game/config').set({
    name: 'Name of the game',
    max_players: 10
});
// stored at /game/config

Note: When storing data, it doesn't matter whether the target path, and/or parent paths exist already. If you store data in 'chats/somechatid/messages/msgid/receipts', it will create any nonexistent node in that path.

Updating data

Updating the value of a node merges the stored value with the new object. If the target node doesn't exist, it will be created with the passed value.

const ref = await db.ref('game/config').update({
    description: 'The coolest game in the history of mankind'
});

// config was updated, now get the value (ref points to 'game/config')
const snapshot = await ref.get();
const config = snapshot.val();

// `config` now has properties "name", "max_players" and "description"

Transactional updating

If you want to update data based upon its current value, and you want to make sure the data is not changed in between your get and update, use transaction. A transaction gets the current value, runs your callback with a snapshot. The value you return from the callback will be used to overwrite the node with. Returning null will remove the entire node, returning nothing (undefined) will cancel the transaction.

await db.ref('accounts/some_account')
.transaction(snapshot => {
    // some_account is locked until its new value is returned by this callback
    var account = snapshot.val();
    if (!snapshot.exists()) {
        // Create it
        account = {
            balance: 0
        };
    }
    account.balance *= 1.02;    // Add 2% interest
    return account; // accounts/some_account will be set to the return value
});

Note: transaction loads the value of a node including ALL child objects. If the node you want to run a transaction on has a large value (eg many nested child objects), you might want to run the transaction on a subnode instead. If that is not possible, consider structuring your data differently.

// Run transaction on balance only, reduces amount of data being loaded, transferred, and overwritten
db.ref('accounts/some_account/balance')
.transaction(snapshot => {
    var balance = snapshot.val();
    if (balance === null) { // snapshot.exists() === false
        balance = 0;
    }
    return balance * 1.02;    // Add 2% interest
});

Removing data

You can remove data with the remove method

db.ref('animals/dog')
.remove()
.then(() => { /* removed successfully */ )};

Removing data can also be done by setting or updating its value to null. Any property that has a null value will be removed from the parent object node.

// Remove by setting it to null
db.ref('animals/dog')
.set(null)
.then(ref => { /* dog property removed */ )};

// Or, update its parent with a null value for 'dog' property
db.ref('animals')
.update({ dog: null })
.then(ref => { /* dog property removed */ )};

Generating unique keys

For all generic data you add, you need to create keys that are unique and won't clash with keys generated by other clients. To do this, you can have unique keys generated with push. Under the hood, push uses cuid to generated keys that are guaranteed to be unique and time-sortable.

db.ref('users')
.push({
    name: 'Ewout',
    country: 'The Netherlands'
})
.then(userRef => {
    // user is saved, userRef points to something 
    // like 'users/jld2cjxh0000qzrmn831i7rn'
};

The above example generates the unique key and stores the object immediately. You can also choose to have the key generated, but store the value later.

const postRef = db.ref('posts').push();
console.log(`About to add a new post with key "${postRef.key}"..`);
// ... do stuff ...
postRef.set({
    title: 'My first post'
})
.then(ref => {
    console.log(`Saved post "${postRef.key}"`);
};

NOTE: This approach is recommended if you want to add multitple new objects at once, because a single update performs way faster:

const newMessages = {};
// We got messages from somewhere else (eg imported from file or other db)
messages.forEach(message => {
    const ref = db.ref('messages').push();
    newMessages[ref.key] = message;
})
console.log(`About to add multiple messages in 1 update operation`);
db.ref('messages').update(newMessages)
.then(ref => {
    console.log(`Added all messages at once`);
};

Using arrays

AceBase supports storage of arrays, but there are some caveats when working with them. For instance, you cannot remove or insert items that are not at the end of the array. AceBase arrays work like a stack, you can add and remove from the top, not within. It is possible however to edit individual entries, or to overwrite the entire array. The safest way to edit arrays is with a transaction, which requires all data to be loaded and stored again. In many cases, it is wiser to use object collections instead.

You can safely use arrays when:

  • The number of items are small and finite, meaning you could estimate the typical average number of items in it.
  • There is no need to retrieve/edit individual items using their stored path. If you reorder the items in an array, their paths change (eg from "playlist/songs[4]" to "playlist/songs[1]")
  • The entries stored are small and do not have a lot of nested data (small strings or simple objects, eg: chat/members with user IDs array ['ewout','john','pete'])
  • The collection does not need to be edited frequently.

Use object collections instead when:

  • The collection keeps growing (eg: user generated content)
  • The path of items are important and preferably not change, eg "playlist/songs[4]" might point to a different entry if the array is edited. When using an object collection, playlist/songs/jld2cjxh0000qzrmn831i7rn will always refer to that same item.
  • The entries stored are large (eg large strings / blobs / objects with lots of nested data)
  • You have to edit the collection frequently.

Having said that, here's how to safely work with arrays:

// Store an array with 2 songs:
await db.ref('playlist/songs').set([
    { id: 13535, title: 'Daughters', artist: 'John Mayer' }, 
    { id: 22345,  title: 'Crazy', artist: 'Gnarls Barkley' }
]);

// Editing an array safely:
await db.ref('playlist/songs').transaction(snap => {
    const songs = snap.val();
    // songs is instanceof Array
    // Add a song:
    songs.push({ id: 7855, title: 'Formidable', artist: 'Stromae' });
    // Edit the second song:
    songs[1].title += ' (Live)';
    // Remove the first song:
    songs.splice(0, 1);
    // Store the edited array:
    return songs;
});

If you do not change the order of the entries in an array, it's safe to use them in referenced paths:

// Update a single array entry:
await db.ref('playlist/songs[4]/title').set('Blue on Black');

// Or:
await db.ref('playlist/songs[4]').update({ title: 'Blue on Black') };

// Or:
await db.ref('playlist/songs').update({
    4: { title: 'Blue on Black', artist: 'Kenny Wayne Shepherd' }
})

// Get value of single array entry:
let snap = await db.ref('playlist/songs[2]').get();

// Get selected entries with an include filter (like you'd use with object collections)
let snap = await db.ref('playlist/songs').get({ include: [0, 5, 8] });
let songs = snap.val();
// NOTE: songs is instanceof PartialArray, which is an object with properties '0', '5', '8'

NOTE: you CANNOT use ref.push() to add entries to an array! push can only be used on object collections because it generates unique child IDs such as "jpx0k53u0002ecr7s354c51l" (which obviously is not a valid array index)

To summarize: ONLY use arrays if using an object collection seems like overkill, and be very cautious! Adding and removing items can only be done to/from the END of an array, unless you rewrite the entire array. That means you will have to know how many entries your array has up-front to be able to add new entries, which is not really desirable in most situations. If you feel the urge to use an array because the order of the entries are important for you or your app: consider using an object collection instead, and add an 'order' property to the entries to perform a sort on.

Counting children

To quickly find out how many children a specific node has, use the count method on a DataReference:

const messageCount = await db.ref('chat/messages').count();

Limit nested data loading

If your database structure is using nesting (eg storing posts in 'users/someuser/posts' instead of in 'posts'), you might want to limit the amount of data you are retrieving in most cases. Eg: if you want to get the details of a user, but don't want to load all nested data, you can explicitly limit the nested data retrieval by passing exclude, include, and/or child_objects options to .get:

// Exclude specific nested data:
db.ref('users/someuser')
.get({ exclude: ['posts', 'comments'] })
.then(snap => {
    // snapshot contains all properties of 'someuser' except 
    // 'users/someuser/posts' and 'users/someuser/comments'
});

// Include specific nested data:
db.ref('users/someuser/posts')
.get({ include: ['*/title', '*/posted'] })
.then(snap => {
    // snapshot contains all posts of 'someuser', but each post 
    // only contains 'title' and 'posted' properties
});

// Combine include & exclude:
db.ref('users/someuser')
.get({ exclude: ['comments'], include: ['posts/*/title'] })
.then(snap => {
    // snapshot contains all user data without the 'comments' collection, 
    // and each object in the 'posts' collection only contains a 'title' property.
});

NOTE: This enables you to do what Firebase can't: store your data in logical places, and only get the data you are interested in, fast! On top of that, you're even able to index your nested data and query it, even faster. See Indexing data for more info.

Iterating (streaming) children

(NEW since v1.4.0)

To iterate through all children of an object collection without loading all data into memory at once, you can use forEach which streams each child and executes a callback function with a snapshot of its data. If the callback function returns false, iteration will stop. If the callback returns a Promise, iteration will wait for it to resolve before loading the next child.

The children to iterate are determined at the start of the function. Because forEach does not read/write lock the collection, it is possible for the data to be changed while iterating. Children that are added while iterating will be ignored, removed children will be skipped.

It is also possible to selectively load data for each child, using the same options object available for ref.get(options)

Examples:

// Stream all books one at a time (loads all data for each book):
await db.ref('books').forEach(bookSnapshot => {
   const book = bookSnapshot.val();
   console.log(`Got book "${book.title}": "${book.description}"`);
});

// Now do the same but only load 'title' and 'description' of each book:
await db.ref('books').forEach(
   { include: ['title', 'description'] }, 
   bookSnapshot => {
      const book = bookSnapshot.val();
      console.log(`Got book "${book.title}": "${book.description}"`);
   }
);

Also see Streaming query results

Asserting data types in TypeScript

If you are using TypeScript, you can pass a type parameter to most data retrieval methods that will assert the type of the returned value. Note that you are responsible for ensuring the value matches the asserted type at runtime.

Examples:

const snapshot = await db.ref<MyClass>('users/someuser/posts').get<MyClass>();
//                            ^ type parameter can go here,        ^ here,
if (snapshot.exists()) {
    config = snapshot.val<MyClass>();
    //                    ^ or here
}

// A type parameter can also be used to assert the type of a callback parameter
await db.ref('users/someuser/posts')
    .transaction<MyClass>(snapshot => {
        const posts = snapshot.val(); // posts is of type MyClass
        return posts;
    })

// Or when iterating over children
await db.ref('users').forEach<UserClass>(userSnapshot => {
    const user = snapshot.val(); // user is of type UserClass
})

Monitoring realtime data changes

You can subscribe to data events to get realtime notifications as the monitored node is being changed. When connected to a remote AceBase server, the events will be pushed to clients through a websocket connection. Supported events are:

  • 'value': triggered when a node's value changes (including changes to any child value)
  • 'child_added': triggered when a child node is added, callback contains a snapshot of the added child node
  • 'child_changed': triggered when a child node's value changed, callback contains a snapshot of the changed child node
  • 'child_removed': triggered when a child node is removed, callback contains a snapshot of the removed child node
  • 'mutated': (NEW v0.9.51) triggered when any nested property of a node changes, callback contains a snapshot and reference of the exact mutation.
  • 'mutations': (NEW v0.9.60) like 'mutated', but fires with an array of all mutations caused by a single database update.
  • 'notify_*': notification only version of above events without data, see "Notify only events" below
// Using event callback
db.ref('users')
.on('child_added', userSnapshot => {
    // fires for all current children, 
    // and for each new user from then on
});
// To be able to unsubscribe later:
function userAdded(userSnapshot) { /* ... */ }
db.ref('users').on('child_added', userAdded);
// Unsubscribe later with .off:
db.ref('users').off('child_added', userAdded);

AceBase uses the same .on and .off method signatures as Firebase, but also offers another way to subscribe to the events using the returned EventStream you can subscribe to. Having a subscription helps to easier unsubscribe from the events later. Additionally, subscribe callbacks only fire for future events by default, as opposed to the .on callback, which also fires for current values of events 'value' and 'child_added':

// Using .subscribe
const addSubscription = db.ref('users')
.on('child_added')
.subscribe(newUserSnapshot => {
    // .subscribe only fires for new children from now on
});

const removeSubscription = db.ref('users')
.on('child_removed')
.subscribe(removedChildSnapshot => {
    // removedChildSnapshot contains the removed data
    // NOTE: snapshot.exists() will return false, 
    // and snapshot.val() contains the removed child value
});

const changesSubscription = db.ref('users')
.on('child_changed')
.subscribe(updatedUserSnapshot => {
    // Got new value for an updated user object
});

// Stopping all subscriptions later:
addSubscription.stop();
removeSubscription.stop();
changesSubscription.stop();

If you want to use .subscribe while also getting callbacks on existing data, pass true as the callback argument:

db.ref('users/some_user')
.on('value', true) // passing true triggers .subscribe callback for current value as well
.subscribe(userSnapshot => {
    // Got current value (1st call), or new value (2nd+ call) for some_user
});

The EventStream returned by .on can also be used to subscribe more than once:

const newPostStream = db.ref('posts').on('child_added');
const subscription1 = newPostStream.subscribe(childSnapshot => { /* do something */ });
const subscription2 = newPostStream.subscribe(childSnapshot => { /* do something else */ });
// To stop 1's subscription:
subscription1.stop(); 
// or, to stop all active subscriptions:
newPostStream.stop();

If you are using TypeScript, you can pass a type parameter to .on or to .subscribe to assert the type of the value stored in the snapshot. This type is not checked by TypeScript; it is your responsibility to ensure that the value stored matches your assertion.

const newPostStream = db.ref('posts').on<MyClass>('child_added');
const subscription1 = newPostStream.subscribe(childSnapshot => {
    const child = childSnapshot.val(); // child is of type MyClass
 });
const subscription2 = newPostStream.subscribe<MyOtherClass>(childSnapshot => { 
    const child = childSnapshot.val(); // child is of type MyOtherClass
    // .subscribe overrode .on's type parameter
 });

### Using variables and wildcards in subscription paths

It is also possible to subscribe to events using wildcards and variables in the path:
```javascript
// Using wildcards:
db.ref('users/*/posts')
.on('child_added')
.subscribe(snap => {
    // This will fire for every post added by any user,
    // so for our example .push this will be the result:
    // snap.ref.vars === { 0: "ewout" }
    const vars = snap.ref.vars;
    console.log(`New post added by user "${vars[0]}"`)
});
db.ref('users/ewout/posts').push({ title: 'new post' });

// Using named variables:
db.ref('users/$userid/posts/$postid/title')
.on('value')
.subscribe(snap => {
    // This will fire for every new or changed post title,
    // so for our example .push below this will be the result:
    // snap.ref.vars === { 0: "ewout", 1: "jpx0k53u0002ecr7s354c51l", userid: "ewout", postid: (...), $userid: (...), $postid: (...) }
    // The user id will be in vars[0], vars.userid and vars.$userid
    const title = snap.val();
    const vars = snap.ref.vars; // contains the variable values in path
    console.log(`The title of post ${vars.postid} by user ${vars.userid} was set to: "${title}"`);
});
db.ref('users/ewout/posts').push({ title: 'new post' });

// Or a combination:
db.ref('users/*/posts/$postid/title')
.on('value')
.subscribe(snap => {
    // snap.ref.vars === { 0: 'ewout', 1: "jpx0k53u0002ecr7s354c51l", postid: "jpx0k53u0002ecr7s354c51l", $postid: (...) }
});
db.ref('users/ewout/posts').push({ title: 'new post' });

Notify only events

In additional to the events mentioned above, you can also subscribe to their notify_ counterparts which do the same, but with a reference to the changed data instead of a snapshot. This is quite useful if you want to monitor changes, but are not interested in the actual values. Doing this also saves serverside resources, and results in less data being transferred from the server. Eg: notify_child_changed will run your callback with a reference to the changed node:

ref.on('notify_child_changed', childRef => {
    console.log(`child "${childRef.key}" changed`);
})

Wait for events to activate

In some situations, it is useful to wait for event handlers to be active before modifying data. For instance, if you want an event to fire for changes you are about to make, you have to make sure the subscription is active before performing the updates.

var subscription = db.ref('users')
.on('child_added')
.subscribe(snap => { /*...*/ });

// Use activated promise
subscription.activated()
.then(() => {
    // We now know for sure the subscription is active,
    // adding a new user will trigger the .subscribe callback
    db.ref('users').push({ name: 'Ewout' });
})
.catch(err => {
    // Access to path denied by server?
    console.error(`Subscription canceled: ${err.message}`);
});

If you want to handle changes in the subscription state after it was activated (eg because server-side access rights have changed), provide a callback function to the activated call:

subscription.activated((activated, cancelReason) => {
    if (!activated) {
        // Access to path denied by server?
        console.error(`Subscription canceled: ${cancelReason}`);
    }
});

Get triggering context of events

(NEW v0.9.51)

In some cases it is benificial to know what (and/or who) triggered a data event to fire, so you can choose what you want to do with data updates. It is now possible to pass context information with all update, set, remove , and transaction operations, which will be passed along to any event triggered on affected paths (on any connected client!)

Imagine the following situation: you have a document editor that allows multiple people to edit at the same time. When loading a document you update its last_accessed property:

// Load document & subscribe to changes
db.ref('users/ewout/documents/some_id').on('value', snap => {
    // Document loaded, or changed. Display its contents
    const document = snap.val();
    displayDocument(document);
});

// Set last_accessed to current time
db.ref('users/ewout/documents/some_id').update({ last_accessed: new Date() })

This will trigger the value event TWICE, and cause the document to render TWICE. Additionally, if any other user opens the same document, it will be triggered again even though a redraw is not needed!

To prevent this, you can pass contextual info with the update:

// Load document & subscribe to changes (context aware!)
db.ref('users/ewout/documents/some_id')
    .on('value', snap => {
        // Document loaded, or changed.
        const context = snap.context();
        if (context.redraw === false) {
            // No need to redraw!
            return;
        }
        // Display its contents
        const document = snap.val();
        displayDocument(document);
    });

// Set last_accessed to current time, with context
db.ref('users/ewout/documents/some_id')
    .context({ redraw: false }) // prevent redraws!
    .update({ last_accessed: new Date() })

Change tracking using "mutated" and "mutations" events

(NEW v0.9.51)

These events are mainly used by AceBase behind the scenes to automatically update in-memory values with remote mutations. See Observe realtime value changes and Realtime synchronization with a live data proxy. It is possible to use these events yourself, but they require some additional plumbing, and you're probably better off using the methods mentioned above.

Having said that, here's how to use them:

If you want to monitor a specific node's value, but don't want to get its entire new value every time a small mutation is made to it, subscribe to the "mutated" event. This event is only fired with the target data actually being changed. This allows you to keep a cached copy of your data in memory (or cache db), and replicate all changes being made to it:

const chatRef = db.ref('chats/chat_id');
// Get current value
const chat = (await chatRef.get()).val();

// Subscribe to mutated event
chatRef.on('mutated', snap => {
    const mutatedPath = snap.ref.path; // 'chats/chat_id/messages/message_id'
    const propertyTrail = 
        // ['messages', 'message_id']
        mutatedPath.slice(chatRef.path.length + 1).split('/');

    // Navigate to the in-memory chat property target:
    let targetObject = propertyTrail.slice(0,-1).reduce((target, prop) => target[prop], chat);
    // targetObject === chat.messages
    const targetProperty = propertyTrail.slice(-1)[</