AWSDOWN

A LevelDOWN API implementation on Amazon DynamoDB and Amazon S3.

This is a drop-in replacement for LevelDOWN that uses Amazon DynamoDB for object storage and Amazon S3 for primitive value storage. It can be used as a backend for LevelUP rather than an actual LevelDB store.

As of version 0.7, LevelUP allows you to pass a db option when you create a new instance. This will override the default LevelDOWN store with a LevelDOWN API compatible object. AWSDOWN conforms exactly to the LevelDOWN API, but performs operations against a DynamoDB database.

Why?

This LevelDOWN implementation is different from others such as AWSDOWN in that objects are cast to actual column types in DynamoDB, and primitive values(e.g. string, number, boolean, buffer/blob) are stored in S3; this allows objects to be natively queryable with DynamoDB, and S3 allows us to store data beyond the 400kb maximum record size that DynamoDB imposes.

The intended use case for this library is with PouchDB, where most database activity involves saving objects(i.e. documents) while a few things such as attachments and database info are better suited for a storage engine like S3. Compatibility with PouchDB is a big win in this case since it provides a common JavaScript interface for interacting with documents as well as full replication, including attachments of any size. Using this LevelDOWN implementation wtih PouchDB can be useful for regular backups as well as migrating data to CouchDB.

Usage Example

const levelup  = require('levelup'),
      AWSDOWN  = require('dynamodbdown'),
    { DynamoDB
      S3 }     = require('aws-sdk');

const options = {
  db: AWSDOWN({
    dynamoDb: new DynamoDB({
      region: 'us-west-1',
      secretAccessKey: 'foo',
      accessKeyId: 'bar'
    }),
    s3: new S3({
      region: 'us-west-1',
      secretAccessKey: 'foo',
      accessKeyId: 'bar'
    })
  })
};

const db = levelup('tableName', options);

db.put('some string', 'LevelUP string');
db.put('some binary', new Buffer('LevelUP buffer'));

const dbReadStream = db.createReadStream();

dbReadStream.on('data', console.log);
dbReadStream.on('close', () => { console.log('read stream closed') });

When running the above example, you should get the following console output:

{ key: 'some binary', value: 'LevelUP buffer' }
{ key: 'some string', value: 'LevelUP string' }
read stream closed

Hash Keys

In DynamoDB, keys consist of two parts: a hash key and a range key. To achieve LevelDB-like behaviour, all keys in a database instance are given the same hash key. That means that you can't do range queries over keys with different hash keys.

The default hash key is !. You can specify it by putting a $ in the location argument. The $ separates the table name from the hash key.

Example

const levelup  = require('levelup'),
      AWSDOWN  = require('dynamodbdown'),
    { DynamoDB
      S3 }     = require('aws-sdk');

const options = {
  db: AWSDOWN({
    dynamoDb: new DynamoDB({
      region: 'us-west-1',
      secretAccessKey: 'foo',
      accessKeyId: 'bar'
    }),
    s3: new S3({
      region: 'us-west-1',
      secretAccessKey: 'foo',
      accessKeyId: 'bar'
    })
  })
};

const db = levelup('tableName', options);

const db = levelup('tableName$hashKey', options)

db.put('some key', 'some value', => err {
  // the DynamoDB object would now look like this:
  // {
  //   '---hkey': 'hashKey',
  //   '---rkey': 'some key',
  // }
});

If you are fine with sharing capacity units across multiple database instances or applications, you can reuse a table by specifying the same table name, but different hash keys.

Table & Bucket Creation

If the table doesn't exist, AWSDOWN will try to create a table. You can specify the read/write throughput. If not specified, it will default to 1/1. If the table already exists, the specified throughput will have no effect. Throughput can be changed for tables that already exist by using the DynamoDB API or the AWS Console.

AWSDOWN will also attempt to create an S3 bucket if it doesn't already exist.

See LevelUP options for more information.

Example

const levelup = require('levelup')
const AWSDOWN = require('dynamodbdown')

const dynamoDBOptions = {
  region: 'eu-west-1',
  secretAccessKey: 'abc',
  accessKeyId: '123',
  ProvisionedThroughput: { // capacity can be specified; defaults to 1/1:
    ReadCapacityUnits: 1,
    WriteCapacityUnits: 1
  }
}

const options = {
  db: AWSDOWN,
  dynamodb: dynamoDBOptions // required AWS configuration
}

const db = levelup('tableName', options)

Table Name Encoding

AWSDOWN encodes table names in hexadecimal if you set the dynamodb.hexEncodeTableName option to true. This can be useful if you'd like pass location parameter values to levelup that aren't compatible with DynamoDB's restrictions on table names (see here).

Example

const levelup    = require('levelup'),
      AWSDOWN    = require('dynamodbdown'),
    { DynamoDB } = require('aws-sdk');

const options = {
  db: AWSDOWN({
    dynamoDb: new DynamoDB({
      region: 'us-west-1',
      secretAccessKey: 'foo',
      accessKeyId: 'bar'
    }),
    s3: new S3({
      region: 'us-west-1',
      secretAccessKey: 'foo',
      accessKeyId: 'bar'
    })
  })
};

const db = levelup('tableName', options) // the DynamoDB table name will
                                         // be '7461626c654e616d65'

Other Considerations

S3 provides read-after-write consistency when PUTing a new file, but provides eventual consistency for overwrite PUTs and DELETEs.

This library may not be suitable for multi-process database access, since there is no mechanism for locking DynamoDB tables or S3 buckets. If you find you need to have multiple processes access your database, it will be necessary to maintain direct-access on a single thread and have other processes communicate with that instance. Using multilevel is one premade way of achieving this.

Changelog

See here.

Acknowledgments

AWSDOWN has been heavily inspired by:

• DynamoDBDOWN by Klaus Trainer

LICENSE

Copyright 2018 Ben Titcomb

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.