Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
239 lines (158 sloc) 7.65 KB

BlobStash

Travis    Go Report Card    License

Your personal database.

Still in early development.

Manifesto

BlobStash is primarily a database, you can store raw blobs, key-value pairs, JSON documents and files/directories.

It can also acts as a web server/reverse proxy.

The web server supports HTTP/2 and can generate you TLS certs on the fly using Let's Encrypt. You can proxy other applications and gives them free certs at the same time, you can also write apps (using Lua) that lets you interact with BlobStash's database. Hosting static content is also an option. It let you easily add authentication to any app/proxied service.

Blobs

The content-addressed blob store (the identifier of a blob is its own hash, the chosen hash function is BLAKE2b) is at the heart of everything in BlobStash. Everything permanently stored in BlobStash ends up in a blob.

BlobStash has its own storage engine: BlobsFile, data is stored in an append-only flat file. All data is immutable, stored with error correcting code for bit-rot protection, and indexed in a temporary index for fast access, only 2 seeks operations are needed to access any blobs.

The blob store supports real-time replication via an Oplog (powered by Server-Sent Events) to replicate to another BlobStash instance (or any system), and also support efficient synchronisation between instances using a Merkle tree to speed-up operations.

Key-values

Key-value pairs lets you keep a mutable reference to an internal or external object, it can be a hash and/or any sequence of bytes.

Each key-value has a timestamp associated, its version. you can easily list all the versions, by default, the latest version is returned. Internally, each "version" is stored as a separate blob, with a specific format, so it can be detected and re-indexed.

Key-Values are indexed in a temporary database (that can be rebuilt at any time by scanning all the blobs) and stored as a blob.

JSON documents

A Lua-powered JSON document store lets you perform powerful queries against a collection of JSON documents.

You can easily reference/embed blob or files.

Internally, each document gets a key-value entry, keeping track of the modification history and documents are stored as raw blobs.

When performing queryies, the embedded Lua interpreter runs through all documents sequentially, and returns you the results (indexes support is on its way).

The document store supports ETag, conditional requests (If-Match...) and JSON Patch for partial/consistent update.

Complex queries can be stored along with the server to prevent wasting bandwith.

Files, tree of files

Files and tree of files are first-class citizen in BlobStash.

Files are split in multiple chunks (stored as blobs, using content-defined chunking, giving deduplication at the file level), and everything is stored in a kind of Merkle tree where the hash of the JSON file containing the file metadata is the final identifier (which will also be stored as blob).

The JSON format also allow to model directory. A regular HTTP multipart endpoint can convert file to BlobStash internal format for you, or you can do it locally to prevent sending blobs that are already present.

Files can be streamed easily, range requests are supported, EXIF metadata automatically extracted and served, and on-the-fly resizing (with caching) for images.

You can also enable a S3 compatible gateway to manage your files.

Role Based Access Control (RBAC)

BlobStash features fine-grained permissions support, with a model similar to AWS roles.

Predefined roles

  • admin: full access to everything
    • action:*/resource:*
  • git-admin: full access to the Git API, including the Git smart HTTP support
    • action:read:git-repo/resource:gitserver:git-repo:*
    • action:write:git-repo/resource:gitserver:git-repo:*
    • action:list:git-ns/resource:gitserver:git-ns:*

Templated roles

  • git-ro: read-only access (clone)
    • action:read:git-repo/resource:gitserver:git-repo:{ns}/{repo}
  • git: read-write access (clone and push)
    • action:read:git-repo/resource:gitserver:git-repo:{ns}/{repo}
    • action:write:git-repo/resource:gitserver:git-repo:{ns}/{repo}

Use Cases

Backups from external servers

Setup an API key with limited permissions (in blobstash.yaml), just enough to save a snapshot of a tree:

# [...]
auth:
 - id: 'my_backup_key'
   password: 'my_api_key'
   roles: 'backup_server1'
roles:
 - name: 'backup_server1'
   perms:
    - action: 'action:stat:blob'
      resource: 'resource:blobstore:blob:*'
    - action: 'action:write:blob'
      resource: 'resource:blobstore:blob:*'
    - action: 'action:snapshot:fs'
      resource: 'resource:filetree:fs:server1'
    - action: 'action:write:kv'
      resource: 'resource:kvstore:kv:_filetree:fs:server1'
    - action: 'action:gc:namespace'
      resource: 'resource:stash:namespace:server1'

Then on "server1":

$ export BLOBS_API_HOST=https://my-blobstash-instance.com BLOBS_API_KEY=my_api_key
$ blobstash-uploader server1 /path/to/data

Git smart HTTP backend

You can store Git repositories via Git smart HTTP, and all the data will be deduplicated (at the chunk level and even across FileTree trees).

Getting Started

To backup a Git repository, just add a new remote (new repositories will be created automatically):

$ git remote add blobstash https://tom:mypass@myinstance.com/git/myns/myrepo.git
$ git push blobstash

For big repositories, you may need to tweak the maximum body size for HTTP requests:

$ git config --global http.postBuffer 1048576000

To restore a Git repository:

$ git clone https://thomas:123@myinstance.com/git/myns/myrepo.git

You can also access Git repositories using the Admin UI.

HTTP API

All the examples are using HTTPie.

GET /api/git

List all the namespaces.

Example
$ http --auth :apikey GET https://myinstance.com/api/git
Response
{
    "data": [
        "myns"
    ], 
    "pagination": {
        "count": 1, 
        "cursor": "", 
        "has_more": false, 
        "per_page": 50
    }
}

GET /api/git/:ns

List all the repositories for the given namespace.

Example
$ http --auth :apikey GET https://myinstance.com/api/git/myns
Response
{
    "data": [
        "myrepo"
    ], 
    "pagination": {
        "count": 1, 
        "cursor": "", 
        "has_more": false, 
        "per_page": 50
    }
}

Lua API

Extra module

extra.glob(pattern, name)

Parses the shell file name pattern/glob and reports wether the file name matches.

Uses go's filepath.Match.

Attributes

Name Type Description
pattern String Glob pattern
name String file name

Returns

Boolean

Contribution

Pull requests are welcome but open an issue to start a discussion before starting something consequent.

Feel free to open an issue if you have any ideas/suggestions!

License

Copyright (c) 2014-2018 Thomas Sileo and contributors. Released under the MIT license.

You can’t perform that action at this time.