The Nebula file server

Introduction

Nebula is a prototype file store that uses a capability model for security and a content-addressable storage scheme for the underlying data. This particular version is a development demonstrator that aims to vet the core ideas.

The motivations and general concepts are explained in more depth in this blog post. The intro.org file in this repo contains additional description of the problem.

The project homepage may also be useful.

Prerequisites

You will need a Lisp interpreter and Quicklisp.

You will need a Postgres database running; the credentials should be in an alist in the file $HOME/.nebula.lisp. This alist should look like

;;; Example names taken from the postmodern docs.
((:DB-HOST "localhost")    ;; hostname of database
 (:DB-NAME "testdb")       ;; database name
 (:DB-PASS "surveiller")   ;; password
 (:DB-PORT 5432)           ;; port
 (:DB-USER "focault"))     ;; username

As it is expected that this will run on localhost, no SSL connections are used.

The database will need to exist and be writeable by the user.

This code should be stored in the Quicklisp local-projects directory so that it may be loaded with Quicklisp.

Using

Nebula is a library for exploring this idea of a filestore.

It should be cloned into the Quicklisp local-projects directory; then it may included as a dependency or loaded with (ql:quickload :nebula).

Once the credentials file is created, the package should be initialised with (nebula:initialize). It will create any tables it needs on startup.

The API

initialize

Lambda-list: (&KEY CRED-PATH STORE-PATH)

Conducts the necessary setup to begin using the filestore. If cred-path is not nil, it is used as the path to the Postgres credentials file (described in the setup section). If store-path is not nil, it is used as the path to the blob store.

retrieve

Lambda-list: (UUID)
Type: (FUNCTION (STRING) (OR (SIMPLE-ARRAY (UNSIGNED-BYTE 8)) NULL))

Given a UUID, follow its targets all the way to the underlying blob. If the UUID doesn’t exist or can’t be followed for some reason, it returns nil; otherwise, it returns a byte array containing underlying blob.

store

Lambda-list: (DATA &KEY (PARENT NIL))
Type: (FUNCTION ((OR STRING (SIMPLE-ARRAY (UNSIGNED-BYTE 8)))
       &KEY (:PARENT (OR STRING NULL)))
       (OR NULL STRING))

Store some data, possibly under a parent entry. If parent is not nil and is a valid UUID, the resulting entry will use parent as its parent. Otherwise, an entry with no history will be created.

expunge

Lambda-list: (UUID)
Derived type: (FUNCTION (STRING) (BOOLEAN))

Expunge the entry named by UUID from the system, garbage collecting as required. If any other entries have this entry as a parent, their parent entry will be cleared. Any proxy entries pointing to this one will be removed.

info

Lambda-list: (UUID)
Derived type: (FUNCTION (STRING) (LIST))

Returns an alist with the keys :id, :created, :size, and :parent, filled with the relevant information for the entry named by uuid.

lineage

Lambda-list: (UUID)
Derived type: (FUNCTION (STRING) CONS)

Return a list of entry IDs of this entry’s lineage. The car of this list will be current entry, and the last element will be the last parent.

proxy

Lambda-list: (UUID)
Derived type: (FUNCTION (STRING) (OR STRING NULL)

Proxy a single entry, removing its history. It returns the identifier for this proxied entry.

proxy-all

Lambda-list: (UUID)
Derived type: (FUNCTION (STRING) CONS)

Proxies an entire lineage for an entry. The history is preserved, though replaced with the proxied equivalent.

TODOs, thoughts, and considerations

• Work on additional front-ends
  • CLOS persistence store?
  • 9P interface?
  • An editor for collaborating and working on files over the network? Maybe an object-capable editor?
• Blobs are stored insecurely; does this matter? What’s a good way to fix this?
• Support for trees of history
• Resource restrictions
  • Quotas
  • File size restrictions
• The backends could use some looking at
  • Support for multiple backends
  • Options like elephant or manardb
  • Custom CLOS persistence store
• Nebula was designed for use on a single host. What would it look like to distribute this?
  • Does it make sense to deal with distribution at this level, or is better to do it at the interface level?
  • What topologies make senes?
  • Do all the nodes share binary data, or just tell each other which blobs and entries they have?
  • Namespacing?
• The documentation could better introduce the ideas and concepts behind this system.