Store encrypted payloads and associated queryable/updateable JSON metadata
Branch: master
Clone or download
Latest commit ee0a5c0 Nov 30, 2018

README.md

Index page

What is it?

Web application that hosts encrypted payloads along with a JSON metadata store. Agnostic when it comes to use cases but I wrote it for hosting cloud-init payloads and orchestrating/coordinating multi-VM boot processes.

Why is it?

I got tired of having to work around cloud-init and user-data size limitations and having to set up consul or DNS just so I could orchestrate some VMs. Cloud-init-buddy acts as the initial glue for bootstrapping the rest of your infrastructure.

Requirements

Node, npm, gpg2, postgres, rake.

How to deploy

# We need ruby/rake so install it first
sudo apt update && sudo apt install --yes ruby
# Make sure we have the rest of the prerequisities
rake setup:initialize
# Flyway is used for migrations so make sure it is available
rake flyway:check || rake flyway:install
# Set up the user and authentication for postgres along with the flyway migrations
rake postgres:configure
# Install the application dependencies with NPM
npm install
# Compile from .ts to .js
./node_modules/.bin/tsc
# Generate the HTTPS key/certificate
node utils/generate-certificate.js
# Start the application and give it some time to start
tmux new-session -d -s cloud-init-buddy 'node app.js' && sleep 1
# Create a user because accessing the application requires user/pass or token
node utils/users.js add-user admin $(gpg2 --gen-random 0 64 | shasum -b -a 512 | head -c 32 | tee password)

Or you can just run

git clone https://github.com/cloudbootup/cloud-init-buddy.git
cd cloud-init-buddy
./init.sh

User / Token management

Users are managed with utils/users.js utility

$ node utils/users.js
# ...
add-user - Add a user: [username] [password]
remove-user - Remove a user: [username] [password]
add-token - Add a token for a user: [username] [password] [read|write|delete]+
remove-token - Remove a token belonging to user: [username] [password] [token]
list-users - List all the users
list-tokens - List all the tokens for a user: [username]

So to add a user just run

$ node utils/users.js add-user $user $pass

All users by default have administrative capabilities so for programmatic access I recommend creating tokens

$ node utils/users.js add-token $user $pass read

The above will create a token associated with the given user but the token can only be used for GET operations so POST and DELETE operations will not be allowed. You can provide a list of capabilities so if you wanted to give administrative capabilities to the token then you'd run

$ node utils/users.js add-token $user $pass read write delete

You use token by providing it in the HTTP header, e.g.

$ curl -H "X-CLOUDBOOTUP-TOKEN: $token" ...

Decrypting Packages

The packages are encrypted with GPG's symmetric mode (AES256) and the key is next to the package name in the UI. After downloading the package you can decrypt with

$ gpg2 --batch --yes --no-tty --passphrase-file <(echo $key) -d $file > $file.decrypted

API Endpoints

GET /index

Go the the UI to see all the namespaces and packages. You will need to log in so if you are not logged in you will be redirected to login page.

POST /packages/:package/:version

Upload a file for later retrieval. Package and version are just conventions and basically give you two degress of freedom in how you want to structure your packages. The package will be encrypted in the background asynchronously so we don't know ahead of time what the encryption key will be. You can query /keys/:package to retrieve the key.

To upload a file with curl

curl -k -XPOST -F 'file=@file' https://$user:$pass@$ip:8443/packages/$package/$version

GET /packages/:package/:version

Get the specific version of a package.

DELETE /packages/:package/:version

Delete the specific version of the package. If the last version of a package is deleted then the key and directory for the package is also deleted.

GET /packages/:package

Grab the list of all the versions for this package. Response is a JSON string array.

DELETE /packages/:package

Delete all the versions of the package along with the key.

GET /packages

Get a list of all the packages. Response is a JSON string array.

GET /keys/:package

Get the encryption key for the given package. Different packages have different keys but different versions of the same package use the same key. The key is initialized lazily when the first package is uploaded. Using the same package key for different versions of the same package is just a convenience so if you want different keys for all the versions then encode the version as part of the package name and make the version a dummy variable, i.e. always upload your packages to something like /packages/:package-:version/1 and you will always get a new key.

GET /metadata

List all the metadata namespaces. Response is a JSON string array.

POST /metadata/:namespace/:path

Store an arbitrary JSON object at the given path for the given namespace. The path consists of strings and numbers, e.g. /metadata/production/one/two/3. If the objects stored at the namespace was {one: {two: ['a', 'b', 'c', 'd', 'e']}} then the previous example would return 'd' as the answer if a GET request was sent. NOTE: You have to initialize the namespace first by posting the desired form of the object at the root of the namespace without any path components following the name. If you don't do this then you won't be able to post updates to the namespace.

It is worthwhile to experiment with the metadata endpoint to get a good feel for what is possible and what happens when the same keys are used. Behind the scenes the code tries to be safe by using append operations even for keys that do not seem initially to be appendable. For example,

# Initialize the namespace
$ curl -k -XPOST -d '{"key":"value"}' https://$user:$pass@$ip:8443/metadata/test
# Grab what's there
$ curl -k https://$user:$pass@$ip:8443/metadata/test
{"key":"value"}
# This will not modify "key" but will instead "append" to it
$ curl -k -XPOST -d '"value2"' https://$user:$pass@$ip:8443/metadata/test/key
# Grab what's there and notice how we "appended" a new value
$ curl -k https://$user:$pass@$ip:8443/metadata/test
{"key":["value","value2"]}
# Reset the key
$ curl -k -XPOST -d '{"key":""}' https://$user:$pass@$ip:8443/metadata/test
# Verify
$ curl -k https://$user:$pass@$ip:8443/metadata/test
{"key":""}

You might find this behavior undesirable but when coordinating nodes it is better to have atomic append behavior for all keys. This way when the same key is modified concurrently you get to see the list of all attempted modifications instead of just whichever update was last.

DELETE /metadata/:namespace/:path

If you want to delete keys in a namespace then provide a path to the key you want to delete. Omitting the path will delete the entire namespace, i.e. all keys and objects will be dropped.

GET /metadata/:namespace/:path

Grab the JSON object in the given namespace and the given path. See above for an explanation of how the path works. If you append .keys or .length to the last path component or to the namespace then you will get a list of keys for the object or the length of the array. If you try to get the length of an object then you will get an empty string back.