Add documentation

Crystal-SDS · Oct 24, 2016 · 2e4d5e0 · 2e4d5e0
1 parent 4dbd5c3
commit 2e4d5e0
Show file tree

Hide file tree

Showing 6 changed files with 1,017 additions and 175 deletions.
diff --git a/README.md b/README.md
@@ -1,11 +1,28 @@
+# Crystal
+
+[Crystal](http://crystal-sds.org/) is a transparent, dynamic and open Software-Defined Storage (SDS) system for [OpenStack Swift](http://swift.openstack.org).
+It is structured in several sub-projects:
+
+* [Controller](https://github.com/Crystal-SDS/controller): the Crystal control plane that offers dynamic meta-programming facilities over the data plane.
+
+* [Introspection middleware](https://github.com/Crystal-SDS/introspection-middleware): the inspection triggers (data plane), that enable controllers to dynamically respond to workload changes in real time.
+
+* [Filter middleware](https://github.com/Crystal-SDS/filter-middleware): the storage filters (data plane), that intercept object flows and run computations or perform transformations on them.
+
+* [Dashboard](https://github.com/iostackproject/SDS-dashboard/tree/urv_dev): A user-friendly dashboard to manage policies, filters and workload metrics.
+
+![alt text](http://crystal-sds.org/wp-content/uploads/2016/05/architecture9-768x575.png "Crystal Architecture")
+
+# Crystal Controller
+
 [![Build Status](https://travis-ci.org/Crystal-SDS/controller.svg?branch=master)](https://travis-ci.org/Crystal-SDS/controller)
 [![Coverage Status](https://coveralls.io/repos/github/Crystal-SDS/controller/badge.svg?branch=master)](https://coveralls.io/github/Crystal-SDS/controller?branch=master)
 [![Code Health](https://landscape.io/github/Crystal-SDS/controller/master/landscape.svg?style=flat)](https://landscape.io/github/Crystal-SDS/controller/master)
 
-
-# Crystal-Controller
-
-This repository contains the code of the SDS Controller for Object Storage in the IOStack architecture. The SDS Controller repository contains two differentiated parts: **The SDS Controller API** and the **Dynamic Policies**. The **SDS Controller API** is a Django project that implements the REST API needed to handle the Storlets and the BW Differentiation. Otherwise, **Dynamic Policies** is a set of python processes who use the [PyActive middleware](https://github.com/cloudspaces/pyactive) (an Object Oriented implementation of the Actor model). This part allows to create simple policies using a DSL (integrated in the SDS Controller API) and deploys them as an actor process, who analyze the system data thanks to the monitoring system, and allows to set or remove filters to tenants depending on the established policy.
+This repository contains the code of Crystal Controller, the Software-Defined-Storage (SDS) REST API in the [IOStack](https://github.com/iostackproject) architecture.
+It is a Django project that implements the REST API needed to handle filters, storlets and policies on top of Openstack Swift object-storage system. This API also includes a set of python processes who use
+the [PyActive middleware](https://github.com/cloudspaces/pyactive), an Object Oriented implementation of the Actor model. This part allows to create simple policies using a DSL (integrated in the Crystal Controller API)
+and to deploy them as an actor process, who analyze the system data thanks to the monitoring system, and allows to set or remove filters to tenants depending on the established policy.
 
 The repository is structured with the next folders:
 
@@ -15,33 +32,49 @@ The repository is structured with the next folders:
 
 * **scripts:** The scripts folder contains all the scripts needed for the project. The file vagrant-init.sh will be executed each time that you start the virtual machine using vagrant.
 
-* **sds_controller:** The sds_controller contains the source code. Its structure follows a standard Django project structure.
+* **api:** The folder contains the API source code. Its structure follows a standard Django project structure.
 
 * **dynamic_policies** The dynamic_policies contains the source code of this part.
 
-* **Vagrantfile:** This is the vagrant config file, where we define all the information that vagrant need to start a virtual machine with all the requirements.  
+* **Vagrantfile:** This is the vagrant config file, where we define all the information that vagrant needs to start a virtual machine with all the requirements.
 
 To build the APIs in an easy way we use [Django REST Framework](http://www.django-rest-framework.org/).
 
-# Requirements
+## Requirements
 
-This project only has two requirements:
+This project includes a Vagrant environment with all the dependencies (Django, pyactive, swiftclient, ...), so the only two requirements are:
 
 1. Install Virtual Box [Visit VirtualBox page!](https://www.virtualbox.org/)
 2. Install Vagrant [Visit Vagrant page!](https://www.vagrantup.com/downloads.html)
 
-That's all! You don't need Django or Python... Vagrant resolves this problem for us.
+## Installation
 
-# Installation
+Once you have already installed the requirements, you only need to go to the project location using a terminal, and execute the command: `vagrant up`. First time, the process may take a few minutes because Vagrant downloads the Operative System to create the Virtual Machine. Next time the process will be faster.
 
-Once you have already installed the requirements, you only need to go in the folder location using a terminal, and execute the command: `vagrant up`. First time, the process may take a few minutes because Vagrant downloads the Operative System to create the Virtual Machine. Next time the process will be faster.
+The Virtual Machine that we started has all the tools that we need to run the server. To connect to this machine you only need to run the command `vagrant ssh`. The repository folder is synchronized between the local machine and the Virtual Machine, so you can develop the code in your local machine with your preferred IDE, and run the project in the Virtual Machine.
 
-The Virtual Machine that we started has all the tools that we need to run the server. To connect to this machine you only need to run the command `vagrant ssh`. The repository folder is synchronized between your machine and the Virtual Machine, so you can develop the code in your local machine with your prefer IDE, and run the project in the Virtual Machine.
+You can start the server using the command into the source folder (./api): `python manage.py runserver 0.0.0.0:8000`. After that, the server starts, and if you prefer to call Crystal controller from the host machine the port to use is `18000`. For instance, to list the Storlets from the host machine the URL should be: localhost:18000/storlets.
 
-You can start the server using the command into the source folder (src/sds_controller): `python manage.py runserver 0.0.0.0:8000`. After that the server starts, and if you prefer to call the SDS controller for your machine the port to use is `18000`. For instance, to call to list the Storlets from your machine the url will be: localhost:18000/storlets. *We have in the TODO list to configure vagrant and puppet to do a deploy of the SDS Controller in Apache each time that starts the Virtual Machine.*
+If some problem appears, make sure that:
 
-If some problem appear, make sure..
-
-1. redis-server service is running? Start this service, because the SDS Controller API stores the meta-data information in redis.
-2. is PyActive in the PYTHONPATH? At home folder you can found the pyactive folder, where you can find another install.txt, please follow this steps.
+1. redis-server service is running? SDS Controller API stores the meta-data information in redis.
+2. is PyActive in the PYTHONPATH? At home folder you can find the pyactive folder, where you can find another install.txt, please follow this steps.
 3. review the settings file from SDS Controller and make sure to write the correct IPs (Swift IP, Keystone IP, PyActive IP)
+
+## Usage
+
+API usage is detailed in the [API specification](/doc/api_specification.md).
+
+A convenient [web dashboard](https://github.com/iostackproject/SDS-dashboard) is also available to simplify these API calls. Refer to the [dashboard overview](/doc/dashboard_overview.md) for detailed information.
+
+### Tests
+
+Run unit tests from the source folder (`./api`) with the following command: `python manage.py test`
+
+## Support
+
+Please [open an issue](https://github.com/Crystal-SDS/controller/issues/new) for support.
+
+## Contributing
+
+Please contribute using [Github Flow](https://guides.github.com/introduction/flow/). Create a branch, add commits, and [open a pull request](https://github.com/Crystal-SDS/controller/compare/).
diff --git a/doc/api_dsl.md b/doc/api_dsl.md
@@ -0,0 +1,27 @@
+Crystal DSL Grammar
+===================
+
+## Overview
+
+![alt text](http://crystal-sds.org/wp-content/uploads/2016/05/policies-768x180.png "Crystal DSL structure")
+
+SDS policies are means of defining storage services or objectives to be achieved by the system. 
+Crystal DSL hides the complexity of low-level policy enforcement, thus achieving simplified storage administration. 
+The structure of our DSL is as follows:
+
+Target: The target of a policy represents the recipient of a policy’s action (e.g., filter enforcement) and it is mandatory to specify it on every policy definition. In our DSL design, targets can be tenants (projects), containers or even individual data objects.
+
+Trigger (optional): Dynamic storage automation policies are characterized by the trigger clause. A policy may have one or more trigger clauses—separated by AND/OR operands—that specify the workload-based situation that will trigger the enforcement of a filter on the target. 
+Condition clauses consist of inspection triggers, operands (e.g, >, <, =) and values.
+
+Action: The action clause of a SDS policy defines how a filter should be executed on an object request once the policy takes place. 
+To this end, the action clause may accept parameters after the WITH keyword in form of key/value pairs that will be passed as input to the filter execution. 
+Retaking the example of a compression filter, we may decide to enforce compression using a gzip or an lz4 engine, and even to decide the compression level of these engines. 
+In Crystal, filters can be executed at the proxy or at storage nodes (i.e., stages). Our DSL enables to explicitly control the execution stage of a filter in the action clause with the ON keyword. 
+For instance, in P1 at the previous figure we may want to execute compression on the proxy to save up bandwidth (ON PROXY) and encrypt the compressed data object on the storage nodes (ON STORAGE_NODE).
+
+Moreover, dynamic storage automation policies can be persistent or transient; a persistent action means that once the policy is triggered the filter enforcement remains indefinitely (keyword PERSISTENT, default), whereas actions to be executed only during the period where the condition is satisfied are transient (keyword TRANSIENT). 
+
+## Grammar
+
+TODO
diff --git a/doc/api_specification_filters.md b/doc/api_specification_filters.md
@@ -10,6 +10,7 @@ Crystal Controller API Specification - Filters
   - [Get Filter metadata](#get-filter-metadata)
   - [Update Filter metadata](#update-filter-metadata)
   - [Deploy a Filter](#deploy-a-filter)
+  - [Undeploy a Filter](#undeploy-a-filter)
 - [Dependencies](#dependencies)
   - [Create a Dependency](#create-a-dependency)
   - [Upload a Dependency Data](#upload-a-dependency-data)
@@ -159,7 +160,7 @@ An application can upload a filter data by issuing an HTTP PUT request. The appl
 
 #### URL structure
 The URL that represents the filter data resource. The URL is
-**/filters/:filter_id/data**.
+**/filters/{filter_id}/data**.
 
 #### Method
 PUT
@@ -190,7 +191,7 @@ An application can delete a filter by issuing an HTTP DELETE request. This call
 
 #### URL structure
 The URL that represents the filter data resource. The URL is
-**/filters/:filter_id**
+**/filters/{filter_id}**
 
 #### Method
 DELETE
@@ -209,7 +210,7 @@ An application can ask for the filter metadata by issuing an HTTP GET request.
 
 #### URL structure
 The URL that represents the filter data resource. The URL is
-**/filters/:filter_id**
+**/filters/{filter_id}**
 
 #### Method
 GET
@@ -254,7 +255,7 @@ An application can update the filter metadata by issuing an HTTP PUT request.
 
 #### URL structure
 The URL that represents the filter data resource. The URL is
- **/filters/:filter_id**
+ **/filters/{filter_id}**
 
 #### Method
 PUT
@@ -318,10 +319,10 @@ An application can deploy a filter to Swift by issuing an HTTP PUT request. This
 
 #### URL structure
 The URL that represents the deployment of a filter to a project. The URL is
-**/filters/:project/deploy/:filter_id/**
+**/filters/{project}/deploy/{filter_id}/**
 
 Similarly, to deploy a filter to a project and a container, the URL is
-**/filters/:project/:container/deploy/:filter_id/**
+**/filters/{project}/{container}/deploy/{filter_id}/**
 
 #### Method
 PUT
@@ -362,6 +363,27 @@ HTTP/1.1 201 Created
 1
 ```
 
+## Undeploy a Filter
+
+An application can undeploy the filter from a Swift project by issuing an HTTP PUT request.
+
+### Request
+
+#### URL structure
+The URL that represents the filter data resource. The URL is
+**/filters/{project}/undeploy/{filter_id}/**
+
+#### Method
+PUT
+
+#### HTTP Request Example
+
+```json
+Content-Type: application/json
+PUT /filters/4f0279da74ef4584a29dc72c835fe2c9/undeploy/3
+
+```
+
 # Dependencies
 ## Create a Dependency
 
@@ -370,7 +392,7 @@ An application can create a Dependency by issuing an HTTP POST request. The appl
 ### Request
 
 #### URL structure
-The URL that represents the filter data resource. The URL is
+The URL that represents the filter dependencies resource. The URL is
 **/filters/dependencies.**
 #### Method
 POST
@@ -424,8 +446,8 @@ An application can upload a Dependency data by issuing an HTTP PUT request. The
 ### Request
 
 #### URL structure
-The URL that represents the filter data resource. The URL is
-**/filters/dependencies/:dependency_id/data.**
+The URL that represents a filter dependency resource. The URL is
+**/filters/dependencies/{dependency_id}/data**
 
 #### Method
 PUT
@@ -457,8 +479,8 @@ An application can delete a Dependency by issuing an HTTP DELETE request. This c
 
 #### URL structure
 
-The URL that represents the filter data resource. The URL is
-**/filters/dependencies/:filter_id.**
+The URL that represents a filter dependency resource. The URL is
+**/filters/dependencies/{dependency_id}**
 
 #### Method
 DELETE
@@ -478,8 +500,8 @@ An application can ask for the Dependency metadata by issuing an HTTP GET reques
 
 #### URL structure
 
-The URL that represents the filter data resource. The URL is
-**/filters/dependencies/:filter_id**
+The URL that represents a filter dependency resource. The URL is
+**/filters/dependencies/{dependency_id}**
 
 #### Method
 GET
@@ -516,8 +538,8 @@ An application can list the Dependencies by issuing an HTTP GET request.
 
 #### URL structure
 
-The URL that represents the filter data resource. The URL is
-**/sotrlets/dependencies**
+The URL that represents filter dependencies resource. The URL is
+**/filters/dependencies**
 
 #### Method
 GET
@@ -567,9 +589,8 @@ An application can update the Dependency metadata by issuing an HTTP PUT request
 
 #### URL structure
 
-The URL that represents the dependency data resource. The URL is
-**/filters/dependencies/:dependency_id**
-
+The URL that represents a filter dependency resource. The URL is
+**/filters/dependencies/{dependency_id}**
 
 #### Method
 PUT
@@ -615,16 +636,15 @@ An application can deploy a Dependency to an account to Swift by issuing an HTTP
 ### Request
 
 #### URL structure
-The URL that represents the dependency data resource. The URL is
-**/filters/dependencies/:account/deploy/:dependency_id/**
+The URL that represents the a filter dependency resource. The URL is
+**/filters/dependencies/{project}/deploy/{dependency_id}/**
 
 #### Method
 PUT
 
 #### HTTP Request Example
 
 ```
-Content-Type: application/json
 PUT /filters/dependencies/4f0279da74ef4584a29dc72c835fe2c9/deploy/3
 
 ```
@@ -636,17 +656,16 @@ An application can undeploy the Dependency of an account from Swift by issuing a
 ### Request
 
 #### URL structure
-The URL that represents the dependency data resource. The URL is
-**/filters/dependencies/:account/undeploy/:dependency_id/**
+The URL that represents the filter dependency data resource. The URL is
+**/filters/dependencies/{project}/undeploy/{dependency_id}/**
 
 #### Method
 PUT
 
 #### HTTP Request Example
 
 ```json
-Content-Type: application/json
-POST /filters/dependencies/4f0279da74ef4584a29dc72c835fe2c9/undeploy/3
+PUT /filters/dependencies/4f0279da74ef4584a29dc72c835fe2c9/undeploy/3
 
 ```
 
@@ -657,17 +676,16 @@ An application can list all the deployed Dependencies of an account to Swift by
 ### Request
 
 #### URL structure
-The URL that represents the filter data resource. The URL is
-**/filters/dependencies/:account/deploy/**
+The URL that represents the filter dependencies data resource. The URL is
+**/filters/dependencies/{project}/deploy/**
 
 #### Method
 GET
 
 #### HTTP Request Example
 
 ```json
-Content-Length: 294
-Content-Type: application/json
+
 GET /filters/dependencies/123/deploy/
 
 ```