Merge branch 'dev'

Crystal-SDS · Oct 24, 2016 · 8295113 · 8295113
2 parents 68fb886 + efb6bc1
commit 8295113
Show file tree

Hide file tree

Showing 14 changed files with 1,064 additions and 194 deletions.
diff --git a/README.md b/README.md
@@ -1,11 +1,28 @@
+# Crystal
+
+[Crystal](http://crystal-sds.org/) is a transparent, dynamic and open Software-Defined Storage (SDS) system for [OpenStack Swift](http://swift.openstack.org).
+It is structured in several sub-projects:
+
+* [Controller](https://github.com/Crystal-SDS/controller): the Crystal control plane that offers dynamic meta-programming facilities over the data plane.
+
+* [Introspection middleware](https://github.com/Crystal-SDS/introspection-middleware): the inspection triggers (data plane), that enable controllers to dynamically respond to workload changes in real time.
+
+* [Filter middleware](https://github.com/Crystal-SDS/filter-middleware): the storage filters (data plane), that intercept object flows and run computations or perform transformations on them.
+
+* [Dashboard](https://github.com/iostackproject/SDS-dashboard/tree/urv_dev): A user-friendly dashboard to manage policies, filters and workload metrics.
+
+![alt text](http://crystal-sds.org/wp-content/uploads/2016/05/architecture9-768x575.png "Crystal Architecture")
+
+# Crystal Controller
+
 [![Build Status](https://travis-ci.org/Crystal-SDS/controller.svg?branch=master)](https://travis-ci.org/Crystal-SDS/controller)
 [![Coverage Status](https://coveralls.io/repos/github/Crystal-SDS/controller/badge.svg?branch=master)](https://coveralls.io/github/Crystal-SDS/controller?branch=master)
 [![Code Health](https://landscape.io/github/Crystal-SDS/controller/master/landscape.svg?style=flat)](https://landscape.io/github/Crystal-SDS/controller/master)
 
-
-# Crystal-Controller
-
-This repository contains the code of the SDS Controller for Object Storage in the IOStack architecture. The SDS Controller repository contains two differentiated parts: **The SDS Controller API** and the **Dynamic Policies**. The **SDS Controller API** is a Django project that implements the REST API needed to handle the Storlets and the BW Differentiation. Otherwise, **Dynamic Policies** is a set of python processes who use the [PyActive middleware](https://github.com/cloudspaces/pyactive) (an Object Oriented implementation of the Actor model). This part allows to create simple policies using a DSL (integrated in the SDS Controller API) and deploys them as an actor process, who analyze the system data thanks to the monitoring system, and allows to set or remove filters to tenants depending on the established policy.
+This repository contains the code of Crystal Controller, the Software-Defined-Storage (SDS) REST API in the [IOStack](https://github.com/iostackproject) architecture.
+It is a Django project that implements the REST API needed to handle filters, storlets and policies on top of Openstack Swift object-storage system. This API also includes a set of python processes who use
+the [PyActive middleware](https://github.com/cloudspaces/pyactive), an Object Oriented implementation of the Actor model. This part allows to create simple policies using a DSL (integrated in the Crystal Controller API)
+and to deploy them as an actor process, who analyze the system data thanks to the monitoring system, and allows to set or remove filters to tenants depending on the established policy.
 
 The repository is structured with the next folders:
 
@@ -15,33 +32,49 @@ The repository is structured with the next folders:
 
 * **scripts:** The scripts folder contains all the scripts needed for the project. The file vagrant-init.sh will be executed each time that you start the virtual machine using vagrant.
 
-* **sds_controller:** The sds_controller contains the source code. Its structure follows a standard Django project structure.
+* **api:** The folder contains the API source code. Its structure follows a standard Django project structure.
 
 * **dynamic_policies** The dynamic_policies contains the source code of this part.
 
-* **Vagrantfile:** This is the vagrant config file, where we define all the information that vagrant need to start a virtual machine with all the requirements.  
+* **Vagrantfile:** This is the vagrant config file, where we define all the information that vagrant needs to start a virtual machine with all the requirements.
 
 To build the APIs in an easy way we use [Django REST Framework](http://www.django-rest-framework.org/).
 
-# Requirements
+## Requirements
 
-This project only has two requirements:
+This project includes a Vagrant environment with all the dependencies (Django, pyactive, swiftclient, ...), so the only two requirements are:
 
 1. Install Virtual Box [Visit VirtualBox page!](https://www.virtualbox.org/)
 2. Install Vagrant [Visit Vagrant page!](https://www.vagrantup.com/downloads.html)
 
-That's all! You don't need Django or Python... Vagrant resolves this problem for us.
+## Installation
 
-# Installation
+Once you have already installed the requirements, you only need to go to the project location using a terminal, and execute the command: `vagrant up`. First time, the process may take a few minutes because Vagrant downloads the Operative System to create the Virtual Machine. Next time the process will be faster.
 
-Once you have already installed the requirements, you only need to go in the folder location using a terminal, and execute the command: `vagrant up`. First time, the process may take a few minutes because Vagrant downloads the Operative System to create the Virtual Machine. Next time the process will be faster.
+The Virtual Machine that we started has all the tools that we need to run the server. To connect to this machine you only need to run the command `vagrant ssh`. The repository folder is synchronized between the local machine and the Virtual Machine, so you can develop the code in your local machine with your preferred IDE, and run the project in the Virtual Machine.
 
-The Virtual Machine that we started has all the tools that we need to run the server. To connect to this machine you only need to run the command `vagrant ssh`. The repository folder is synchronized between your machine and the Virtual Machine, so you can develop the code in your local machine with your prefer IDE, and run the project in the Virtual Machine.
+You can start the server using the command into the source folder (./api): `python manage.py runserver 0.0.0.0:8000`. After that, the server starts, and if you prefer to call Crystal controller from the host machine the port to use is `18000`. For instance, to list the Storlets from the host machine the URL should be: localhost:18000/storlets.
 
-You can start the server using the command into the source folder (src/sds_controller): `python manage.py runserver 0.0.0.0:8000`. After that the server starts, and if you prefer to call the SDS controller for your machine the port to use is `18000`. For instance, to call to list the Storlets from your machine the url will be: localhost:18000/storlets. *We have in the TODO list to configure vagrant and puppet to do a deploy of the SDS Controller in Apache each time that starts the Virtual Machine.*
+If some problem appears, make sure that:
 
-If some problem appear, make sure..
-
-1. redis-server service is running? Start this service, because the SDS Controller API stores the meta-data information in redis.
-2. is PyActive in the PYTHONPATH? At home folder you can found the pyactive folder, where you can find another install.txt, please follow this steps.
+1. redis-server service is running? SDS Controller API stores the meta-data information in redis.
+2. is PyActive in the PYTHONPATH? At home folder you can find the pyactive folder, where you can find another install.txt, please follow this steps.
 3. review the settings file from SDS Controller and make sure to write the correct IPs (Swift IP, Keystone IP, PyActive IP)
+
+## Usage
+
+API usage is detailed in the [API specification](/doc/api_specification.md).
+
+A convenient [web dashboard](https://github.com/iostackproject/SDS-dashboard) is also available to simplify these API calls. Refer to the [dashboard overview](/doc/dashboard_overview.md) for detailed information.
+
+### Tests
+
+Run unit tests from the source folder (`./api`) with the following command: `python manage.py test`
+
+## Support
+
+Please [open an issue](https://github.com/Crystal-SDS/controller/issues/new) for support.
+
+## Contributing
+
+Please contribute using [Github Flow](https://guides.github.com/introduction/flow/). Create a branch, add commits, and [open a pull request](https://github.com/Crystal-SDS/controller/compare/).
diff --git a/api/bw/tests.py b/api/bw/tests.py
@@ -111,7 +111,7 @@ def test_bw_detail_ok(self, mock_get_project_list, mock_is_valid_request):
         mock_get_project_list.return_value = {'0123456789abcdef': 'tenantA', 'abcdef0123456789': 'tenantB'}
 
         project_policy_key = '0123456789abcdef:2'
-        request = self.factory.get('/bw/slas/' + project_policy_key)
+        request = self.factory.get('/bw/sla/' + project_policy_key)
         response = bw_detail(request, project_policy_key)
         self.assertEqual(response.status_code, status.HTTP_200_OK)
         json_data = json.loads(response.content)
@@ -127,13 +127,13 @@ def test_update_sla_ok(self, mock_get_project_list, mock_is_valid_request):
 
         project_policy_key = '0123456789abcdef:2'
         sla_data = {'bandwidth': '10000'}
-        request = self.factory.put('/bw/slas/' + project_policy_key, sla_data, format='json')
+        request = self.factory.put('/bw/sla/' + project_policy_key, sla_data, format='json')
         response = bw_detail(request, project_policy_key)
         self.assertEqual(response.status_code, status.HTTP_201_CREATED)
 
         # Verify the SLA was updated
         mock_get_project_list.return_value = {'0123456789abcdef': 'tenantA', 'abcdef0123456789': 'tenantB'}
-        request = self.factory.get('/bw/slas/' + project_policy_key)
+        request = self.factory.get('/bw/sla/' + project_policy_key)
         response = bw_detail(request, project_policy_key)
         self.assertEqual(response.status_code, status.HTTP_200_OK)
         json_data = json.loads(response.content)
@@ -145,7 +145,7 @@ def test_delete_sla_ok(self, mock_get_project_list, mock_is_valid_request):
 
         mock_is_valid_request.return_value = 'fake_token'
         project_policy_key = '0123456789abcdef:2'
-        request = self.factory.delete('/bw/slas/' + project_policy_key)
+        request = self.factory.delete('/bw/sla/' + project_policy_key)
         response = bw_detail(request, project_policy_key)
         self.assertEqual(response.status_code, status.HTTP_204_NO_CONTENT)
 

diff --git a/api/filters/tests.py b/api/filters/tests.py
@@ -42,6 +42,7 @@ def test_list_storlet_ok(self, mock_is_valid_request):
         self.assertNotEqual(response.content, "[]")
         storlets = json.loads(response.content)
         self.assertEqual(storlets[0]['main'], "com.example.FakeMain")
+        self.assertEqual(storlets[0]['id'], "1")
 
     def test_delete_storlet_ok(self, mock_is_valid_request):
         """
@@ -249,6 +250,7 @@ def test_filter_deploy_to_project_ok(self, mock_put_object, mock_is_valid_reques
         request.META['HTTP_X_AUTH_TOKEN'] = 'fake_token'
         response = filter_deploy(request, "1", "0123456789abcdef")
         self.assertEqual(response.status_code, status.HTTP_201_CREATED)
+        self.assertEqual(response.content, '1')
         mock_put_object.assert_called_with(settings.SWIFT_URL + settings.SWIFT_API_VERSION + "/AUTH_0123456789abcdef",
                                            'fake_token', "storlet", "test-1.0.jar", mock.ANY, mock.ANY, mock.ANY,
                                            mock.ANY, mock.ANY, mock.ANY, mock.ANY, mock.ANY, mock.ANY, mock.ANY)
@@ -275,6 +277,7 @@ def test_filter_deploy_to_project_and_container_ok(self, mock_put_object, mock_i
         request.META['HTTP_X_AUTH_TOKEN'] = 'fake_token'
         response = filter_deploy(request, "1", "0123456789abcdef", "container1")
         self.assertEqual(response.status_code, status.HTTP_201_CREATED)
+        self.assertEqual(response.content, '1')
         mock_put_object.assert_called_with(settings.SWIFT_URL + settings.SWIFT_API_VERSION + "/AUTH_0123456789abcdef",
                                            'fake_token', "storlet", "test-1.0.jar", mock.ANY, mock.ANY, mock.ANY,
                                            mock.ANY, mock.ANY, mock.ANY, mock.ANY, mock.ANY, mock.ANY, mock.ANY)

diff --git a/api/filters/urls.py b/api/filters/urls.py
@@ -14,9 +14,9 @@
     url(r'^/(?P<account>\w+)/(?P<container>[-\w]+)/(?P<swift_object>[-\w]+)/deploy/(?P<filter_id>[0-9]+)/?$', views.filter_deploy),
 
     # Undeploy to tenant container or object
-    # url(r'^/(?P<account>\w+)/undeploy/(?P<filter_id>[0-9]+)/?$', views.filter_undeploy),
-    # url(r'^/(?P<account>\w+)/(?P<container>[-\w]+)/undeploy/(?P<filter_id>[0-9]+)/?$', views.filter_undeploy),
-    # url(r'^/(?P<account>\w+)/(?P<container>[-\w]+)/(?P<swift_object>[-\w]+)/undeploy/(?P<filter_id>[0-9]+)/?$', views.filter_undeploy),
+    url(r'^/(?P<account>\w+)/undeploy/(?P<filter_id>[0-9]+)/?$', views.filter_undeploy),
+    url(r'^/(?P<account>\w+)/(?P<container>[-\w]+)/undeploy/(?P<filter_id>[0-9]+)/?$', views.filter_undeploy),
+    url(r'^/(?P<account>\w+)/(?P<container>[-\w]+)/(?P<swift_object>[-\w]+)/undeploy/(?P<filter_id>[0-9]+)/?$', views.filter_undeploy),
 
     url(r'^/dependencies/?$', views.dependency_list),
     url(r'^/dependencies/(?P<dependency_id>\w+)/?$', views.dependency_detail),

diff --git a/api/filters/views.py b/api/filters/views.py
@@ -318,7 +318,7 @@ def storlet_list_deployed(request, account):
 @csrf_exempt
 def filter_undeploy(request, filter_id, account, container=None, swift_object=None):
     """
-    Undeploy a storlet from a specific swift account.
+    Undeploy a filter from a specific swift account.
     """
     # Validate request: only admin user can access to this method
     token = is_valid_request(request)

diff --git a/api/registry/tests.py b/api/registry/tests.py
@@ -150,7 +150,6 @@ def test_registry_dynamic_policy_create_spawn_id_ok(self, mock_create_local_host
     #     response = policy_list(request)
     #     self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)
 
-
     #
     # Metric tests
     #
@@ -486,13 +485,13 @@ def test_get_storage_node_ok(self, mock_is_valid_request):
         self.assertEqual(response.status_code, status.HTTP_200_OK)
         metric_data = json.loads(response.content)
         self.assertEqual(metric_data['name'], 'storagenode1')
-        self.assertEqual(metric_data['location'], 'location1')
-        self.assertEqual(metric_data['type'], 'type1')
+        self.assertEqual(metric_data['location'], 'r1z1-192.168.1.5:6000/sdb1')
+        self.assertEqual(metric_data['type'], 'hdd')
 
     def test_update_storage_node_ok(self, mock_is_valid_request):
         mock_is_valid_request.return_value = 'fake_token'
         snode_id = 1
-        data = {'name': 'storagenode1updated', 'location': 'location1updated', 'type': 'type1updated'}
+        data = {'name': 'storagenode1updated', 'location': 'r1z1-192.168.1.6:6000/sdb1', 'type': 'hddupdated'}
         request = self.factory.put('/registry/snode/' + str(snode_id), data, format='json')
         response = storage_node_detail(request, str(snode_id))
         self.assertEqual(response.status_code, status.HTTP_201_CREATED)
@@ -503,8 +502,8 @@ def test_update_storage_node_ok(self, mock_is_valid_request):
         self.assertEqual(response.status_code, status.HTTP_200_OK)
         metric_data = json.loads(response.content)
         self.assertEqual(metric_data['name'], 'storagenode1updated')
-        self.assertEqual(metric_data['location'], 'location1updated')
-        self.assertEqual(metric_data['type'], 'type1updated')
+        self.assertEqual(metric_data['location'], 'r1z1-192.168.1.6:6000/sdb1')
+        self.assertEqual(metric_data['type'], 'hddupdated')
 
     def test_delete_storage_node_ok(self, mock_is_valid_request):
         mock_is_valid_request.return_value = 'fake_token'
@@ -1083,6 +1082,27 @@ def test_registry_static_policy_detail_ok(self, mock_get_project_list, mock_is_v
         json_data = json.loads(response.content)
         self.assertEqual(json_data["target_name"], 'tenantA')
 
+    @mock.patch('registry.views.get_project_list')
+    def test_registry_static_policy_update(self, mock_get_project_list, mock_is_valid_request):
+        mock_is_valid_request.return_value = 'fake_token'
+        mock_get_project_list.return_value = {'0123456789abcdef': 'tenantA', '2': 'tenantB'}
+
+        # Create an instance of a PUT request.
+        data = {"execution_server": "object", "execution_server_reverse": "object"}
+        request = self.factory.put('/registry/static_policy/0123456789abcdef:1', data, format='json')
+        request.META['HTTP_X_AUTH_TOKEN'] = 'fake_token'
+        response = static_policy_detail(request, '0123456789abcdef:1')
+        self.assertEqual(response.status_code, status.HTTP_201_CREATED)
+
+        # Create an instance of a GET request.
+        request = self.factory.get('/registry/static_policy/0123456789abcdef:1')
+        request.META['HTTP_X_AUTH_TOKEN'] = 'fake_token'
+        response = static_policy_detail(request, '0123456789abcdef:1')
+        self.assertEqual(response.status_code, status.HTTP_200_OK)
+        json_data = json.loads(response.content)
+        self.assertEqual(json_data["execution_server"], 'object')
+        self.assertEqual(json_data["execution_server_reverse"], 'object')
+
     @mock.patch('registry.views.get_project_list')
     def test_registry_static_policy_detail_delete(self, mock_get_project_list, mock_is_valid_request):
         mock_is_valid_request.return_value = 'fake_token'
@@ -1174,8 +1194,10 @@ def create_object_type_docs(self, mock_is_valid_request):
         self.assertEqual(response.status_code, status.HTTP_201_CREATED)
 
     def setup_dsl_parser_data(self):
-        self.r.hmset('dsl_filter:compression', {'identifier': '1', 'valid_parameters': '{"cparam1": "integer", "cparam2": "integer", "cparam3": "integer"}'})
-        self.r.hmset('dsl_filter:encryption', {'identifier': '2', 'valid_parameters': '{"eparam1": "integer", "eparam2": "bool", "eparam3": "string"}'})
+        self.r.hmset('dsl_filter:compression', {'identifier': '1', 'valid_parameters': '{"cparam1": "integer", "cparam2": "integer", "cparam3": "integer"}',
+                                                'activation_url': 'http://10.30.1.6:9000/filters'})
+        self.r.hmset('dsl_filter:encryption', {'identifier': '2', 'valid_parameters': '{"eparam1": "integer", "eparam2": "bool", "eparam3": "string"}',
+                                               'activation_url': 'http://10.30.1.6:9000/filters'})
         self.r.hmset('metric:metric1', {'network_location': '?', 'type': 'integer'})
         self.r.hmset('metric:metric2', {'network_location': '?', 'type': 'integer'})
         self.r.rpush('G:1', '1234567890abcdef')
@@ -1202,7 +1224,7 @@ def create_nodes(self):
 
     def create_storage_nodes(self):
         self.r.incr("storage_nodes:id")  # setting autoincrement to 1
-        self.r.hmset('SN:1', {'name': 'storagenode1', 'location': 'location1', 'type': 'type1'})
+        self.r.hmset('SN:1', {'name': 'storagenode1', 'location': 'r1z1-192.168.1.5:6000/sdb1', 'type': 'hdd'})
 
     def create_metric_modules(self):
         self.r.incr("workload_metrics:id")  # setting autoincrement to 1

diff --git a/api/registry/views.py b/api/registry/views.py
@@ -199,7 +199,7 @@ def add_dynamic_filter(request):
         if not name:
             return JSONResponse('Filter must have a name', status=400)
         r.hmset('dsl_filter:' + str(name), data)
-        return JSONResponse('Filter has been added in the registy', status=201)
+        return JSONResponse('Filter has been added to the registy', status=201)
     return JSONResponse('Method ' + str(request.method) + ' not allowed.', status=405)
 
 

diff --git a/api/swift/views.py b/api/swift/views.py
@@ -88,7 +88,7 @@ def storage_policies(request):
                 return JSONResponse('Error creating the Storage Policy: ' + e, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
 
         return JSONResponse('Account created successfully', status=status.HTTP_201_CREATED)
-    return JSONResponse('Only HTTP POST /spolicies/ requests allowed.', status=status.HTTP_405_METHOD_NOT_ALLOWED)
+    return JSONResponse('Only HTTP POST requests allowed.', status=status.HTTP_405_METHOD_NOT_ALLOWED)
 
 
 @csrf_exempt