Merge branch 'dev'

.travis.yml

@@ -9,7 +9,7 @@ install:
 - git clone https://github.com/cloudspaces/pyactive.git && cd pyactive/pyactive_project
   && python setup.py develop && cd ../..
 script:
-- cd sds_controller/ && coverage run --source='.' manage.py test
+- cd api/ && coverage run --source='.' manage.py test
 after_success: coveralls
 notifications:
   slack:

README.md

@@ -5,7 +5,7 @@
 
 # Crystal-Controller
 
-This repository contains the code of the SDS Controller for Object Storage in the IOStack architecture. The SDS Controller repository contains two differentiated parts: **The SDS Controller API** and the **Dynamic Policies**. The **SDS Controller API** is a Django project that implements the REST API needed to handle the Storlets and the BW Differentiation. Otherwise, **Dynamic Policies** is a set of python processes who use the [PyActive middleware](https://github.com/cloudspaces/pyactive) (an Object Oriented implementation of the Actor model). This part allows to create simple policies using a DSL (integrated in the SDS Controller API) and deploys them as an actor process, whose analyze the system data, thanks to the monitoring system, allows to set or remove filters to tenants depending of the policy established.
+This repository contains the code of the SDS Controller for Object Storage in the IOStack architecture. The SDS Controller repository contains two differentiated parts: **The SDS Controller API** and the **Dynamic Policies**. The **SDS Controller API** is a Django project that implements the REST API needed to handle the Storlets and the BW Differentiation. Otherwise, **Dynamic Policies** is a set of python processes who use the [PyActive middleware](https://github.com/cloudspaces/pyactive) (an Object Oriented implementation of the Actor model). This part allows to create simple policies using a DSL (integrated in the SDS Controller API) and deploys them as an actor process, who analyze the system data thanks to the monitoring system, and allows to set or remove filters to tenants depending on the established policy.
 
 The repository is structured with the next folders:
 
@@ -15,7 +15,7 @@ The repository is structured with the next folders:
 
 * **scripts:** The scripts folder contains all the scripts needed for the project. The file vagrant-init.sh will be executed each time that you start the virtual machine using vagrant.
 
-* **sds_controller:** The sds_controller contains the source code. It's structure follows a standard Django project structure.
+* **sds_controller:** The sds_controller contains the source code. Its structure follows a standard Django project structure.
 
 * **dynamic_policies** The dynamic_policies contains the source code of this part.
 
@@ -25,7 +25,7 @@ To build the APIs in an easy way we use [Django REST Framework](http://www.djang
 
 # Requirements
 
-These project only have two requirements:
+This project only has two requirements:
 
 1. Install Virtual Box [Visit VirtualBox page!](https://www.virtualbox.org/)
 2. Install Vagrant [Visit Vagrant page!](https://www.vagrantup.com/downloads.html)
@@ -34,7 +34,7 @@ That's all! You don't need Django or Python... Vagrant resolves this problem for
 
 # Installation
 
-Once you have already installed the requirements, you only need to go in the folder location using a terminal, and execute the command: `vagrant up`. First time, the process may take a few minutes because Vagrant downloads the Operative System to create the Virtual Machine. The other times the process will be faster.
+Once you have already installed the requirements, you only need to go in the folder location using a terminal, and execute the command: `vagrant up`. First time, the process may take a few minutes because Vagrant downloads the Operative System to create the Virtual Machine. Next time the process will be faster.
 
 The Virtual Machine that we started has all the tools that we need to run the server. To connect to this machine you only need to run the command `vagrant ssh`. The repository folder is synchronized between your machine and the Virtual Machine, so you can develop the code in your local machine with your prefer IDE, and run the project in the Virtual Machine.
 
@@ -45,16 +45,3 @@ If some problem appear, make sure..
 1. redis-server service is running? Start this service, because the SDS Controller API stores the meta-data information in redis.
 2. is PyActive in the PYTHONPATH? At home folder you can found the pyactive folder, where you can find another install.txt, please follow this steps.
 3. review the settings file from SDS Controller and make sure to write the correct IPs (Swift IP, Keystone IP, PyActive IP)
-
-
-# Monitoring
-<!-- out of date -->
-To enable the monitoring module you need to follow this steps. First create a new queue at OpenStack controller host. You need to be logged into the OpenStack controller host and run this command: `sudo rabbitmqadmin declare queue name="myQueue" durable=true auto_delete=false` and assign a binding to it with the service you want to monitor with `sudo rabbitmqadmin declare binding source="ceilometer" destination_type="queue" destination="myQueue" routing_key="metering.sample"` where myQueue will be the name of the queue to retrieve the events.
-
-`TODO:` Then, you need to edit the config file `x` and add the ip:port tuple of the RabbitMQ at OpenStack controller host (by default `rabbitmq_host_ip:5672`) and the name of the queue that you created before, myQueue in this lines.
-
-# Future Work
-
-- [x] Communicate Storlet module with OpenStack Swift.
-- [x] Communicate BW module with OpenStack Swift.
-- [x] Add Monitoring module and communicate with OpenStack Swift using RabbitMQ.

sds_controller/sds_controller/common_utils.py → api/api/common_utils.py

@@ -7,6 +7,7 @@
 from datetime import datetime
 import os
 
+valid_tokens = dict()
 
 class JSONResponse(HttpResponse):
     """
@@ -43,22 +44,32 @@ def get_keystone_admin_auth():
 def is_valid_request(request):    
     token = request.META['HTTP_X_AUTH_TOKEN']
     is_admin = False
-    keystone = get_keystone_admin_auth()
+    now = datetime.now()
 
-    try:
-        token_data = keystone.tokens.validate(token)
-        token_expiration = datetime.strptime(token_data.expires, '%Y-%m-%dT%H:%M:%SZ')
-        now = datetime.now()
+    if token not in valid_tokens:
+        keystone = get_keystone_admin_auth()
+
+        try:
+            token_data = keystone.tokens.validate(token)
+        except:
+            return False
+
+        token_expiration = datetime.strptime(token_data.expires, 
+                                             '%Y-%m-%dT%H:%M:%SZ')
 
         token_roles = token_data.user['roles']
         for role in token_roles:
             if role['name'] == 'admin':
                 is_admin = True
 
         if token_expiration > now and is_admin:
+            valid_tokens[token] = token_expiration
+            return token
+
+    else:
+        token_expiration = valid_tokens[token]
+        if token_expiration > now:
             return token
-    except:
-        return False
 
     return False
 

sds_controller/sds_controller/settings.py → api/api/settings.py

@@ -1,5 +1,5 @@
 """
-Django settings for sds_controller project.
+Django settings for controller project.
 
 Generated by 'django-admin startproject' using Django 1.8.1.
 
@@ -44,7 +44,7 @@
     'django.contrib.staticfiles',
     'bootstrap3',
     'rest_framework',
-    'storlet',
+    'filters',
     'bw',
     'swift',
     'registry'
@@ -61,7 +61,7 @@
     'django.middleware.security.SecurityMiddleware',
 )
 
-ROOT_URLCONF = 'sds_controller.urls'
+ROOT_URLCONF = 'api.urls'
 
 TEMPLATES = [
     {
@@ -79,7 +79,7 @@
     },
 ]
 
-WSGI_APPLICATION = 'sds_controller.wsgi.application'
+WSGI_APPLICATION = 'api.wsgi.application'
 
 
 # Database
@@ -93,45 +93,47 @@
     }
 }
 
-# DATABASES = {
-#     'default': {
-#         'ENGINE': 'django.db.backends.postgresql_psycopg2',
-#         'NAME': 'sds_controller',
-#         'USER': 'sds_controller_user',
-#         'PASSWORD': 'sds_controller_pass',
-#         'HOST': 'localhost',
-#         'PORT': '5432',
-#     }
-# }
-
 LANGUAGE_CODE = 'en-us'
-TIME_ZONE = 'UTC'
+TIME_ZONE = 'CET'
 USE_I18N = True
 USE_L10N = True
 USE_TZ = True
 
 STATIC_URL = '/static/'
 
-# Swift info
+# Keystone
 KEYSTONE_ADMIN_URL = "http://localhost:5000/v2.0"
 KEYSTONE_URL = "http://localhost:35357/v2.0"
-SWIFT_URL= "http://localhost:8080/"
+
+# Swift
+SWIFT_URL = "http://localhost:8080/"
 SWIFT_API_VERSION = "v1"
 
-# redis
+# Redis
 REDIS_CON_POOL = redis.ConnectionPool(host='localhost', port=6379, db=0)
 
 # SDS Project
 STORLET_BIN_DIR = "/opt/ibm"
 STORLET_DOCKER_IMAGE = "192.168.2.1:5001/ubuntu_14.04_jre8_storlets"
 STORLET_TAR_FILE = "ubuntu_14.04_jre8_storlets.tar"
 
+# Openstack Admin
 MANAGEMENT_ACCOUNT = "management"
 MANAGEMENT_ADMIN_USERNAME = "manager"
-MANAGEMENT_ADMIN_PASSWORD = "changeme"  # noqa
+MANAGEMENT_ADMIN_PASSWORD = "manager"  # noqa
 
 # pyactive
-PYACTIVE_URL = "tcp://127.0.0.1:6375/"
-PYACTIVE_IP = "127.0.0.1"
-PYACTIVE_PORT = 6375
 PYACTIVE_TRANSPORT = "tcp"
+PYACTIVE_IP = "127.0.0.1"
+PYACTIVE_PORT = 6899
+PYACTIVE_URL = PYACTIVE_TRANSPORT+'://'+PYACTIVE_IP+':'+str(PYACTIVE_PORT)
+
+# Mertrics
+METRIC_CLASS = 'registry.dynamic_policies.metrics.swift_metric'
+METRIC_MAIN = 'SwiftMetric'
+
+# Rules
+RULE_CLASS = 'registry.dynamic_policies.rules.rule'
+RULE_MAIN = 'Rule'
+RULE_TRANSIENT_CLASS = 'registry.dynamic_policies.rules.rule_transient'
+RULE_TRANSIENT_MAIN = 'TransientRule'

api/api/startup.py

@@ -0,0 +1,20 @@
+from api.common_utils import get_redis_connection
+
+def run():
+    """
+    When the controller is started (or restarted) all the actors  
+    are stopped, so we need to ensure the correct values in redis.
+    """
+    r = get_redis_connection()
+
+    # Workload metric definitions
+    for key in r.keys('workload_metric:*'):
+        r.hset(key,'enabled', False)
+
+    # Workload metric Actors
+    for key in r.keys('metric:*'):
+        r.delete(key)
+
+    # Dynamic policies
+    for key in r.keys('policy:*'):
+        r.hset(key,'alive', 'False')

sds_controller/sds_controller/tests.py → api/api/tests.py

@@ -45,7 +45,7 @@ def test_to_json_bools_ok(self):
         self.assertEqual(bdict['d'], 'False')
         self.assertNotEqual(bdict['d'], False)
 
-    @mock.patch('sds_controller.common_utils.os.system')
+    @mock.patch('api.common_utils.os.system')
     def test_rsync_dir_with_nodes_ok(self, mock_os_system):
         mock_os_system.return_value = 0  # return value when rsync succeeds
 
@@ -62,7 +62,7 @@ def test_rsync_dir_with_nodes_when_username_and_password_not_present(self):
         with self.assertRaises(FileSynchronizationException):
             rsync_dir_with_nodes(settings.WORKLOAD_METRICS_DIR)
 
-    @mock.patch('sds_controller.common_utils.os.system')
+    @mock.patch('api.common_utils.os.system')
     def test_rsync_dir_with_nodes_when_rsync_fails(self, mock_os_system):
         mock_os_system.return_value = 1  # return value when rsync fails
 
@@ -76,14 +76,14 @@ def test_rsync_dir_with_nodes_when_rsync_fails(self, mock_os_system):
 
     def test_urls(self):
         resolver = resolve('/filters/')
-        self.assertEqual(resolver.view_name, 'storlet.views.storlet_list')
+        self.assertEqual(resolver.view_name, 'filters.views.storlet_list')
 
         resolver = resolve('/filters/123')
-        self.assertEqual(resolver.view_name, 'storlet.views.storlet_detail')
+        self.assertEqual(resolver.view_name, 'filters.views.storlet_detail')
         self.assertEqual(resolver.kwargs, {'storlet_id': '123'})
 
         resolver = resolve('/filters/123/data')
-        self.assertEqual(resolver.view_name, 'storlet.views.StorletData')
+        self.assertEqual(resolver.view_name, 'filters.views.StorletData')
         self.assertEqual(resolver.kwargs, {'storlet_id': '123'})
 
         resolver = resolve('/registry/nodes/')

sds_controller/sds_controller/urls.py → api/api/urls.py

@@ -1,4 +1,4 @@
-"""sds_controller URL Configuration
+"""controller URL Configuration
 
 The `urlpatterns` list routes URLs to views. For more information please see:
     https://docs.djangoproject.com/en/1.8/topics/http/urls/
@@ -18,7 +18,7 @@
 
 urlpatterns = [
     url(r'^admin/', include(admin.site.urls)),
-    url(r'^filters', include('storlet.urls')),
+    url(r'^filters', include('filters.urls')),
     url(r'^bw', include('bw.urls')),
     url(r'^swift', include('swift.urls')),
     url(r'^registry', include('registry.urls'))

sds_controller/sds_controller/wsgi.py → api/api/wsgi.py

@@ -1,16 +1,18 @@
 """
-WSGI config for sds_controller project.
+WSGI config for controller project.
 
 It exposes the WSGI callable as a module-level variable named ``application``.
 
 For more information on this file, see
 https://docs.djangoproject.com/en/1.8/howto/deployment/wsgi/
 """
-
-import os
-
 from django.core.wsgi import get_wsgi_application
+import os
 
-os.environ.setdefault("DJANGO_SETTINGS_MODULE", "sds_controller.settings")
+import api.startup as startup
+startup.run()
 
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "api.settings")        
 application = get_wsgi_application()
+
+

sds_controller/bw/views.py → api/bw/views.py

@@ -2,7 +2,8 @@
 from redis.exceptions import RedisError, DataError
 from rest_framework import status
 from rest_framework.parsers import JSONParser
-from sds_controller.common_utils import  JSONResponse, get_redis_connection, get_project_list, is_valid_request
+
+from api.common_utils import  JSONResponse, get_redis_connection, get_project_list, is_valid_request
 
 
 @csrf_exempt
@@ -12,7 +13,7 @@ def bw_list(request):
     """ 
     token = is_valid_request(request)
     if not token:
-        return JSONResponse('You must be authenticated as Crystal admin.', status=status.HTTP_401_UNAUTHORIZED) 
+        return JSONResponse('You must be authenticated as admin.', status=status.HTTP_401_UNAUTHORIZED) 
 
     try:
         r = get_redis_connection()
@@ -56,7 +57,7 @@ def bw_detail(request, project_key):
     """
     token = is_valid_request(request)
     if not token:
-        return JSONResponse('You must be authenticated as Crystal admin.', status=status.HTTP_401_UNAUTHORIZED)
+        return JSONResponse('You must be authenticated as admin.', status=status.HTTP_401_UNAUTHORIZED)
 
     try:
         r = get_redis_connection()

sds_controller/docs/conf.py → api/docs/conf.py

@@ -51,9 +51,9 @@
 master_doc = 'index'
 
 # General information about the project.
-project = u'SDS Controller for Object Storage'
-copyright = u'2015, Sir. Edgar Zamora-Gómez, Dr. Raúl Gracia'
-author = u'Sir. Edgar Zamora-Gómez, Dr. Raúl Gracia'
+project = u'Crystal Controller'
+copyright = u'2015-2016, Edgar Zamora-Gómez, Raúl Gracia'
+author = u'Edgar Zamora-Gómez, Raúl Gracia'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
@@ -228,7 +228,7 @@
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
   (master_doc, 'SDSControllerforObjectStorage.tex', u'SDS Controller for Object Storage Documentation',
-   u'Sir. Edgar Zamora-Gómez, Dr. Raúl Gracia', 'manual'),
+   u'Edgar Zamora-Gómez, Raúl Gracia', 'manual'),
 ]
 
 # The name of an image file (relative to this directory) to place at the top of