Refactor Storage to allow different backends

I am unable to run Redis in my environment, but am able to run Couchbase
and also use local disk storage.

I refactored the StorageMixin so that it's abstract and a new setting
named STORAGE_BACKEND defines which backend actually handles the
persisting of the values.

This doesn't introduce anything that's not backwards compatible and
allows existing deployments of Will to run unchanged.

As part of this I refactored the requirements so that there could be a
requirements file for couchbase as well as redis.

.gitignore

@@ -38,3 +38,6 @@ shelf.db
 .idea/*
 site
 docs/.DS_Store
+
+# VIM swap files
+.*.sw[a-z]

AUTHORS

@@ -24,3 +24,4 @@ Ryan Murfitt, https://github.com/puug
 Piotr 'keNzi' Czajkowski, http://www.videotesty.pl/
 Dmitri Muntean, https://github.com/dmuntean
 Ben lau, Mashery, https://github.com/netjunki
+Evan Borgstrom, https://github.com/borgstrom

README.md

@@ -11,7 +11,7 @@ He makes teaching your chat bot this simple:
 ```python
 @respond_to("hi")
 def say_hello(self, message):
-    self.say("oh, hello!")
+    self.say("oh, hello!", message=message)
 ```
 
 Will was first built by [Steven Skoczen](http://stevenskoczen.com) while in the [Greenkahuna Skunkworks](http://skunkworks.greenkahuna.com), and has been [contributed to by lots of folks](http://skoczen.github.io/will/improve/#the-shoulders-of-giants).

docs/index.md

@@ -10,7 +10,7 @@ He makes teaching your chat bot this simple:
 ```
 @respond_to("hi")
 def say_hello(self, message):
-    self.say("oh, hello!")
+    self.say("oh, hello!", message=message)
 ```
 
 Lots of batteries are included, and you can get your own will up and running in a couple of minutes.  
@@ -123,4 +123,38 @@ Once your will is up and running, hop into any of your hipchat rooms, and say he
 
 ![Help, will](img/help.gif)
 
-You're up and running - now it's time to [teach your will a few things](plugins/basics.md)!
+You're up and running - now it's time to [teach your will a few things](plugins/basics.md)!
+
+
+## Storage Backends
+
+Will's default storage backend is Redis, but he supports some others if you can't run Redis.
+
+To change the backend you just need to set `STORAGE_BACKEND` in your config and then supply any other needed settings for the new storage backend.  The currently supported backends are:
+
+ * `redis` - The default Redis backend
+ * `couchbase` - A Couchbase backend
+ * `file` - Keeps the settings as files on a local filesystem
+
+
+#### Couchbase
+
+Couchbase requries you set `COUCHBASE_URL` in your config.
+
+You are also required to have the python Couchbase client (and thus, libcouchbase) installed.  If you are installing for development you can use `pip install -r requirements.couchbase.txt` to pull in the Couchbase client.  See [the Python Couchbase client repo](https://github.com/couchbase/couchbase-python-client) for more info.
+
+Examples:
+
+ * `COUCHBASE_URL='couchbase:///bucket'`
+ * `COUCHBASE_URL='couchbase://hostname/bucket'`
+ * `COUCHBASE_URL='couchbase://host1,host2/bucket'`
+ * `COUCHBASE_URL='couchbase://hostname/bucket?password=123abc&timeout=5'`
+
+#### File
+
+File requires you set `FILE_DIR` in your config to point to an empty directory.
+
+Examples:
+
+ * `FILE_DIR='/var/run/will/settings/'`
+ * `FILE_DIR='~will/settings/'`

requirements.base.txt

@@ -0,0 +1,31 @@
+bottle==0.12.7
+clint==0.3.7
+dill==0.2.1
+dnspython==1.12.0
+natural==0.1.5
+requests==2.4.1
+parsedatetime==1.1.2
+pyasn1==0.1.7
+pyasn1-modules==0.0.5
+sleekxmpp==1.3.1
+APScheduler==2.1.2
+CherryPy==3.6.0
+Fabric==1.10.0
+Jinja2==2.7.3
+Markdown==2.3.1
+MarkupSafe==0.23
+PyYAML==3.10
+argh==0.25.0
+args==0.1.0
+ecdsa==0.11
+futures==2.1.6
+ghp-import==0.4.1
+greenlet==0.4.4
+paramiko==1.14.1
+pathtools==0.1.2
+pycrypto==2.6.1
+pytz==2014.7
+six==1.8.0
+tzlocal==1.1.1
+watchdog==0.7.0
+wsgiref==0.1.2

requirements.couchbase.txt

@@ -0,0 +1,2 @@
+-e git+git://github.com/couchbase/couchbase-python-client#egg=couchbase-python-client
+-r requirements.base.txt

requirements.dev.txt

@@ -1,24 +1,9 @@
-apscheduler<3.0
-bottle>=0.12.6
-cherrypy
-coverage
-clint
-dill>=0.2b1
-dnspython
-jinja2
-hiredis
-natural
-redis
-requests
-parsedatetime==1.1.2
-pyasn1
-pyasn1_modules
-sleekxmpp>=1.2
+-r requirements.txt
 
 # Dev only
 pyandoc
 mkdocs
 fabric
 flake8
 mock
-nose
+nose

requirements.txt

@@ -1,33 +1,3 @@
-bottle==0.12.7
-clint==0.3.7
-dill==0.2.1
-dnspython==1.12.0
 hiredis==0.1.4
-natural==0.1.5
 redis==2.10.3
-requests==2.4.1
-parsedatetime==1.1.2
-pyasn1==0.1.7
-pyasn1-modules==0.0.5
-sleekxmpp==1.3.1
-APScheduler==2.1.2
-CherryPy==3.6.0
-Fabric==1.10.0
-Jinja2==2.7.3
-Markdown==2.3.1
-MarkupSafe==0.23
-PyYAML==3.10
-argh==0.25.0
-args==0.1.0
-ecdsa==0.11
-futures==2.1.6
-ghp-import==0.4.1
-greenlet==0.4.4
-paramiko==1.14.1
-pathtools==0.1.2
-pycrypto==2.6.1
-pytz==2014.7
-six==1.8.0
-tzlocal==1.1.1
-watchdog==0.7.0
-wsgiref==0.1.2
+-r requirements.base.txt

setup.py

@@ -9,9 +9,12 @@
 SOURCE_DIR = os.path.join(ROOT_DIR)
 
 reqs = []
-with open("requirements.txt", "r+") as f:
-    for line in f.readlines():
-        reqs.append(line.strip())
+for req_file in ("requirements.base.txt", "requirements.txt"):
+    with open(req_file, "r+") as f:
+        for line in f.readlines():
+            if line[0] == "-":
+                continue
+            reqs.append(line.strip())
 
 try:
     import pypandoc

will/main.py

@@ -146,6 +146,9 @@ def bootstrap(self):
                         time.sleep(0.5)
 
     def verify_individual_setting(self, test_setting, quiet=False):
+        if not test_setting.get("only_if", True):
+            return True
+
         if hasattr(settings, test_setting["name"][5:]):
             with indent(2):
                 show_valid(test_setting["name"])
@@ -189,9 +192,10 @@ def verify_environment(self):
             },
             {
                 "name": "WILL_REDIS_URL",
+                "only_if": getattr(settings, "STORAGE_BACKEND", "redis") == "redis",
                 "obtain_at": """1. Set up an accessible redis host locally or in production
 2. Set WILL_REDIS_URL to its full value, i.e. redis://localhost:6379/7""",
-            }
+            },
         ]
 
         puts("")
@@ -211,8 +215,8 @@ def verify_environment(self):
             puts("")
 
         puts("Verifying credentials...")
-        # Parse 11111_222222@chat.hipchat.com into id, where 222222 is the id.  Yup.
-        user_id = settings.USERNAME[0:settings.USERNAME.find("@")][settings.USERNAME.find("_") + 1:]
+        # Parse 11111_222222@chat.hipchat.com into id, where 222222 is the id.
+        user_id = settings.USERNAME.split('@')[0].split('_')[1]
 
         # Splitting into a thread. Necessary because *BSDs (including OSX) don't have threadsafe DNS.
         # http://stackoverflow.com/questions/1212716/python-interpreter-blocks-multithreaded-dns-requests
@@ -295,10 +299,10 @@ def bootstrap_storage_mixin(self):
         try:
             self.bootstrap_storage()
             with indent(2):
-                show_valid("Connection to %s successful." % settings.REDIS_URL)
+                show_valid("Bootstrapped!")
             puts("")
         except:
-            error("Unable to connect to %s" % settings.REDIS_URL)
+            error("Unable to bootstrap!")
             sys.exit(1)
 
     def bootstrap_scheduler(self):

will/mixins/storage.py

@@ -1,10 +1,8 @@
+import importlib
 import logging
-import redis
-import traceback
-import urlparse
 import dill as pickle
+import functools
 from will import settings
-from will.utils import show_valid, error, warn, note
 
 
 class StorageMixin(object):
@@ -13,58 +11,59 @@ def bootstrap_storage(self):
             if hasattr(self, "bot") and hasattr(self.bot, "storage"):
                 self.storage = self.bot.storage
             else:
-                # redis://localhost:6379/7
-                # or
-                # redis://rediscloud:asdfkjaslkdjflasdf@pub-redis-12345.us-east-1-1.2.ec2.garantiadata.com:12345
-                url = urlparse.urlparse(settings.REDIS_URL)
+                # The STORAGE_BACKEND setting points to a specific module namespace
+                # redis => will.storage.redis_backend
+                # couchbase => will.storage.couchbase_backend
+                # etc...
+                module_name = ''.join([
+                    'will.storage.',
+                    getattr(settings, 'STORAGE_BACKEND', 'redis'),
+                    '_storage'
+                ])
 
-                if hasattr(url, "path"):
-                    db = url.path[1:]
-                else:
-                    db = 0
-                max_connections = getattr(settings, 'REDIS_MAX_CONNECTIONS',
-                                          None)
-                connection_pool = redis.ConnectionPool(
-                    max_connections=max_connections, host=url.hostname,
-                    port=url.port, db=db, password=url.password
-                )
-                self.storage = redis.Redis(connection_pool=connection_pool)
+                # XXX TODO How to handle import errors here?
+                # Since this is required to work (I think), I've just let
+                # the exceptions bubble up for now.
+                storage_module = importlib.import_module(module_name)
 
-    def save(self, key, value, expire=0):
-        if not hasattr(self, "storage"):
-            self.bootstrap_storage()
+                # Now create our storage object using the bootstrap function
+                # from within the import
+                self.storage = storage_module.bootstrap(settings)
 
+    def save(self, key, value, expire=None):
+        self.bootstrap_storage()
         try:
-            if expire:
-                ret = self.storage.setex(key, pickle.dumps(value), expire)
-            else:
-                ret = self.storage.set(key, pickle.dumps(value))
-
-            return ret
-        except:
-            logging.critical("Unable to save %s: \n%s" %
-                             (key, traceback.format_exc()))
+            return self.storage.save(key, pickle.dumps(value), expire=expire)
+        except Exception:
+            logging.exception("Unable to save %s", key)
 
     def clear(self, key):
-        if not hasattr(self, "storage"):
-            self.bootstrap_storage()
-        return self.storage.delete(key)
+        self.bootstrap_storage()
+        try:
+            return self.storage.clear(key)
+        except Exception:
+            logging.exception("Unable to clear %s", key)
 
     def clear_all_keys(self):
-        if not hasattr(self, "storage"):
-            self.bootstrap_storage()
-        return self.storage.flushdb()
+        self.bootstrap_storage()
+        try:
+            return self.storage.clear_all_keys()
+        except Exception:
+            logging.exception("Unable to clear all keys")
 
     def load(self, key, default=None):
-        if not hasattr(self, "storage"):
-            self.bootstrap_storage()
-
+        self.bootstrap_storage()
         try:
-            val = self.storage.get(key)
+            val = self.storage.load(key)
             if val is not None:
                 return pickle.loads(val)
+            return default
+        except Exception:
+            logging.exception("Failed to load %s", key)
 
-        except:
-            logging.warn("Unable to load %s" % key)
-
-        return default
+    def size(self):
+        self.bootstrap_storage()
+        try:
+            return self.storage.size()
+        except Exception:
+            logging.exception("Failed to get the size of our storage")

will/plugins/admin/storage.py

@@ -7,23 +7,27 @@ class StoragePlugin(WillPlugin):
     @respond_to("^How big is the db?", admin_only=True)
     def db_size(self, message):
         self.bootstrap_storage()
-        self.say("It's %s." % self.storage.info()["used_memory_human"], message=message)
+        self.say("It's %s." % self.storage.size(), message=message)
 
     @respond_to("^SERIOUSLY. Clear (?P<key>.*)", case_sensitive=True, admin_only=True)
     def clear_storage(self, message, key=None):
         if not key:
             self.say("Sorry, you didn't say what to clear.", message=message)
         else:
-            self.clear(key)
             self.say("Ok. Clearing the storage for %s" % key, message=message)
+            res = self.clear(key)
+            if res not in (None, True, False):
+                self.say("Something happened while clearing: %s" % res, message=message)
 
     @respond_to("^SERIOUSLY. REALLY. Clear all keys.", case_sensitive=True, admin_only=True)
     def clear_all_keys_listener(self, message):
         self.say(
             "Ok, I'm clearing them. You're probably going to want to restart me."
             "I just forgot everything, including who I am and where the chat room is.", message=message
         )
-        self.clear_all_keys()
+        res = self.clear_all_keys()
+        if res not in (None, True, False):
+            self.say("Something happened while clearing all keys: %s" % res, message=message)
 
     @respond_to("^Show (?:me )?(?:the )?storage for (?P<key>.*)", admin_only=True)
     def show_storage(self, message, key=None):