Merge branch 'feature/fixDigestEmail' of git.assembla.com:lp-changeby…

…us into feature/fixDigestEmail

Conflicts:
	scripts/digest_emailer.py

README.txt

@@ -53,6 +53,115 @@ AWS configuration:
     If thumbnails are being mirrored to S3, complete the info in the aws and media sections
     of the config file accordingly. Otherwise image uploads will be saved to the local volume only.
 
+-------------------------------------------------
+DEVELOPMENT RECOMMENDATIONS (not required, but helps)
+-------------------------------------------------
+Set up virtualenv and virtualenvwrapper so that python libs are easy to version,
+and so you don't have conflicts between different python projects:
+    http://www.doughellmann.com/projects/virtualenvwrapper/
+
+This allows you to do stuff like:
+    workon cbu
+which will switch to the cbu virtual environment.
+
+-------------------------------------------------
+DEPLOYMENT PROCESS
+-------------------------------------------------
+The deployment process for CBU uses Fabric:
+    pip install fabric
+
+It requires that you have at least fabric Fabric==1.2.0:
+    pip freeze | grep -i fabric
+
+Fabric uses a script and different configuration files. The default script 
+filename is fabfile.py, and configuration files are called "rcfile". To 
+facilitate this naming convention, all configuration files are rcfile.ENV, 
+where ENV is the environment to which to deploy. Eg. rcfile.demo would be
+the configuration set for the 'demo' environment.
+
+The fabric script is located in the root of the project as of this writing. 
+However, this location may change in future, since deployment scripts really 
+should be versioned separately from the code project.
+
+Configuration files are NEVER versioned with the project since they contain
+sensitive information. Configuration files (rcfiles) are versioned in a private
+repository. Please consult with a sysadmin for the location of this repo.
+
+SETTING UP FOR DEPLOYMENT:
+The deployment host is assumed to have the following:
+    * SSH key (or pem file) to connect to the target environment under the 
+      appropriate user contexts. These files are usually in:
+        ~/.ssh/localprojects/*.pem or ~/.ssh/localprojects/id_rsa
+      The actual file is defined in the rcfile, so it can be changed based on
+      user environment.
+      However, it is recommended that all deployment administrators use a 
+      standardized structure for storage of these files so as to limit
+      confusion or inconsistency.
+    * A source tree in a standardized location. The git source tree is expected
+      to be at ~/Projects/LP/<project-name>:
+        ~/Projects/LP/lp-changebyus
+      This is also configurable in the RCfile, so can be changed as needed.
+      [NOTE: this really should not be required since the deployer should 
+      have a way of pulling the code to a working location and creating the 
+      final configurations from it. Unfortunately this functionality is not yet
+      implemented.]
+    * The necessary RCfiles in a separate location. Eg.:
+        ~/Projects/LP/configs/lp-changebyus/rcfile.*
+
+Once the system is set up, the deployment steps are:
+    fab -f /path/to/fabfile.py \
+        --config=/path/to/rcfile.XX ENV \
+        <list of tasks>
+where:
+    /path/to/ : the path to the fabfile.py or rcfile.env. NOTE that fabric 
+                does NOT handle ~ (user-path) correctly, so always specify
+                full path. Relative paths from current location work fine 
+    rcfile.XX : The rcfile to use. Ensure that you provide the /path/to/
+                correctly. If you receive an error such as:
+                    "AttributeError: local_path"
+                this is indication that your rcfile was not read correctly.
+                Check your paths!
+    ENV       : The environment that the configuration is specific to. This 
+                is actually a task which processes all the system and 
+                required variables, and is usually one of live, prod, demo, dev.
+
+Deployment TASKS:
+The tasks for deployment are generally found at the top of the fabfile.py file,
+in the CookBook section. To get detailed information about the function of each
+available task, run:
+    fab --list
+
+Some of the more common sets of tasks are below.
+* Standard deployment of branch defined in rcfile, with webserver restart:
+    fab --config=/path/to/rcfile.demo demo \
+        bundle_code deploy_webapp deploy_configurations
+
+* Only deploy configuration updates, but not new code
+    fab --config=/path/to/rcfile.demo demo \
+        deploy_configurations
+
+* Deploy the latest code, change the 'current' symlink, but don't restart 
+  webserver. This is useful if you want to just put code up, but don't want
+  the webserver to know about it yet. 
+    fab --config=/path/to/rcfile.demo demo \
+        bundle_code deploy_app
+
+  You can then run the
+    fab --config=/path/to/rcfile.demo demo \
+        restart_webserver
+  task to ... um ... restart the webserver.
+
+* Deploy the latest code, but don't symlink, and don't do anything else:
+    fab --config=/path/to/rcfile.demo demo \
+       bundle_code upload_and_explode_code_bundle
+
+* Rollback the deploment (NOT YET TESTED):
+    fab --config=/path/to/rcfile.demo demo \
+        rollback
+  This will:
+    mv current next
+    mv previous current
+
 -------------------------------------------------
 Troubleshooting and Debugging Notes
 -------------------------------------------------
@@ -62,3 +171,5 @@ Error:
 Solution:
   This is invariably due to an AWS configuration issue. Either the key or the secret are wrong, or
   incorrectly configured in the config file.
+
+

etc/config.yaml.tmpl

@@ -66,6 +66,10 @@ email:
     # so that we don't spam live users.
     # Used by the digest_emailer.py script, if 'dev' == Yes | True
     digest_debug_recipients: sundar@localprojects.net
+
+    # determines max number retry attempts on failed email send
+    max_retries: 10
+
 
   # If both SMTP and AWS_SES are enabled then the system uses AWS first, and if
   # the aws send quota is close then we switch over to SMTP. This is handled in code

fabfile.py

@@ -161,7 +161,11 @@ def wrapper():
         # The scratch/work space for putting temporary stuff while we deploy from local dev
         env.tmp_path = "/tmp/%(application)s/releases" % env
         env.app_path = '%(deploy_to)s/%(application)s' % env
+
         env.current_path = "%(app_path)s/current" % env
+        env.previous_path = "%(app_path)s/previous" % env
+        env.next_path = "%(app_path)s/next" % env
+
         env.shared_path = "%(app_path)s/shared" % env
         env.run_path = "%(app_path)s/var/run" % env
 
@@ -561,13 +565,13 @@ def symlink_current_release():
     """
     require('release', provided_by=[deploy_webapp, setup_application])
     # if exists('%(app_path)s/previous' % env):
-    run('if [ -e %(app_path)s/previous ];then rm %(app_path)s/previous; fi; if [ -e %(app_path)s/current ];then mv %(app_path)s/current %(app_path)s/previous; fi' % env)
+    run('if [ -e %(previous_path)s ];then rm %(previous_path)s; fi; if [ -e %(current_path)s ];then mv %(current_path)s %(previous_path)s; fi' % env)
     # Link the shared config file into the current configuration
     for item in env.config_files:
         run('rm -f %s/releases/%s/%s/%s' % (env.app_path, env.release, item.get('path'), item.get('filename')))
         run('ln -nsf %s %s' % (os.path.join(env.app_path, 'etc', item.get('filename')), os.path.join(env.app_path, 'releases', env.release, item.get('path'), item.get('filename'))))
 
-    run('ln -s %(app_path)s/releases/%(release)s %(app_path)s/current' % env)
+    run('ln -s %(app_path)s/releases/%(release)s %(current_path)s' % env)
 
 def install_requirements():
     """
@@ -782,26 +786,22 @@ def secure_website():
 """
 Rollback deployed code tasks
 """
-def rollback(commit_id):
+def rollback(commit_id=None):
     """
     Rolls back to specified git/svn commit hash or tag or timestamp.
-    Deployments should be to timestamp-commitTag
+    Deployments should be to timestamp-commitTag.
+    If commit_id is not provided, move current to next and previous to current
 
-    There is NO guarantee we have committed a valid dataset for an arbitrary
-    commit hash.
+    Obviously there is NO guarantee we have deployed this commit-hash!
     """
-#    require('settings', provided_by=[production, staging])
-#    require('branch', provided_by=[master, branch])
-#
-#    maintenance_up()
-#    checkout_latest()
-#    git_reset(commit_id)
-#    gzip_assets()
-#    deploy_to_s3()
-#    refresh_widgets()
-#    maintenance_down()
-    pass
+    if commit_id is not None:
+        raise Exception('Rolling back to a specific commit-id is not yet supported')
+
+    run('if [ [ -e %(previous_path)s ] && [ -e %(current_path)s ] ];then mv %(current_path)s %(next_path)s && mv %(previous_path)s %(current_path)s; fi' % env)
 
+    stop_webserver()
+    start_webserver()
+
 """
 Database Related Tasks 
 """

scripts/digest_emailer.py

@@ -118,23 +118,28 @@ def _enable_aws_ses(self, settings):
     def htmlify(self, body):
         return "<html><head></head><body>%s</body></html>" % body 
 
-    def sendEmail(self, to=None, recipients=None, subject=None, body=None):
+    def sendEmail(self, to=None, recipients=None, subject=None, body=None, maxRetries=10):
         complete = False
+        failNum = 0
+
         while not complete:
-            try:
-                retval = Emailer.send(addresses=to,
-                            subject=subject,
-                            text=body,
-                            html=self.htmlify(body),
-                            from_name = self.MailerSettings.get('FromName'),
-                            from_address = self.MailerSettings.get('FromEmail'),
-                            bcc=recipients)
-                complete = True
-            except Exception, e:
-                # Most probably we got an SES error, which means we should wait and retry
-                logging.error(e)
-                time.sleep(1)
-                continue
+            complete = Emailer.send(addresses=to,
+                        subject=subject,
+                        text=body,
+                        html=self.htmlify(body),
+                        from_name = self.MailerSettings.get('FromName'),
+                        from_address = self.MailerSettings.get('FromEmail'),
+                        bcc=recipients)
+
+            if (not complete):
+                if (failNum < maxRetries):
+                    failNum += 1
+
+                    # Most probably we got an SES error, which means we should wait and retry
+                    time.sleep(1)
+                    pass
+                else:
+                    logging.error("Failed to send digest email '%s'. Quit after %s tries." % (subject, str(maxRetries)))
 
 
 
@@ -538,6 +543,11 @@ def sendDigests(self):
         Email out all the digests that we find, based on what's in self.Digests
         self.Digests should be an array of all the digests that need to be sent out
         """
+        if (self.Config.get('email').get('digest').get('max_retries')):
+            maxRetries = int(self.Config.get('email').get('digest').get('max_retries'))
+        else:
+            maxRetries = 10        
+
         for digest in self.Digests:
             currentDigest = None
             if type(digest) == dict:
@@ -555,7 +565,7 @@ def sendDigests(self):
                 body += "\n\n Development Information:\nRecipients are: %s\n" % ', '.join(recipients)
 
             logging.info('Digest recipients: %s' % recipients)
-            self.sendEmail(to=self.Config.get('email').get('from_email'), recipients=recipients, subject=subject, body=body)
+            self.sendEmail(to=self.Config.get('email').get('from_email'), recipients=recipients, subject=subject, body=body, maxRetries=maxRetries)
 
             if (digest.get('digest_id')):
                 # Means that we've been called from a database record