Support rsyncing generated repo to remote server

We want to support the option to, as the last build step, rsync the
contents of the build directory (+report.html, status_report.html)
to a backup site, defined in the configuration. This is to allow
DLRN to be executed in a non internet-accessible server, while at
the same time make its generated repos available from the outside.

This change adds two options to the projects.ini configuration file:

- rsyncdest: destination path (e.g. root@serverX:/home/repos). If not
  set, no rsync will be attempted.
- rsyncport: SSH port to use to connect to destination server. By default
  22

Change-Id: If7bb350b7b4f9f9e8d35724950e85fe6ecfcedf9

dlrn/shell.py

@@ -79,7 +79,8 @@
 default_options = {'maxretries': '3', 'tags': None, 'gerrit': None,
                    'templatedir': os.path.join(
                        os.path.dirname(os.path.realpath(__file__)),
-                       "templates")
+                       "templates"),
+                   'rsyncdest': '', 'rsyncport': '22'
                    }
 
 
@@ -623,7 +624,6 @@ def run(program, cp, commit, env_vars, dev_mode, use_public, bootstrap,
 
 
 def build(cp, packages, commit, env_vars, dev_mode, use_public, bootstrap):
-
     # Set the build timestamp to now
     commit.dt_build = int(time())
 
@@ -711,6 +711,32 @@ def build(cp, packages, commit, env_vars, dev_mode, use_public, bootstrap):
                                    os.path.join(datadir, "repos")),
                    target_repo_dir + "_")
         os.rename(target_repo_dir + "_", target_repo_dir)
+
+    rsyncdest = cp.get("DEFAULT", "rsyncdest")
+    rsyncport = cp.getint("DEFAULT", "rsyncport")
+    if rsyncdest != '':
+        # We are only rsyncing the current repo dir to rsyncdest
+        rsyncpaths = []
+        # We are inserting a dot in the path after repos, this is used by
+        # rsync -R (see man rsync)
+        commitdir_abs = os.path.join(datadir, "repos", ".",
+                                     commit.getshardedcommitdir())
+        rsyncpaths.append(commitdir_abs)
+        # We also need report.html, status_report.html and styles.css
+        for filename in ['report.html', 'status_report.html', 'styles.css']:
+            filepath = os.path.join(datadir, "repos", ".", filename)
+            rsyncpaths.append(filepath)
+
+        rsh_command = '-e ssh -p %s' % rsyncport
+        for dirname in rsyncpaths:
+            try:
+                sh.rsync('-avzR', '--delete',
+                         rsh_command,
+                         dirname, rsyncdest)
+            except Exception as e:
+                logger.warn('Failed to sync directory %s to %s ,'
+                            'got error %s' % (dirname, rsyncdest, e))
+
     return built_rpms, notes
 
 

doc/source/installation.rst

@@ -57,6 +57,8 @@ The configuration file looks like this:
     maxretries=3
     target=centos
     gerrit=yes
+    rsyncdest=
+    rsyncport=22
 
 * ``datadir`` is the directory where the packages and repositories will be created.
 
@@ -80,6 +82,14 @@ The configuration file looks like this:
   review when a build fails. See next section for details on how to
   configure gerrit to work.
 
+* ``rsyncdest`` if set, specifies a destination path where the hashed repository
+  directories created by DLRN will be synchronized using ``rsync``, after each commit build.
+  An example would be ``root@backupserver.example.com:/backupdir``.
+  Make sure the user running DLRN has access to the destination server using passswordless SSH.
+
+* ``rsyncport`` is the SSH port to be used when synchronizing the hashed repository. If
+  ``rsyncdest`` is not defined, this option will be ignored.
+
 Configuring for gerrit
 ++++++++++++++++++++++
 

projects.ini

@@ -11,3 +11,5 @@ templatedir=./dlrn/templates
 maxretries=3
 tags=
 #tags=mitaka
+rsyncdest=
+rsyncport=22