install.upgrade: implement pre-upgrade sanity check

Add a pre-upgrade sanity check to the install.upgrade task. This sanity check:

1. determines the version of Ceph installed on the remote ("installed_version")
2. determines the version of Ceph in the target gitbuilder repo
("upgrade_version")
3. if "installed_version" is greater than "upgrade_version", fail immediately

Since the upgrade itself cannot succeed if upgrade_version < installed_version,
it's better to fail before attempting the upgrade.

Before this patch, we were failing in the verify_package_version() sanity check
which is run after attempting (and failing) to upgrade the packages.

Fixes: http://tracker.ceph.com/issues/16521
Signed-off-by: Nathan Cutler <ncutler@suse.com>

teuthology/task/install.py

@@ -14,6 +14,7 @@
 from teuthology.parallel import parallel
 from ..orchestra import run
 from . import ansible
+from distutils.version import LooseVersion
 
 log = logging.getLogger(__name__)
 
@@ -963,8 +964,22 @@ def upgrade_common(ctx, config, deploy_style):
             # FIXME: again, make extra_pkgs distro-agnostic
         pkgs += extra_pkgs
 
+        installed_version = packaging.get_package_version(remote, 'ceph-common')
+        upgrade_version = _get_gitbuilder_project(ctx, remote, node).version
+        log.info("Ceph {s} upgrade from {i} to {u}".format(
+            s=system_type,
+            i=installed_version,
+            u=upgrade_version
+        ))
+        if LooseVersion(installed_version) > LooseVersion(upgrade_version):
+            raise RuntimeError(
+                "An attempt to upgrade from a higher version to a lower one "
+                "will always fail. Hint: check tags in the target git branch."
+            )
+
         deploy_style(ctx, node, remote, pkgs, system_type)
         verify_package_version(ctx, node, remote)
+
     return len(remotes)
 
 docstring_for_upgrade = """"