Use PlacementFixture in functional tests

Change the functional tests to use the PlacementFixture instead
of devstack as the source of a placement API. This speeds up
the tests considerably and lowers the number of dependencies.

There are four primary changes:

* For each test a PlacementFixture is instantiated, using the
  usual in-RAM db and in-process placement.

* Because of some exceedingly confusing optimizations in
  osc_lib and python-openstackclient, done to improve start
  up time, a session to placement was caching the service
  url. This meant that after a first test succeeded, every
  subsequent one would not because it was trying to talk
  to a fake hostname that was no longer being intercepted.

  The workaround for this was to monkeypatch the method
  in the ClientCache class which provides access to a client (per
  service-type). The replacement method makes a new client
  every time.

* The previous tests would subprocess out to a real call of
  the openstack command and then interpret the results.

  Now, a run() method on OpenStackShell is called instead.
  This accepts arguments in the same way, but we need to
  a) capture stderr and stdout, b) make a try/except for
  SystemExit to get some error responses (mostly from
  the argparse lib which has a tendency to exit for you
  instead of politely telling you it wants to), c) deal
  with errors from commands ourself rather than using
  exceptions from the subprocess module.

  Switching to this form means that logging becomes in-process
  and more visible. To accomodate this the Capture fixture
  from placement is used. This was chosen because we are already
  pulling in the PlacementFixture. If this seems icky, I can fix
  it with a local one. This was the shorter path.

* The legacy dsvm jobs have been removed in favor of "standard"
  functional jobs for 2.7 and 3.6 that require openstack/placement.
  The playbooks associated with the legacy jobs are removed.

  tox.ini is adjusted to reflect this new setup. Because tox-siblings
  functional is being used, we don't want to share tox envs with the unit
  tests. The 3.5 functional job is removed because we no longer target
  that.

After running these for a while it became clear that there were
intermittent failures being caused by subunit attachments being too
large. This was eventually traced back to logging from all packages
being set to DEBUG even when something else was requested. That was
traced back to a poor interaction between the way that osc does
logging and the way oslo_logging does logging (used by placement and
thus the placement fixture). The workaround, embodied in the
RESET_LOGGING list in osc_placement/tests/functional/base.py, is to
get and reset the log level for a subset of the packages that are
used.

Change-Id: I7deda200b372ff6a7ba67b0c4fa0e53c4fa16ffc
Story: 2005411
Task: 30428

.zuul.yaml

@@ -1,26 +1,3 @@
-- job:
-    name: osc-placement-dsvm-functional
-    parent: legacy-dsvm-base
-    run: playbooks/legacy/osc-placement-dsvm-functional/run.yaml
-    post-run: playbooks/legacy/osc-placement-dsvm-functional/post.yaml
-    timeout: 7200
-    required-projects:
-      - openstack/devstack-gate
-      - openstack/osc-placement
-
-- job:
-    name: osc-placement-dsvm-functional-py3
-    parent: legacy-dsvm-base
-    description: |
-      Runs the osc-placement functional tests in a python 3 devstack
-      environment.
-    run: playbooks/legacy/osc-placement-dsvm-functional-py3/run.yaml
-    post-run: playbooks/legacy/osc-placement-dsvm-functional-py3/post.yaml
-    timeout: 7200
-    required-projects:
-      - openstack/devstack-gate
-      - openstack/osc-placement
-
 - project:
     templates:
       - openstack-python-jobs
@@ -32,9 +9,17 @@
       - release-notes-jobs-python3
     check:
       jobs:
-        - osc-placement-dsvm-functional
-        - osc-placement-dsvm-functional-py3
+        - openstack-tox-functional:
+            required-projects:
+              - openstack/placement
+        - openstack-tox-functional-py36:
+            required-projects:
+              - openstack/placement
     gate:
       jobs:
-        - osc-placement-dsvm-functional
-        - osc-placement-dsvm-functional-py3
+        - openstack-tox-functional:
+            required-projects:
+              - openstack/placement
+        - openstack-tox-functional-py36:
+            required-projects:
+              - openstack/placement

osc_placement/tests/functional/base.py

@@ -10,14 +10,35 @@
 # License for the specific language governing permissions and limitations
 # under the License.
 
+import logging
 import random
+
+import fixtures
 import six
-import subprocess
 
+from openstackclient import shell
 from oslotest import base
+from placement.tests.functional.fixtures import capture
+from placement.tests.functional.fixtures import placement
 import simplejson as json
 
 
+# A list of logger names that will be reset to a log level
+# of WARNING. Due (we think) to a poor interaction between the
+# way osc does logging and oslo.logging, all packages are producing
+# DEBUG logs. This results in test attachments (when capturing logs)
+# that are sometimes larger than subunit.parser can deal with. The
+# packages chosen here are ones that do not provide useful information.
+RESET_LOGGING = [
+    'keystoneauth.session',
+    'oslo_policy.policy',
+    'placement.objects.trait',
+    'placement.objects.resource_class',
+    'placement.objects.resource_provider',
+    'oslo_concurrency.lockutils',
+    'osc_lib.shell',
+]
+
 RP_PREFIX = 'osc-placement-functional-tests-'
 
 # argparse in python 2 and 3 have different error messages
@@ -28,31 +49,77 @@
     ARGUMENTS_REQUIRED = 'the following arguments are required: %s'
 
 
+class CommandException(Exception):
+    def __init__(self, *args, **kwargs):
+        super(CommandException, self).__init__(args[0])
+        self.cmd = kwargs['cmd']
+
+
 class BaseTestCase(base.BaseTestCase):
     VERSION = None
 
-    @classmethod
-    def openstack(cls, cmd, may_fail=False, use_json=False):
-        result = None
-        try:
-            to_exec = ['openstack'] + cmd.split()
-            if use_json:
-                to_exec += ['-f', 'json']
-            if cls.VERSION is not None:
-                to_exec += ['--os-placement-api-version', cls.VERSION]
-
-            output = subprocess.check_output(to_exec, stderr=subprocess.STDOUT)
-            result = (output or b'').decode('utf-8')
-        except subprocess.CalledProcessError as e:
-            msg = 'Command: "%s"\noutput: %s' % (' '.join(e.cmd), e.output)
-            e.cmd = msg
+    def setUp(self):
+        super(BaseTestCase, self).setUp()
+        self.useFixture(capture.Logging())
+        self.placement = self.useFixture(placement.PlacementFixture())
+
+        # Work around needing to reset the session's notion of where
+        # we are going.
+        def mock_get(obj, instance, owner):
+            return obj.factory(instance)
+
+        # NOTE(cdent): This is fragile, but is necessary to work around
+        # the rather complex start up optimizations that are done in osc_lib.
+        # If/when osc_lib changes this will at least fail fast.
+        self.useFixture(fixtures.MonkeyPatch(
+            'osc_lib.clientmanager.ClientCache.__get__',
+            mock_get))
+
+        # Reset log level on a set of packages. See comment on RESET_LOGGING
+        # assigment, above.
+        for name in RESET_LOGGING:
+            logging.getLogger(name).setLevel(logging.WARNING)
+
+    def openstack(self, cmd, may_fail=False, use_json=False):
+        to_exec = []
+        # Make all requests as a noauth admin user.
+        to_exec += [
+            '--os-url', self.placement.endpoint,
+            '--os-token', self.placement.token,
+        ]
+        if self.VERSION is not None:
+            to_exec += ['--os-placement-api-version', self.VERSION]
+        to_exec += cmd.split()
+        if use_json:
+            to_exec += ['-f', 'json']
+
+        # Context manager here instead of setUp because we only want
+        # output trapping around the run().
+        self.output = six.StringIO()
+        self.error = six.StringIO()
+        stdout_fix = fixtures.MonkeyPatch('sys.stdout', self.output)
+        stderr_fix = fixtures.MonkeyPatch('sys.stderr', self.error)
+        with stdout_fix, stderr_fix:
+            try:
+                os_shell = shell.OpenStackShell()
+                return_code = os_shell.run(to_exec)
+            # Catch SystemExit to trap some error responses, mostly from the
+            # argparse lib which has a tendency to exit for you instead of
+            # politely telling you it wants to.
+            except SystemExit as exc:
+                return_code = exc.code
+
+        if return_code:
+            msg = 'Command: "%s"\noutput: %s' % (' '.join(to_exec),
+                                                 self.error.getvalue())
             if not may_fail:
-                raise
+                raise CommandException(msg, cmd=' '.join(to_exec))
 
-        if use_json and result:
-            return json.loads(result)
+        output = self.output.getvalue() + self.error.getvalue()
+        if use_json and output:
+            return json.loads(output)
         else:
-            return result
+            return output
 
     def rand_name(self, name='', prefix=None):
         """Generate a random name that includes a random number
@@ -79,10 +146,9 @@ def assertCommandFailed(self, message, func, *args, **kwargs):
         try:
             func(*args, **kwargs)
             self.fail('Command does not fail as required (%s)' % signature)
-
-        except subprocess.CalledProcessError as e:
+        except CommandException as e:
             self.assertIn(
-                message, six.text_type(e.output),
+                message, six.text_type(e),
                 'Command "%s" fails with different message' % e.cmd)
 
     def resource_provider_create(self,
@@ -99,9 +165,9 @@ def resource_provider_create(self,
         def cleanup():
             try:
                 self.resource_provider_delete(res['uuid'])
-            except subprocess.CalledProcessError as exc:
+            except CommandException as exc:
                 # may have already been deleted by a test case
-                err_message = exc.output.decode('utf-8').lower()
+                err_message = six.text_type(exc).lower()
                 if 'no resource provider' not in err_message:
                     raise
         self.addCleanup(cleanup)
@@ -166,9 +232,9 @@ def resource_allocation_set(self, consumer_uuid, allocations,
         def cleanup(uuid):
             try:
                 self.openstack('resource provider allocation delete ' + uuid)
-            except subprocess.CalledProcessError as exc:
+            except CommandException as exc:
                 # may have already been deleted by a test case
-                if 'not found' in exc.output.decode('utf-8').lower():
+                if 'not found' in six.text_type(exc).lower():
                     pass
         self.addCleanup(cleanup, consumer_uuid)
 
@@ -264,9 +330,9 @@ def trait_create(self, name):
         def cleanup():
             try:
                 self.trait_delete(name)
-            except subprocess.CalledProcessError as exc:
+            except CommandException as exc:
                 # may have already been deleted by a test case
-                err_message = exc.output.decode('utf-8').lower()
+                err_message = six.text_type(exc).lower()
                 if 'http 404' not in err_message:
                     raise
         self.addCleanup(cleanup)

osc_placement/tests/functional/test_allocation.py

@@ -10,9 +10,10 @@
 # License for the specific language governing permissions and limitations
 # under the License.
 
-import subprocess
 import uuid
 
+import six
+
 from osc_placement.tests.functional import base
 
 
@@ -68,11 +69,11 @@ def test_allocation_create(self):
     def test_allocation_create_empty(self):
         consumer_uuid = str(uuid.uuid4())
 
-        exc = self.assertRaises(subprocess.CalledProcessError,
+        exc = self.assertRaises(base.CommandException,
                                 self.resource_allocation_set,
                                 consumer_uuid, [])
         self.assertIn('At least one resource allocation must be specified',
-                      exc.output.decode('utf-8'))
+                      six.text_type(exc))
 
     def test_allocation_delete(self):
         consumer_uuid = str(uuid.uuid4())
@@ -91,9 +92,9 @@ def test_allocation_delete_not_found(self):
         consumer_uuid = str(uuid.uuid4())
 
         msg = "No allocations for consumer '{}'".format(consumer_uuid)
-        exc = self.assertRaises(subprocess.CalledProcessError,
+        exc = self.assertRaises(base.CommandException,
                                 self.resource_allocation_delete, consumer_uuid)
-        self.assertIn(msg, exc.output.decode('utf-8'))
+        self.assertIn(msg, six.text_type(exc))
 
 
 class TestAllocation18(base.BaseTestCase):