jsonhandler: allow reading yaml data from disk (#254)

allow reading yaml data from disk with YAMLDiskLoadingJSONHandler

This commit aims to extend the jsonhandler to be able to read
data from disk if it is a yaml file.

 Simply replacing the loads call with yaml.safe_load is not enough
 due to the nature of the NaN checker requiring an unsafe load[1].

closes #253

Tests with the same filename were failing when running the tests
under pytest, see #255

To avoid running more than one response handler for the same type,
use a set to track the handers we have available.

docs/source/handlers.rst

@@ -22,6 +22,13 @@ string into a data structure in ``response_data`` that is used when
 evaluating ``response_json_paths`` entries in a test or doing
 JSONPath-based ``$RESPONSE[]`` substitutions.
 
+A YAMLDiskLoadingJSONHandler has been added to extend the JSON handler.
+It works the same way as the JSON handler except for when evaluating the
+``response_json_paths`` handle, data that is read from disk can be either in
+JSON or YAML format. The YAMLDiskLoadingJSONHandler is not enabled by default
+and must be added as shown in the :ref:`Extensions` section in order to be
+used in the tests.
+
 Further content handlers can be added as extensions. Test authors
 may need these extensions for their own suites, or enterprising
 developers may wish to create and distribute extensions for others
@@ -31,6 +38,8 @@ to use.
           that turns ``data`` into url-encoded form data suitable
           for POST and turns an HTML response into a DOM object.
 
+.. _Extensions:
+
 Extensions
 ----------
 
@@ -55,9 +64,14 @@ If pytest is being used::
     driver.py_test_generator(test_dir, intercept=simple_wsgi.SimpleWsgi,
                              content_handlers=[MyContenHandler])
 
-.. warning:: When there are multiple handlers listed that accept the
-             same content-type, the one that is earliest in the list
-             will be used.
+Gabbi provides an additional custom handler named YAMLDiskLoadingJSONHandler.
+This can be used for loading JSON and YAML files from disk when evaluating the
+``response_json_paths`` handle.
+
+.. warning:: YAMLDiskLoadingJSONHandler shares the same content-type as
+             the default JSONHandler. When there are multiple handlers
+             listed that accept the same content-type, the one that is
+             earliest in the list will be used.
 
 With ``gabbi-run``, custom handlers can be loaded via the
 ``--response-handler`` option -- see
@@ -118,6 +132,18 @@ If ``accepts`` is defined two additional static methods should be defined:
   attribute on the test, where it can be used in the evaluations
   described above (in the  ``action`` method) or in ``$RESPONSE`` handling.
   An example usage here would be to turn HTML into a DOM.
+* ``load_data_file``: Load data from disk into a Python data structure. Gabbi
+  will call this method when ``response_<something>`` contains an item where
+  the right hand side value starts with ``<@``. The ``test`` param allows you
+  to access the current test case and provides a load_data_file method
+  which should be used because it verifies the data is loaded within the test
+  diectory and returns the file source as a string. The ``load_data_file``
+  method was introduced to re-use the JSONHandler in order to support loading
+  YAML files from disk through the implementation of an additional custom
+  handler, see
+  :class:`~gabbi.handlers.yaml_disk_loading_jsonhandler.YAMLDiskLoadingJSONHandler`
+  for details.
+
 
 Finally if a ``replacer`` class method is defined, then when a
 ``$RESPONSE`` substitution is encountered, ``replacer`` will be

docs/source/jsonpath.rst

@@ -78,9 +78,21 @@ or::
 
 Examples like this can be found in one of gabbi's `own tests`_.
 
+If it is desired to load YAML files like the JSON ones above, two things must
+be done:
+
+#. The YAMLDiskLoadingJSONHandler custom content handler must be passed to the
+   driver through the content_handlers argument. See :ref:`Extensions` on how
+   to do this.
+#. The YAML files to load must be placed in a subdirectory to prevent the test
+   runner from consuming them as test files to run::
+
+    response_json_paths:
+        $: @<subdir/values.yaml
+
 When reading from disk you can apply the same JSONPath by adding a ':' to the
 end of your file name. This allows you to store multiple API responses into
-a JSON file to reduce file management when constructing your tests.
+a single file to reduce file management when constructing your tests.
 
 .. highlight:: json
 
@@ -118,6 +130,8 @@ Although placing more than one API response into a single JSON file may seem
 convenient, keep in mind there is a tradeoff in readability that should not
 be overlooked before implementing this technique.
 
+Examples like this can be found in one of gabbi's `yaml-from-disk tests`_.
+
 There are more JSONPath examples in :doc:`example` and in the
 `jsonpath_rw`_ and `jsonpath_rw_ext`_ documentation.
 
@@ -149,3 +163,4 @@ to quote the result of the substitution.
 .. _jsonpath_rw: http://jsonpath-rw.readthedocs.io/en/latest/
 .. _jsonpath_rw_ext: https://python-jsonpath-rw-ext.readthedocs.io/en/latest/
 .. _own tests: https://github.com/cdent/gabbi/blob/master/gabbi/tests/gabbits_intercept/data.yaml
+.. _yaml-from-disk tests: https://github.com/cdent/gabbi/blob/master/gabbi/tests/gabbits_handlers/yaml-from-disk.yaml

gabbi/case.py

@@ -191,7 +191,6 @@ def load_data_file(self, filename):
     def _assert_response(self):
         """Compare the response with expected data."""
         self._test_status(self.test_data['status'], self.response['status'])
-
         for handler in self.response_handlers:
             handler(self)
 

gabbi/handlers/base.py

@@ -12,7 +12,6 @@
 # under the License.
 """Base classes for response and content handlers."""
 
-
 from gabbi.exception import GabbiFormatError
 
 
@@ -112,3 +111,8 @@ def dumps(data, pretty=False, test=None):
     def loads(data):
         """Create structured (Python) data from a stream."""
         return data
+
+    @staticmethod
+    def load_data_file(test, file_path):
+        """Return the string content of the file specified by the file_path."""
+        return test.load_data_file(file_path)

gabbi/handlers/jsonhandler.py

@@ -56,6 +56,12 @@ def dumps(data, pretty=False, test=None):
     def loads(data):
         return json.loads(data)
 
+    @staticmethod
+    def load_data_file(test, file_path):
+        info = test.load_data_file(file_path)
+        info = six.text_type(info, 'UTF-8')
+        return json.loads(info)
+
     @staticmethod
     def extract_json_path_value(data, path):
         """Extract the value at JSON Path path from the data.
@@ -93,9 +99,7 @@ def action(self, test, path, value=None):
             if ':' in value:
                 value, rhs_path = value.split(':$', 1)
                 rhs_path = test.replace_template('$' + rhs_path)
-            info = test.load_data_file(value.replace('<@', '', 1))
-            info = six.text_type(info, 'UTF-8')
-            value = self.loads(info)
+            value = self.load_data_file(test, value.replace('<@', '', 1))
             if rhs_path:
                 try:
                     rhs_match = self.extract_json_path_value(value, rhs_path)
@@ -104,7 +108,7 @@ def action(self, test, path, value=None):
                                          'disk')
                 except ValueError:
                     raise AssertionError('right hand side json path %s cannot '
-                                         'match %s' % (rhs_path, info))
+                                         'match %s' % (rhs_path, value))
 
         # If expected is a string, check to see if it is a regex.
         is_regex = (isinstance(value, six.string_types) and

gabbi/handlers/yaml_disk_loading_jsonhandler.py

@@ -0,0 +1,40 @@
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+"""JSON-related content handling with YAML data disk loading."""
+
+import yaml
+
+import six
+
+from gabbi.handlers import jsonhandler
+
+
+class YAMLDiskLoadingJSONHandler(jsonhandler.JSONHandler):
+    """A ContentHandler for JSON responses that loads YAML from disk
+
+    * Structured test ``data`` is turned into JSON when request
+      content-type is JSON.
+    * Response bodies that are JSON strings are made into Python
+      data on the test ``response_data`` attribute when the response
+      content-type is JSON.
+    * A ``response_json_paths`` response handler is added. Data read
+      from disk during this handle will be loaded with the yaml.safe_load
+      method to support both JSON and YAML data sources from disk.
+    * JSONPaths in $RESPONSE substitutions are supported.
+    """
+
+    @staticmethod
+    def load_data_file(test, file_path):
+        info = test.load_data_file(file_path)
+        info = six.text_type(info, 'UTF-8')
+        return yaml.safe_load(info)

gabbi/suitemaker.py

@@ -217,10 +217,13 @@ def test_suite_from_dict(loader, test_base_name, suite_dict, test_directory,
 
     # Merge global with per-suite defaults
     default_test_dict = copy.deepcopy(case.HTTPTestCase.base_test)
+    seen_keys = set()
     for handler in handlers:
         default_test_dict.update(handler.test_base)
         if handler.response_handler:
-            response_handlers.append(handler.response_handler)
+            if handler.test_key_suffix not in seen_keys:
+                response_handlers.append(handler.response_handler)
+                seen_keys.add(handler.test_key_suffix)
         if handler.content_handler:
             content_handlers.append(handler.content_handler)
 

gabbi/tests/gabbits_handlers/cat.json

@@ -0,0 +1,4 @@
+{
+    "type": "cat",
+    "sound": "meow"
+}

gabbi/tests/gabbits_handlers/data.json

@@ -0,0 +1 @@
+{"foo": {"bár": 1}}

gabbi/tests/gabbits_handlers/pets.json

@@ -0,0 +1,10 @@
+[
+    {
+        "type": "cat",
+        "sound": "meow"
+    },
+    {
+        "type": "dog",
+        "sound": "woof"
+    }
+]

gabbi/tests/gabbits_handlers/subdir/data.yaml

@@ -0,0 +1,2 @@
+foo:
+  bár: 1

gabbi/tests/gabbits_handlers/subdir/pets.yaml

@@ -0,0 +1,4 @@
+- type: cat
+  sound: meow
+- type: dog
+  sound: woof

gabbi/tests/gabbits_handlers/subdir/values.yaml

@@ -0,0 +1,12 @@
+values:
+- pets:
+  - type: cat
+    sound: meow
+  - type: dog
+    sound: woof
+- people:
+  - name: chris
+    id: 1
+  - name: justin
+    id: 2
+

gabbi/tests/gabbits_handlers/yaml-from-disk.yaml

@@ -0,0 +1,29 @@
+# Test loading expected data from file on disk with JSONPath
+#
+defaults:
+  method: POST
+  url: /somewhere
+  request_headers:
+      content-type: application/json
+  verbose: True
+
+tests:
+    - name: yaml encoded value from disk
+      data: <@data.json
+      response_json_paths:
+          $.foo['bár']: <@subdir/data.yaml:$.foo['bár']
+
+    - name: json encoded value from disk
+      data: <@data.json
+      response_json_paths:
+          $.foo['bár']: <@data.json:$.foo['bár']
+
+    - name: yaml parital from disk
+      data: <@cat.json
+      response_json_paths:
+          $: <@subdir/pets.yaml:$[?type = "cat"]
+
+    - name: yaml partial both sides
+      data: <@pets.json
+      response_json_paths:
+          $[?type = "cat"].sound: <@subdir/values.yaml:$.values[0].pets[?type = "cat"].sound

gabbi/tests/gabbits_intercept/subdir/values.yaml

@@ -0,0 +1,12 @@
+values:
+- pets:
+  - type: cat
+    sound: meow
+  - type: dog
+    sound: woof
+- people:
+  - name: chris
+    id: 1
+  - name: justin
+    id: 2
+

gabbi/tests/test_handlers.py

@@ -21,6 +21,7 @@
 from gabbi.exception import GabbiFormatError
 from gabbi.handlers import core
 from gabbi.handlers import jsonhandler
+from gabbi.handlers import yaml_disk_loading_jsonhandler
 from gabbi import suitemaker
 
 
@@ -302,6 +303,22 @@ def test_response_json_paths_from_disk_json_path_fail(self):
         with self.assertRaises(AssertionError):
             self._assert_handler(handler)
 
+    def test_response_json_paths_yamlhandler(self):
+        handler = yaml_disk_loading_jsonhandler.YAMLDiskLoadingJSONHandler()
+        lhs = '$.pets[?type = "cat"].sound'
+        rhs = '$.values[0].pets[?type = "cat"].sound'
+        self.test.test_directory = os.path.dirname(__file__)
+        self.test.test_data = {'response_json_paths': {
+            lhs: '<@gabbits_handlers/subdir/values.yaml:' + rhs,
+        }}
+        self.test.response_data = {
+            "pets": [
+                {"type": "cat", "sound": "meow"},
+                {"type": "dog", "sound": "woof"}
+            ]
+        }
+        self._assert_handler(handler)
+
     def test_response_headers(self):
         handler = core.HeadersResponseHandler()
         self.test.response = {'content-type': 'text/plain'}

gabbi/tests/test_suitemaker.py

@@ -14,6 +14,8 @@
 import unittest
 
 from gabbi import exception
+from gabbi import handlers
+from gabbi.handlers import yaml_disk_loading_jsonhandler as ydlj_handler
 from gabbi import suitemaker
 
 
@@ -118,3 +120,42 @@ def test_dict_on_invalid_key(self):
             "invalid key in test: 'response_html'",
             str(failure.exception)
         )
+
+    def test_response_handlers_same_test_key_yaml_last(self):
+        test_yaml = {'tests': [{
+            'name': '...',
+            'GET': '/',
+            'response_json_paths': {
+                'foo': 'hello',
+                'bar': 'world',
+            }
+        }]}
+        handler_objects = []
+        ydlj_handler_object = ydlj_handler.YAMLDiskLoadingJSONHandler()
+        for handler in handlers.RESPONSE_HANDLERS:
+            handler_objects.append(handler())
+        handler_objects.append(ydlj_handler_object)
+        file_suite = suitemaker.test_suite_from_dict(
+            self.loader, 'foo', test_yaml, '.', 'localhost', 80, None, None,
+            handlers=handler_objects)
+        response_handlers = file_suite._tests[0].response_handlers
+        self.assertNotIn(ydlj_handler_object, response_handlers)
+
+    def test_response_handlers_same_test_key_yaml_first(self):
+        test_yaml = {'tests': [{
+            'name': '...',
+            'GET': '/',
+            'response_json_paths': {
+                'foo': 'hello',
+                'bar': 'world',
+            }
+        }]}
+        ydlj_handler_object = ydlj_handler.YAMLDiskLoadingJSONHandler()
+        handler_objects = [ydlj_handler_object]
+        for handler in handlers.RESPONSE_HANDLERS:
+            handler_objects.append(handler())
+        file_suite = suitemaker.test_suite_from_dict(
+            self.loader, 'foo', test_yaml, '.', 'localhost', 80, None, None,
+            handlers=handler_objects)
+        response_handlers = file_suite._tests[0].response_handlers
+        self.assertIn(ydlj_handler_object, response_handlers)