idea for expected failure tests

failures/README.md

@@ -0,0 +1,49 @@
+## Expected Failure Tests
+
+The tests in this folder are expected to cause a failure of some sort in an implementation.
+
+Failures can occur when attempting to load the schema or validate an instance.
+
+## Structure
+
+Each file is a JSON array of test scenarios.
+
+Each scenario has a description, a schema, and an instance that should exercise the keyword being tested.
+
+## Running the Suite
+
+It's recommended that a test runner perform two tasks for each scenario:  load the schema and evaluate an instance.
+
+The scenario passes if the implementation raises an error condition during these tasks.
+
+## Examples
+
+### Schema Load Error
+
+This test presents an invalid schema.
+
+```json
+{
+    "description": "cannot be number",
+    "schema": { "items": 42 },
+    "instance": [ "foo" ]
+}
+```
+
+An implementation may use whatever means in order to determine that it is invalid, including meta-schema evaluation, static typing, or another mechanism.
+
+Because some implementations may choose not to validate a schema until instance evaluation, an instance is provided to exercise the `items` keyword.
+
+### Evaluation Error
+
+This test provides a technically valid schema, however an implementation should be able to detect the infinite loop and raise an error.
+
+```json
+{
+    "description": "infinite loop $ref",
+    "schema": { "$ref": "#" },
+    "instance": {}
+}
+```
+
+While this schema will likely pass any loading validation, it should definitely raise an error when evaluating an instance.

failures/draft6/items.json

@@ -0,0 +1,37 @@
+[
+    {
+        "description": "cannot be number",
+        "schema": { "items": 42 },
+        "instance": [ "foo" ]
+    },
+    {
+        "description": "cannot be string",
+        "schema": { "items": "value" },
+        "instance": [ "foo" ]
+    },
+    {
+        "description": "cannot be null",
+        "schema": { "items": null },
+        "instance": [ "foo" ]
+    },
+    {
+        "description": "array form cannot contain number",
+        "schema": { "items": [ 42 ] },
+        "instance": [ "foo" ]
+    },
+    {
+        "description": "array form cannot contain string",
+        "schema": { "items": [ "value" ] },
+        "instance": [ "foo" ]
+    },
+    {
+        "description": "array form cannot contain null",
+        "schema": { "items": [ null ] },
+        "instance": ["foo" ]
+    },
+    {
+        "description": "array form cannot contain array",
+        "schema": { "items": [ [ true ] ] },
+        "instance": [ "foo" ]
+    }
+]

failures/draft6/ref.json

@@ -0,0 +1,53 @@
+[
+    {
+        "description": "cannot be object",
+        "schema": { "$ref": {} },
+        "instance": {}
+    },
+    {
+        "description": "cannot be array",
+        "schema": { "$ref": [] },
+        "instance": {}
+    },
+    {
+        "description": "cannot be boolean",
+        "schema": { "$ref": false },
+        "instance": {}
+    },
+    {
+        "description": "cannot be number",
+        "schema": { "$ref": 42 },
+        "instance": {}
+    },
+    {
+        "description": "cannot be null",
+        "schema": { "$ref": null },
+        "instance": {}
+    },
+    {
+        "description": "infinite loop $ref",
+        "schema": { "$ref": "#" },
+        "instance": {}
+    },
+    {
+        "description": "infinite two-part loop $ref",
+        "schema": {
+            "$ref": "#/definitions/a",
+            "definitions": {
+                "a": { "$ref": "#/definitions/b" },
+                "b": { "$ref": "#/definitions/a" }
+            }
+        },
+        "instance": {}
+    },
+    {
+        "description": "unresolvable reference - pointer",
+        "schema": { "$ref": "#/definitions/a" },
+        "instance": {}
+    },
+    {
+        "description": "unresolvable reference - anchor",
+        "schema": { "$ref": "#foo" },
+        "instance": {}
+    }
+]