parsing recursive json schema

[gfarcieri]

I am using connexion and prance with openapi and a recursive json schema that looks like:

"information": {
    "type": "object",
    "properties": {
        "object": {
            "type": "array",
            "items": {
                "anyOf": [
                    {
                        "$ref": "#/definitions/object"
                    },
                    {
                        "$ref": "#/definitions/information"
                    }
                ]
            }
        },
        "feature": {
            "type": "array",
            "items": {
                 "$ref": "#/definitions/feature"
            }
        }
    }
}

where the object can be a simple object or a nested information
I have seen issue #55 and tried to set a max recursion limit together with an handle that return None but I get a prance.ValidationError

following is the traceback
traceback (most recent call last):
File "C:\code\test\test_openapi_json_schema_validation\env\lib\site-packages\prance_init_.py", line 224, in _validate_openapi_spec_validator
validate_v3_spec(self.specification)
File "C:\code\test\test_openapi_json_schema_validation\env\lib\site-packages\openapi_spec_validator\shortcuts.py", line 7, in validate
return validator_callable(spec, spec_url=spec_url)
File "C:\code\test\test_openapi_json_schema_validation\env\lib\site-packages\openapi_spec_validator\validators.py", line 48, in validate
raise err
openapi_spec_validator.exceptions.OpenAPIValidationError

and here is the code

def recursion_limit_handler_none(limit, refstring, recursions):
    return None

def get_bundled_specs(main_file: Path) -> Dict[str, Any]:
    parser = prance.ResolvingParser(str(main_file.absolute()),
    lazy=True,
    recursion_limit=1,
    recursion_limit_handler=recursion_limit_handler_none,
    backend='openapi-spec-validator')
    parser.parse()
    return parser.specification

app = connexion.FlaskApp(name)
app.add_api(
    get_bundled_specs(Path("api.yaml")),
    strict_validation=True,
    validate_responses=True,
    resolver=connexion.RestyResolver("cms_rest_api"))

It is not clear to me whether it is a problem due to the recursive json, that might not be supported or I made it wrong, or is due to my poor understanding on issue #55 and how to use it

Any help it is appreciated!!

[jfinkhaeuser]

Thanks for this! It looks good at first glance, but I can dig into it a bit more tomorrow.

But I didn't understand this in #55, and don't understand it now - why are you using a recursive spec in the first place? I'm still struggling to find a good use-case for one.

[gfarcieri]

Here is an example to try to explain why I am using recursion. Generally an information is a couple (object, feature) i.e. the feature of the object has a certain value. But an information can be itself an object to express for example how credible is that information. or to add other features that depend on the information itself

So, if I get the following information related to the position of a car

		"information": {
			"object": [
				{
					"type": "car",
					"model": "honda"
				}
			],
			"feature": [
				{
					"type": "position",
					"position": [
						{
							longitude: x,
							latitude: y
						}
					]
				}
			]
		}

I can then enrich this information by expressing its credibility. This reads like the "credibility" (feature) that the "position of the car" (object) is (x,y) is 0.5

		"information": {
			"object": [
				{
					"object": [
						{
							"type": "car",
							"model": "honda"
						}
					],
					"feature": [
						{
							"type": "position",
							"position": [
								{
									longitude: x,
									latitude: y
								}
							]
						}
					]
				}
			],
			"feature": [
				{
					"type": "credibility",
					"credibility": [
						{
							"unit": "probability",
							"value": 0.5
						}
					]
				}
			]
		}

This allows the possibility of decorating the information as much as needed, where on each step one more feature can be evaluated based on the others

Issue #55 is an enhancement of issue #10. The way I understood it is that you can fix a limit to the schema recursion by defining an handler to return None once a recursion limit is reached.
One more thing to notice is that I get

openapi_spec_validator.exceptions.OpenAPIValidationError 

So I am wondering that maybe the parsing of the Json schema may results in a non valid schema.
I am using draft7. I was able to validate the schema and the message examples using other tools and it works fine.
Maybe I am missing something

Again thanks for your time

[gfarcieri]

I made it work... the error was indeed on another circular reference of the schema that did not show up in the traceback. Once I changed that everything was parsed properly
So no problem with prance (that is btw one of the best on the subject)

Thanks a lot for your time

[gfarcieri]

One more thing...

I had to change the handler

def recursion_limit_handler_none(limit, refstring, recursions):
    return None

to the following

def recursion_limit_handler_none(limit, refstring, recursions):
    return {}

[jfinkhaeuser]

Well, there might still be an issue with prance, because the test code seems to succeed with returning None - I will have to investigate why.

Unfortunately, #74 is what you effectively ran into.

Since prance wraps validators, and doesn't validate itself, it can only report the exceptions from those validators up. But it passes the resolved spec to the validators, for a bunch of reasons - which means that information as to what file and line number triggered the issue is simply lost. I'm looking into ways to improve that.

Sorry I didn't come back with something useful earlier. I'm glad you could proceed! I'll leave this issue as a reminder to investigate, though.

[jfinkhaeuser]

But back to the recursion limit handler: I should probably fix the documentation there. What it does is help deal with circular references. Referenced objects containing more references is handled up to... well, there's going to be a stack limit in Python, but practically indefinitely deep. It's circular references that trigger the handler.

So what prance does is walk down references and create a stack of them as it goes along. If the exact same object path (path in the spec tree) appears more than recursion_limit in this stack - typically 1 - prance considers this a reason to trigger the recursion_limit_handler. Changing the limit allows you to control the depth to which circular recursions are resolved, but the handler will trigger eventually.

You probably don't want this kind of circular reference.

What the handler does is return how the reference should be resolved. Returning None is an option, but the resulting spec may not be valid. If returning {} validates for you, that means the circular reference is one that should point to an object according to the specs, and an empty object that doesn't contain further references works just fine.

I hope this makes more sense. I really do need to fix the documentation here!

[gfarcieri]

Totally agree with what you said. Prance deals well with circular references but the resulting spec may not be valid for the validator.

The truth is that circular reference are as much of a pain to deal with as they are useful and elegant to describe a schema. Here is the classical example:

"definitions": {
    "person": {
        "type": "object",
        "properties": {
            "name": { "type": "string" },
            "children": {
                "type": "array",
                "items": { "$ref": "#/definitions/person" }
            }
        }
    }
}

But in the reality you deal with a finite number of recursion steps. So having the possibility to tune this in code is great. And the handler is perfect for modifying the resulting spec,, once a recursion limit is fixed, with something that doesn't affect the validation of the schema.

Thanks

[jfinkhaeuser]

Ah, ok, so you're actively exploiting prance to make the schema work! Fair enough, that's not a use case I considered :)