KeyError with nested patternProperties

[j616]

When an object using patternProperties is further down the tree than another patternProperties field you get a KeyError.

Example schema:-

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "type": "object",
    "patternProperties": {
        "^foo:": {
            "type": "object",
            "properties": {
                "baz": {
                    "type": "object",
                    "patternProperties": {
                        "^b": {
                            "type": "object"
                        }
                    }
                },
                "bit": {
                    "type": "object"
                }
            },
            "required": [
                "baz",
                "bit"
            ]
        }
    }
}

Example document to validate against it :-

{
    "foo:bar": {
        "baz": {
            "bat": {
            }
        },
        "bit": {
        }
    }
}

This results in :-

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 38, in validate
KeyError: 'bit'

The following schema where the second patternProperties is replaced with a regular properties validates correctly:-

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "type": "object",
    "patternProperties": {
        "^foo:": {
            "type": "object",
            "properties": {
                "baz": {
                    "type": "object",
                    "properties": {
                        "bat": {
                            "type": "object"
                        }
                    }
                },
                "bit": {
                    "type": "object"
                }
            },
            "required": [
                "baz",
                "bit"
            ]
        }
    }
}

The following where the first patterProperties is replaced also validates correctly:-

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "type": "object",
    "properties": {
        "foo:bar": {
            "type": "object",
            "properties": {
                "baz": {
                    "type": "object",
                    "patternProperties": {
                        "^b": {
                            "type": "object"
                        }
                    }
                },
                "bit": {
                    "type": "object"
                }
            },
            "required": [
                "baz",
                "bit"
            ]
        }
    }
}

[j616]

Doing some more digging with this. Turned out I was still on 1.6. Now on 2.3. The test case above no longer fails for me but I'm still getting the same sort of error on slightly more complex cases. I'll try and put together a new test case.

[j616]

I don't have much more useful info yet, but it does seem that the problem is non-deterministic. Running the same time multiple times results in different key errors (in the example above, they would be with keys at the baz/bit level) or occasionally results in the document validating successfully...

[j616]

Any chance someone could take a look at this?

[FrederikP]

Just discovered a bug related to this by using your example schema and data and debugging into the generated code (using compile_to_code). The key error is reproducable for me.
There is an issue with variables being overwritten, because in python the scopes of variables are not very strict. I'm not very familiar with the code, but maybe I can find a way to fix this.

The issue seems to be with self.l('for key, val in {variable}.items():'): from generate_pattern_properties. When these statements are nested, key and val get overwritten in certain circumstances. In your example this leads to val being assigned an empty object in the deeper object and then when it wants to check for bit, it uses this empty object instead of the correct one that was assigned earlier.

Also looking at the way this is built, I'm not sure that similiar issues can not be forced when it comes to nested properties with the same property names. Fully classifying all variables based on the properties would fix this.

e.g. data.item.foo.bar_value, data.item.foo.bar_key

[horejsek]

I plan to look at it tomorrow.

[FrederikP]

Okay, my first naive idea was to use something like:
{name}_key, {name}_val
but the name template variable seems to be only made for use in strings and not in the code as it can depend on one more formatting step.

I'll let you look into it @horejsek . If you have any questions about what I found out, let me know. When debugging the generated code using the example from this issue, the problem becomes clear quite quickly.

[horejsek]

Thanks @FrederikP, you found the bug. :-) The same problem was on more places. It's fixed in master. I plan to do more stuff tomorrow, do you want to release it today or can you wait till tomorrow, @j616?

[j616]

I can wait until tomorrow. For now it's useful to know I don't have to find a work around and to know I'm not doing anything silly on my end! Thanks.