Problem with custom types: they are used for both instance and schema

[mitar]

It seems this package is using custom types for both instance and schema. This is making a problem for me because I have instances which are immutable objects (I use tuples instead of lists, and frozendict instead of dicts) and I would like to validate them against a JSON schema which are normal dicts and lists. But this does not seem possible because code uses things like: validator.is_type(items, "object") to check if schema is describing an object or not, but if object is a frozendict, this breaks validation. Even if my schema itself passed validation.

Example.

import frozendict
import jsonschema

schema = {
    'type': 'object',
    'properties': {
        'tags': {
            'type': 'array',
            'items': {
                'type': 'string'
            }
        }
    }
}

jsonschema.Draft4Validator.check_schema(schema)

validator = jsonschema.Draft4Validator(
    schema=schema,
    types=dict(jsonschema.Draft4Validator.DEFAULT_TYPES, **{
        'array': tuple,
        'object': frozendict.frozendict
    })
)

validator.validate(frozendict.frozendict({'tags': ('foobar',)}))

[Julian]

Hi.

This use case should work fine, can you show a runnable example and the errors you're encountering? What you've pasted has some obvious problems, like how you're only passing one argument to validate.

If you want both normal dicts (and your frozen ones) to be acceptable, you need to pass both, but if you just want to use your alternate types everywhere, including in your schema, you're free to do so.

[mitar]

What you've pasted has some obvious problems, like how you're only passing one argument to validate.

Ehm, it is a runnable example. I am passing only one argument because validator is an instance of Draft4Validator and has schema provided there.

[mitar]

If you want both normal dicts (and your frozen ones) to be acceptable, you need to pass both, but if you just want to use your alternate types everywhere, including in your schema, you're free to do so.

Yes, I understand that. But this issue is that I cannot provide one set of types for the instance, and the other set of types for the schema.

[Julian]

Ehm, it is a runnable example. I am passing only one argument because validator is an instance of Draft4Validator and has schema provided there.

Got it, sorry (would still have liked to see your error though).

Will have to dig in here, I can't quite tell what you mean, but from running it (and now looking at it more carefully), you're not actually passing in a valid schema for your validator, you're calling check_schema (which uses the default types), and under which your schema is valid, but then passing your schema into a validator where you're saying the types are different. You need to be validating your schema under the same types.

Whether you still get an error after doing that or not I'll have a look at, can't tell from your responses whether you've already done that.

[Julian]

So, I got a second to re-run this after making it valid by either deciding to allow both dicts and frozendicts or by converting your schema to use frozendicts, under your type definition.

In my case, I did the latter (and used the new type checker API which will be out any day now):

from frozendict import frozendict
from jsonschema.validators import extend
import jsonschema

schema = frozendict(
    {
        'type': 'object',
        'properties': frozendict(
            {
                'tags': frozendict(
                    {
                        'type': 'array',
                        'items': frozendict(
                            {
                                'type': 'string'
                            }
                        )
                    }
                ),
            }
        ),
    },
)
checker = jsonschema.Draft4Validator.TYPE_CHECKER.redefine(
    "array", lambda checker, value: isinstance(value, tuple),
).redefine(
    "object", lambda checker, value: isinstance(value, frozendict),
)

Validator = extend(jsonschema.Draft4Validator, type_checker=checker)
Validator.check_schema(schema)
validator = Validator(schema=schema)
validator.validate(frozendict({'tags': ('foobar',)}))

That works fine here, so I'm guessing all you need to do is follow the above on making sure you actually provide a valid schema under your redefinition.

Gonna close this because I don't think there's a bug here, but if you disagree or the above is unclear please do follow up, happy to help.

Thanks!

[mitar]

Hm, but still. Why would have schema to be defined in the same way as the document? Schema should be able to use dicts, while document should be able to use frozendicts? I think mixing schema and document type should not be necessary.

[mitar]

I made a workaround by extending my instance to allow both dicts and frozen dicts because I know that they cannot be mixed based on other logic. But it is just not nice.

[Julian]

@mitar because there's just one JSON type "object", and you need to say what it should mean -- I'm not sure what's not nice about it (why can't you specify your schema using the same types as your instances?) but if you really wanted to, you can define a metaschema where your JSON type for a schema is not type: object it's type: schemaobject and provide a separate definition for that, at which point you can probably separate the two from each other.

But otherwise, it's this way because yeah, you have just one type in JSON land, and you need to tell jsonschema what that one type corresponds to.

[mitar]

I'm not sure what's not nice about it

It is just strange that if you want to change what values are in the document you are trying to validate, you have to be changing how you describe the schema. It is a breaking a bit the abstraction.

why can't you specify your schema using the same types as your instances?

Because schema might be coming from JSON file using a regular json parser, while the instance might be coming from Python, constructed in Python. At least this is how I am using this library. Having a standard schema provided together with the code I load in to create a validator, which I then use to validate Python objects to adhere to this schema. And Python objects, coming from API users, should be immutable. While JSON schema is internal to validator and should just be in whatever it the easiest. I could be converting JSON schema to frozendicts and so on, like you suggested, but I think that this should be unnecessary.

But otherwise, it's this way because yeah, you have just one type in JSON land, and you need to tell jsonschema what that one type corresponds to.

For values. I agree with this. But I disagree with why JSON schema has to use the same type as what is used for values.

It is like if I would want to define in a programming language that some value should be of type A, now the programming language itself would require this type to be used internally to describe program's code. Very strange.

[Julian]

It is like if I would want to define in a programming language that some value should be of type A, now the programming language itself would require this type to be used internally to describe program's code. Very strange.

That's how JSON Schema (the specification) works. A schema is self descriptive, and a meta schema defines its types. If you want different types, you use a different JSON type.

Because schema might be coming from JSON file using a regular json parser, while the instance might be coming from Python, constructed in Python. At least this is how I am using this library. Having a standard schema provided together with the code I load in to create a validator, which I then use to validate Python objects to adhere to this schema. And Python objects, coming from API users, should be immutable. While JSON schema is internal to validator and should just be in whatever it the easiest. I could be converting JSON schema to frozendicts and so on, like you suggested, but I think that this should be unnecessary.

I happen to do exactly this myself (use pyrsistent.pmaps for schemas instead of dicts) but I do so consistently -- all schemas and all objects are immutable, and if I happen to be using json.load, I call pyrsistent.freeze immediately afterwards.

Anyways, I disagree here that there's anything missing from the library, sorry. If you'd like to concretely suggest something that you think would improve things I'm all ears.

[mitar]

No, it is OK. I have a working workaround for this. :-)