issues with float, None and scientific notation

[gijzelaerr]

Hi!

I'm trying validate some json I have. The json contains a mapping with float values. pykwalify probably correctly pykwalify doesn't accept float values like 1e-06 and None:

pykwalify.errors.SchemaError: <SchemaError: error code 2: Schema validation failed:
 - Value '1e-06' is not of type 'float'. Path: '/moresane_accuracy'.
 - Value 'None' is not of type 'float'. Path: '/sefd'.

So 2 questions (or maybe issues):

• isn't 1e06 just valid syntax? Python eats it as float:

>>> x = float(1e-06)
>>> type(x)
<type 'float'>

• Is it possible to give multiple types for a specified key in a mapping? I just can't figure it out. My none problem would be solved if i was able to set the type to 'float' or 'none'.

awesome, thanks!

[Grokzen]

Hi.

Can you provide an example of the data in yaml format that you are trying to validate? I think that if pyyaml do not automatically convert the data on load then i am not sure if it is currently possible or not. If you build a datastructure in python and use the Core object directly, your data should already be casted to a float as is.

About multiple types for mappings, it was not part (to my knowledge) of the original pykwalify standard. I have tried to not include to much new stuff into pykwalify that extends the original framework. But there might be 1 possibility that you could explore. You could possibly set required=False for the key and that would silence the errors some and None could be accepted. This however is not perfect becuase you might disable/silence some errors that should really be raised.

[gijzelaerr]

Hm, yes, you are right, yaml doesn't parse the scientific notation as a float but as a string.

example:

from pykwalify.core import Core
import yaml

validated = yaml.load("{'fluxrange': 1e-06, 'radius': null}")
validator = yaml.load("{'mapping': {'fluxrange': {'type': 'float'}, 'radius': {'type': 'float'}, }, 'type': 'map'}")

print(type(validated['fluxrange']))
print(type(validated['radius']))

c = Core(source_data=validated, schema_data=validator)
c.validate(raise_exception=True)

output:

<class 'str'>
validation.invalid
<class 'NoneType'>
 --- All found errors ---
["Value '1e-06' is not of type 'float'. Path: '/fluxrange'", "Value 'None' is not of type 'float'. Path: '/radius'"]
Traceback (most recent call last):
  File "/Users/gijs/Work/kliko/test.py", line 11, in <module>
    c.validate(raise_exception=True)
  File "/Users/gijs/Work/rodrigues/.virtualenv/lib/python3.5/site-packages/pykwalify/core.py", line 157, in validate
    error_msg=u'.\n - '.join(self.validation_errors)))
pykwalify.errors.SchemaError: <SchemaError: error code 2: Schema validation failed:
 - Value '1e-06' is not of type 'float'. Path: '/fluxrange'.
 - Value 'None' is not of type 'float'. Path: '/radius'.: Path: '/'>

[gijzelaerr]

Sorry, i've been mixing up parsing yaml and json in my example. here an other example

from pykwalify.core import Core
import json

validated = json.loads('{"fluxrange": 1e-06, "radius": null}')
validator = json.loads("""{"mapping": {"fluxrange": {"type": "float"}, "radius": {"type": "float"}}, "type": "map"}""")

print(type(validated['fluxrange']))
print(type(validated['radius']))

c = Core(source_data=validated, schema_data=validator)
c.validate(raise_exception=True)

output:

<class 'float'>
<class 'NoneType'>
validation.invalid
 --- All found errors ---
["Value 'None' is not of type 'float'. Path: '/radius'"]
Traceback (most recent call last):
  File "/Users/gijs/Work/kliko/test.py", line 15, in <module>
    c.validate(raise_exception=True)
  File "/Users/gijs/Work/rodrigues/.virtualenv/lib/python3.5/site-packages/pykwalify/core.py", line 157, in validate
    error_msg=u'.\n - '.join(self.validation_errors)))
pykwalify.errors.SchemaError: <SchemaError: error code 2: Schema validation failed:
 - Value 'None' is not of type 'float'. Path: '/radius'.: Path: '/'>

so the problem is indeed in the yaml parsing (shouldn't parse json with jaml parser anyway). When parsed with the json parser it works. Sorry for the confusion.

Still having support for multiple types for a value in a mapping would be very useful!

[gijzelaerr]

Related to your suggestion to setting the required filed to silence the None error, that doesn't seem to work:

from pykwalify.core import Core
validated = {"radius": None}
validator = {"mapping": {"radius": {"type": "float"}}, "type": "map", "required": False}
Core(source_data=validated, schema_data=validator).validate(raise_exception=True)

output:

validation.invalid
 --- All found errors ---
["Value 'None' is not of type 'float'. Path: '/radius'"]
Traceback (most recent call last):
  File "/Users/gijs/Work/kliko/test.py", line 4, in <module>
    Core(source_data=validated, schema_data=validator).validate(raise_exception=True)
  File "/Users/gijs/Work/rodrigues/.virtualenv/lib/python3.5/site-packages/pykwalify/core.py", line 157, in validate
    error_msg=u'.\n - '.join(self.validation_errors)))
pykwalify.errors.SchemaError: <SchemaError: error code 2: Schema validation failed:
 - Value 'None' is not of type 'float'. Path: '/radius'.: Path: '/'>

[Grokzen]

Hmm that is true that it should not silence them, the type validation will still fail, required for mappings is that data can be allowed to be missing.

The main problem with adding some kind of support for multiple type validations for a key is that how the synxat should be changed to still preserv backwards compatibility with kwalify but at the same time add this new feature and also at the same time not making the syntax super horrible.

[gijzelaerr]

so currently syntax for a mapping is:

type: map
mapping:
  key_one:
    type: str

How about adding a keyword types, which accepts a list of types:

type: map
mapping:
  key_one:
    types:
      - str
      - none

I don't see any backward compatibility issues there and it makes sense.

[Grokzen]

I like the idea, but i do think that problems could apear that makes it abit more complicated to code and work with.

Depending on what type you specify, you need to have some fields to support validation the specefied types. For example, if you want to specify both map, seq and str as valid types for a data structure (there might be several cases where this is valid) you need to have mapping and sequence added to the same level to continue to validation of the content inside that type. And if you take this one step further, there might be cases where values that is required to validate one time, can't be included or you want to have different values depending on the sub type. This would be even more problematic if you only want to validate scalar values and you want different values for range or unique.

[Grokzen]

The following commit 4372ade fixes the scientific notation part of the problem. The multiple types in mapping is still a open problem, but i have been thinking about it some and i might have some ideas on how to possibly support it, but it will take some time to test it out and possible implement it.

[gijzelaerr]

awesome, thanks!

[gijzelaerr]

let me know if you need some input or to share some thoughts.

[Grokzen]

@gijzelaerr Would you mind if i close this one as fixed because of the scientific notation fixes that is included in 1.5.1 and create a new one that investigates the support for multiple types in the same level?

[gijzelaerr]

sure!

2016-03-07 14:10 GMT+02:00 Grokzen notifications@github.com:

@gijzelaerr https://github.com/gijzelaerr Would you mind if i close
this one as fixed because of the scientific notation fixes that is included
in 1.5.1 and create a new one that investigates the support for multiple
types in the same level?

—
Reply to this email directly or view it on GitHub
#39 (comment).

Gijs Molenaar
http://pythonic.nl

[Grokzen]

Ticket created. Closing this because scientific notation was fixed in 1.5.1