Allow for missing values #60

Closed
scott2b opened this Issue Jul 22, 2012 · 8 comments

Comments

Projects
None yet
4 participants
@scott2b

scott2b commented Jul 22, 2012

It is not clear why colander strictly enforces the presence of non-required values. There should at least be an 'optional' parameter which would enable validation of schemata with non-required nodes without forcing undesired default or sentinel values.

@mcdonc

This comment has been minimized.

Show comment Hide comment
@mcdonc

mcdonc Sep 21, 2012

Member

Can you provide a bit of sample code that doesn't behave as you'd like/expect? Does your schema have "missing=" values?

Member

mcdonc commented Sep 21, 2012

Can you provide a bit of sample code that doesn't behave as you'd like/expect? Does your schema have "missing=" values?

@mcdonc

This comment has been minimized.

Show comment Hide comment
@mcdonc

mcdonc Sep 25, 2012

Member

No response, closing.

Member

mcdonc commented Sep 25, 2012

No response, closing.

@mcdonc mcdonc closed this Sep 25, 2012

@scott2b

This comment has been minimized.

Show comment Hide comment
@scott2b

scott2b Sep 25, 2012

Chris, sorry I did not get back to you on this. It was a weird week and my inbox is kind of piled up. The issue was that missing= is really treated as a default=, meaning that if the value is not provided, then a default is provided. My feeling is that it would be really useful to actually allow for missing values which do not have a default. The primary use case I have is in validating schema for NoSQL or similar solutions where you don't want empty values cluttering up the data. To do this, I am defaulting to None, and then just popping the keys with None values before pushing into a MongoDB.

This is inconvenient but it works. I think this really goes in the category of design choice more than bug.

scott2b commented Sep 25, 2012

Chris, sorry I did not get back to you on this. It was a weird week and my inbox is kind of piled up. The issue was that missing= is really treated as a default=, meaning that if the value is not provided, then a default is provided. My feeling is that it would be really useful to actually allow for missing values which do not have a default. The primary use case I have is in validating schema for NoSQL or similar solutions where you don't want empty values cluttering up the data. To do this, I am defaulting to None, and then just popping the keys with None values before pushing into a MongoDB.

This is inconvenient but it works. I think this really goes in the category of design choice more than bug.

@mcdonc

This comment has been minimized.

Show comment Hide comment
@mcdonc

mcdonc Sep 25, 2012

Member

Ah, right! I'l reopen this. It's a reasonable feature request (albeit one we might not be able to get around to in any predictable amount of time). I might suggest something like "missing=colander.drop" for child nodes of sequences and mappings. If subnode of a sequence or mapping deserializes to "colander.drop", the container implementation would just omit it from the deserialization value.

Member

mcdonc commented Sep 25, 2012

Ah, right! I'l reopen this. It's a reasonable feature request (albeit one we might not be able to get around to in any predictable amount of time). I might suggest something like "missing=colander.drop" for child nodes of sequences and mappings. If subnode of a sequence or mapping deserializes to "colander.drop", the container implementation would just omit it from the deserialization value.

@mcdonc mcdonc reopened this Sep 25, 2012

@scott2b

This comment has been minimized.

Show comment Hide comment
@scott2b

scott2b Sep 25, 2012

That seems like a good approach.

scott2b commented Sep 25, 2012

That seems like a good approach.

@kusut

This comment has been minimized.

Show comment Hide comment
@kusut

kusut Jan 29, 2013

Request extended.

Say you've implemented colander.drop and people are able to set it on their nodes. Would it be convenient to have a schema level setting to make all nodes dropable?

Here be example:

class Foo(Schema):
    bar = String
    baz = String

The idea is to make this schema works normally as it currently does, and provides a switch to make all nodes droppable when missing.

Maybe give deserialize in _SchemaNode another kwarg:

class _SchemaNode(object):
    deserialize(self, cstruct=null, drop=False)

So it will be like this:

>>> f = Foo()
>>> f.deserialize({'bar': 'abc'}, True) # all missings, in this case baz, will be dropped
{'bar': u'abc'}

Note: maybe raising invalid if all nodes are missing? Another kwarg?

This way, we can use the same schema on both creating and updating an object.

Is this a good idea? Thoughts?

kusut commented Jan 29, 2013

Request extended.

Say you've implemented colander.drop and people are able to set it on their nodes. Would it be convenient to have a schema level setting to make all nodes dropable?

Here be example:

class Foo(Schema):
    bar = String
    baz = String

The idea is to make this schema works normally as it currently does, and provides a switch to make all nodes droppable when missing.

Maybe give deserialize in _SchemaNode another kwarg:

class _SchemaNode(object):
    deserialize(self, cstruct=null, drop=False)

So it will be like this:

>>> f = Foo()
>>> f.deserialize({'bar': 'abc'}, True) # all missings, in this case baz, will be dropped
{'bar': u'abc'}

Note: maybe raising invalid if all nodes are missing? Another kwarg?

This way, we can use the same schema on both creating and updating an object.

Is this a good idea? Thoughts?

@AndreLouisCaron

This comment has been minimized.

Show comment Hide comment
@AndreLouisCaron

AndreLouisCaron Feb 6, 2013

+1 for the deserialize(..., drop=True) suggestion. To me, this is not a property of the schema, but rather a filter on the result of deserialization.

+1 for the deserialize(..., drop=True) suggestion. To me, this is not a property of the schema, but rather a filter on the result of deserialization.

@mcdonc

This comment has been minimized.

Show comment Hide comment
@mcdonc

mcdonc Aug 10, 2013

Member

The missing=colander.drop feature was implemented. I'm going to close this issue as a result. If the suggestions above need further consideration, please open new issues with them attached.

Member

mcdonc commented Aug 10, 2013

The missing=colander.drop feature was implemented. I'm going to close this issue as a result. If the suggestions above need further consideration, please open new issues with them attached.

@mcdonc mcdonc closed this Aug 10, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment