Two possible improvements to type coercion

[gtuccini]

It would be useful to have a coercible string representation for null. "null" or '' would work just fine.

I think also that the coercion should happen only when the subject doesn't match any of the allowed types.
Currently, given the following subject

{
	"value": "true"
}

and the following schema

{
	"properties": {
		"value": {
			"type": ["boolean", "string"]
		}
	},
	"type": "object"
}

"value" is coerced to boolean, even if its type -- string -- is among the allowed ones.

What do you think?

[erayd]

@shmax - type coercion is yours; what do you think of this? IMO both are excellent ideas.

[shmax]

Will definitely look into these issues this weekend. My first 10-second impression is that coercing to null for a string would violate the type (as null is not a string), so at best it would only be valid if you had some kind of mixed type in your schema. Not sure about the second point, but I can tell you that the current logic we have in place doesn't really have any opinion about the order, so adding some logic like you suggest probably would be no worse. More soon. Thanks for the contribution.

[erayd]

Re 10 second impression - isn't that the point of the coercion feature in the first place? If coercing "true" to a boolean is OK, that feels fairly equivalent to coercing "null" (or an empty string) to a null.

[shmax]

Hmm, well I don't think he really gave us enough detail about what he wants for the null case, but I hope we can agree that coercing string "null" to actual null for type string is a non-starter, because again, you would be coercing to something that is the wrong type. And further, I think you would be asking for a lot of bugs in your client app. Suppose you have a form for validating user names on your social site and a user happens to want to call himself "null". Perfectly legitimate string, but it would get converted to null by the coercion feature and cause a black hole somewhere.

The only coercion case involving strings and the type "null" that I can think of that would make sense would be to coerce empty string to type null. Our pals over at ajv seem to agree: http://take.ms/QYQEo

Anyway, I would be happy to add that feature if it would help. Will get back to you on the other point.

[erayd]

...I hope we can agree that coercing string "null" to actual null for type string is a non-starter...

Oh, I certainly wasn't suggesting that - we agree there completely! If the schema demands a string, and the document being validated supplies a string, then it should be left well alone.

What I meant (and what I hope the OP was asking for) was that if the schema demands a null (i.e. defines something with the explicit type null), but the document contains a string "null" (or an empty string), then type-coercing that string into an actual null to force compliance seems reasonable, given that type-coercion is what the user asked for when they enabled CHECK_MODE_COERCE_TYPES.

[erayd]

For the record, I disagree with ajv on not coercing "null" to null, on the grounds that "true" and "false" are already considered valid targets for coercion, and "null" is qualitatively exactly the same type of situation.

[erayd]

Do we care about coercion reversibility? That could be an argument against coercing "null".

[shmax]

Hmm, I just don't know. You could be right. One thing to consider is that the stringification of null is sort of language dependent, and it's only a keyword in certain languages. PHP doesn't seem too keen to do it:

echo http_build_query(array(
  'foo'=>'bar',
  'baz'=>'boom',
  'cow'=>null
));
// "foo=bar&baz=boom

but if you want to force it to a string, it comes out in caps:

var_export(null); // "NULL"

Javascript does:

console.log(encodeURIComponent(null)); // "null"

Python (not an expert, apologies):

from urllib.parse import urlencode
url = urlencode({'pram1': 'foo', 'param2': None})

print(url) # "param2=None&pram1=foo"

So I guess I wonder if teaching json-schema about "null" might be pushing things a little too far away from language agnosticism...

[erayd]

I do feel that this one is your call - it's 'your' feature, you did all the initial research and work around it, and it's not something I use myself.

My personal opinion is that we should coerce when (and only when) the schema defines a type, validation would fail without coercion, and the mapping from the input data to the correct value of the schema type is obvious. But I'd like to defer to your preference here, whatever that may be.

...but if you want to force it to a string, it comes out in caps...

var_export isn't returning a stringified value, it's returning valid PHP code (you'll note that NULL is not quoted). Caps follows convention, but is not required - NULL is case-insensitive in PHP.

So I guess I wonder if teaching json-schema about "null" might be pushing things a little too far away from language agnosticism...

That's a really good point, although again I feel it falls into the same basket as coercing "true" / "false" to boolean. If we refuse to support "null", then why should we support those? The only argument I can think of to not support "null", given the boolean thing, is that the boolean coercion is directly reversible, whereas "null" is not. So it would depend whether reversibility matters here.

[shmax]

NULL is case-insensitive in PHP

That was exactly my point. "null" and "NULL' are equivalent, so if we support coercing "null" then sooner or later someone is going to ask us to coerce 'NULL', and then after that someone will ask for "None", then "nil", and on, and on. Now we have this wad of strings that we will consider to be "null", thus weakening the "null" type's ability to do its job (suppose you're on a Python box, and a request comes in from some PHP client with "null". Rather than catch what is clearly an invalid value in the Python environment, it passes coercion with flying colors).

That's a really good point, although again I feel it falls into the same basket as coercing "true" / "false" to boolean. If we refuse to support "null"

Because "true" and "false" are fairly universal. I don't know of any major (non-deliberately oddball) languages that don't have them as keywords. "null" is a keyword in some languages, but not all, and even the ones that do use it don't necessarily stringify it when building query params.

Rather than beat it to death, why don't we start with coercing empty string to null, and go from there?

[gtuccini]

I'll try to explain my use case a bit.

Suppose you have a list of entities, whose attributes comprises the (integer) attribute "rank".

The list is accessible at entities.html and by default lists all the entities - i.e. there is no filter set on "rank".

You can optionally set or remove a filter on "rank" through a GET form, which generates the following urls:

• entities.html?rankisgreaterthan=[integer-serialized-as-string]
• entities.html?rankisgreaterthan=[empty-string]

I'd like to coerce and validate the query string using the following schema:

{
	"properties": {
		"rankisgreaterthan": {
			"default": null,
			"type": ["null", "integer"]
		}
	},
	"required": ["rankisgreaterthan"],
	"type": "object"
}

For this to work empty-string has to be coerced to null. There is no ambiguity, however, because null is explicitely listed among the expected types.

[shmax]

Okay, so that's coercing empty string to null, which should be fine, right?

[gtuccini]

I just read your last comments. I concur with your preference for '' :)

[erayd]

Rather than beat it to death, why don't we start with coercing empty string to null, and go from there?

This sounds like a great idea - you're making really good points re the various strings that might represent null, so maybe that particular case is best left for now, until it's had more thought / someone specifically asks for it.