Reference objects don't combine well with “nullable”

[Rinzwind]

Perhaps more of a question than an issue report, but I am not sure how to combine “nullable” with a reference object; I have the impression they don't combine well?

In Swagger-UI issue 3325 I posted an example with the following schema. The intent is to express that both “dataFork” and “resourceFork” contain a Fork, except that “resourceFork” can also be null, but “dataFork” cannot be null. As was pointed out though, the schema is not in accordance with the OpenAPI v3.0.0 specification, due to the use of JSON Schema's {"type":"null"} which has not been adopted in OpenAPI.

...
"File": {
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "dataFork": { "$ref": "#/components/schemas/Fork" },
    "resourceFork": {
      "anyOf": [
        { "type": "null" },
        {"$ref": "#/components/schemas/Fork" } ] },
    ...

I considered I can instead express the example as follows, which as far as I can tell is at least in accordance with the OpenAPI specification? But the use of “anyOf” with only one schema inside seems clumsy, is there a better way to handle this? Essentially, a better way to combine “nullable” with the reference object?

... 
"File": {
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "dataFork": { "$ref": "#/components/schemas/Fork" },
    "resourceFork": {
      "nullable": true,
      "anyOf": [
        { "$ref": "#/components/schemas/Fork" } ] },
    ...

[handrews]

@Rinzwind I use allOf with one entry regularly (for one entry, the *Of's behave the same and allOf is the simplest). And I do it exactly as you show, with the $ref in the allOf and everything else outside of it.

This is how boolean flags like nullable, readOnly, writeOnly, etc. are intended to be used AFAIK (certainly in JSON Schema for readOnly and as of draft-07 writeOnly).

See also #1389 for nullable vs "type": "null" discussion.

[tedepstein]

About the example given in the original post: if we're interpreting nullable as described in #1389, then I don't think an object with resourceFork: null would be valid against the File schema:

    File:
      type: object
      properties:
        name:
          type: string
        dataFork:
          $ref: '#/components/schemas/Fork'
        resourceFork:
          nullable: true
          anyOf:
          - $ref: '#/components/schemas/Fork'
          
    Fork:
      type: object
      description: Information aspect of a file...

I added a 'Fork' stub schema to illustrate. Fork itself is not nullable, it requires an object value.

(And BTW, even if Fork didn't specify a type, it's not clear to me that null is allowed. Does anyone know whether schemas without a type and without an explicit nullable: true would permit null values?)

In the File schema, resourceFork has nullable: true. But in OpenAPI Schema Object, as in JSON Schema, constraints are cumulative and cannot be relaxed or overridden. The allOf expression means that all assertions in the referenced subschema apply here. Those inherited assertions, whether implicit or explicit, prohibit a null value. Adding a new, more permissive type assertion on top of that doesn't change anything.

So I think the first attempted schema in the original post is closer to the mark. But we have to do something kind of awkward to allow either null or Fork:

    File:
      type: object
      properties:
        name:
          type: string
        dataFork:
          $ref: '#/components/schemas/Fork'
        resourceFork:
          oneOf:
          - $ref: '#/components/schemas/Fork'
          - $ref: '#/components/schemas/NullValue'
          
    Fork:
      type: object
      description: Information aspect of a file...
      
    NullValue:
      # Allow null values.
      nullable: true
      # Disallow any non-null value.
      not:
        anyOf:
        - type: string
        - type: number
        - type: boolean
        - type: object
        - type: array
          # Array schema must specify items, so we'll use an 
          # empty object to mean arrays of any type are included.
          items: {}

It might be cleaner to start with a nullable Fork object, and restrict it to non-null as needed:

    File:
      type: object
      properties:
        name:
          type: string
        dataFork:
          allOf:
          - nullable: false
          - $ref: '#/components/schemas/Fork'
        resourceFork:
          $ref: '#/components/schemas/Fork'
          
    Fork:
      type: object
      nullable: true
      description: Information aspect of a file...

[Rinzwind]

But in OpenAPI Schema Object, as in JSON Schema, constraints are cumulative and cannot be relaxed or overridden.

I'm not sure that's right. The JSON schema validation specification says that “validation keywords typically operate independent of each other” (section 4.3) (*), but then makes some exceptions.

The OpenAPI Specification 3.0.2 isn't exactly clear on how it defines “nullable”, but I think it would have to be an exception too. Otherwise “nullable” would seem to always conflict with other keywords, in particular “type”. For example, consider the schema {"nullable":true,"type":"object"}. The instance null would not match this schema if it has to match each keyword independently, as it matches the “nullable” keyword, but not the “type” keyword (null is not an object). In order for this schema to have the intended semantics, I think “nullable” would have to be defined as overriding other keywords: if “nullable” has the value true and the instance is null, it overrides all other keywords. So null would match the schema because null matches “nullable”, and “type” does not apply. Similarly, null would also match the schema {"nullable":true,"anyOf":[{"type":"object","nullable":false}]} because the “anyOf” does not apply.

(*) Section 4.3 in the version of the JSON schema validation specification linked to from the OpenAPI specification 3.0.2. In newer versions, the sentence moved to section 3.1.1.

But we have to do something kind of awkward to allow either null or Fork: […]

I hadn't considered that, if your schema “NullValue” is correct and allowed by OpenAPI, that seems like a way of sneaking the easier to understand {"type":"null"} back in!

[handrews]

Hi @Rinzwind , I'm one of the current primary editors of the JSON Schema spec (and yes, it's my fault that draft-08 is taking forever to appear 😭 )

What "validation keywords typically operate independently" is that in the majority of cases, each keyword's validation outcome can be determined without examining any other keyword.

As of draft-06 and later, that is true of any assertion keyword, such as type, maxLength, or required. These keywords produce their own pass/fail validation results.

Some applicator keywords (which apply subschemas and combine their assertion results, rather than producing their own results) do interact. The most notorious being that determining to which properties additionalProperties applies depends on properties and patternProperties. The fact that that inter-keyword dependency causes so many misunderstandings is good evidence of why we try to minimize this behavior.

Sadly, in draft-04 (which is effectively what OAS uses- there's not really a draft-05, it's a long story), the exclusiveMaximum and exclusiveMinimum assertion keywords were not independent keywords, which also causes problems. That has since been changed.

OAS's nullable is another assertion keyword that is not independent. And that causes many problems when you write complex schemas with *Of and similar keywords. Which is why we removed assertions with that behavior, and have done other things to mitigate the problems seen in the applicator keywords.

That's probably more than you wanted to know, but the upshot is that nullable (if indeed it's default is false and applies when type is absent) violates two fundamental principles of JSON Schema keyword design:

• The empty schema MUST successfully validate any and all possible instances
• Assertion keywords SHOULD function independently; failure to follow this guideline produces behavior that is very confusing in complex schemas

[Rinzwind]

OAS's nullable is another assertion keyword that is not independent. […]
nullable […] violates two fundamental principles of JSON Schema keyword design

OK, but regardless of the violations of nullable with respect to those principles, the intent of the OpenAPI 3.0.2 specification was at least that nullableis not independent, and kind of “overrides” other keywords such that null is validated by all of the following schemas?

• { "nullable": true, "type": "object" }
• { "nullable": true, "anyOf": [ { "type": "object" } ] }
• { "nullable": true, "anyOf": [ { "type": "object", "nullable": false } ] }
• { "nullable": true, "anyOf": [ { "$ref": "#/components/schemas/Fork" } ] }

[tedepstein]

@Rinzwind,

There are discussions in #1389 and #1900 essentially trying to figure out how to interpret nullable.

OK, but regardless of the violations of nullable with respect to those principles, the intent of the OpenAPI 3.0.2 specification was at least that nullableis not independent, and kind of “overrides” other keywords such that null is validated by all of the following schemas?

That's how I assumed it worked. But then I dug deeper and now I'm much less certain of how the founding fathers intended nullable to work, or if they ever discussed use cases cases where nullable might conflict with enum, how it might behave in allOf inheritance hierarchies, etc.

We could say that nullable always takes precedence over other constraints, such that those other constraints only apply to non-null values. If that's what it means, we should document that, and we should expect that it's going to break some implementations that assumed otherwise.

I think we're still waiting to hear from the TSC about this one.

[Rinzwind]

There are discussions in #1389 and #1900 essentially trying to figure out how to interpret nullable.

It's not so clear to me what the alternative interpretation(s) is / are by which any of the following schemas would not allow null, could you clarify?

1. { "nullable": true, "type": "object" }
2. { "nullable": true, "anyOf": [ { "type": "object" } ] }
3. { "nullable": true, "anyOf": [ { "type": "object", "nullable": false } ] }
4. { "nullable": true, "anyOf": [ { "$ref": "#/components/schemas/Fork" } ] }

I think we can assume that in any interpretation, at least schema 1 should allow null otherwise there's no point to “nullable”? The second and third schema are essentially the same, as “nullable” is specified as having false as its default value, and the fourth schema is just another variation on these two where a reference is used. So I assume the point of disagreement here would be whether there's a difference between combining “nullable” with “type” (schema 1) and combining “nullable” with “anyOf” (schemas 2,3,4)? I would argue that in both cases, there's a validation keyword that only allows objects, but which is overridden by the “nullable” keyword to also allow null.

[tedepstein]

It's not so clear to me what the alternative interpretation(s) is / are by which any of the following schemas would not allow null, could you clarify?

One of the interpretations under discussion is nullable: true --> type: [t, 'null']. Under that interpretation, your schema 1 would allow null. Schemas 2 and 3 would not. Schema 4 would only allow null if the referenced Fork schema is also nullable.

Taking schema 2 as an example:

1. Start with nullable: true and apply the interpretation nullable: true --> type: [t, 'null'].
   
   
   • In this case, there is no type assertion, so t is the list of non-null types:
     string, number, integer, boolean, object, and array.
     
     
   • We'd append null to that list, so we have:
     type: [string, number, integer, boolean, object, array, null].
     It's the complete list of all possible types, so it's equivalent to having no type constraint at all.
     
     
2. The "anyOf": [ { "type": "object" } ] applies independently. This disallows null.

So I assume the point of disagreement here would be whether there's a difference between combining “nullable” with “type” (schema 1) and combining “nullable” with “anyOf” (schemas 2,3,4)?

It's not limited to that. Interpretation of nullable also has implications in terms of how it combines with enum, and how it works with the other *Of applicators, including allOf which is commonly used for inheritance hierarchies.

I would argue that in both cases, there's a validation keyword that only allows objects, but which is overridden by the “nullable” keyword to also allow null.

The problem with that interpretation is that JSON Schema doesn't have any way of "overriding" a constraint. Constraints can be added, but they can't be taken away.

If you really want to introduce "override" as a whole new mechanism in OpenAPI schema, then we have no clean way of translating OpenAPI schemas to JSON Schemas. And in that case, API implementations, tools and libraries can't make use of standard JSON Schema validators; they have to use specialized OAS schema validators. And this mismatch makes for a steeper learning curve for developers moving from one language to the other. That's the gap we're trying to narrow, not just with nullable, but in other areas as well.

There is another possible interpretation that doesn't break with JSON Schema:
nullable: true --> anyOf: [s, type: 'null']

That is much closer to an override. In this case, s means "the rest of the schema, aside from nullable." So null values are allowed, and non-null values are subject to the other constraints in s, including type, enum, *Of, etc. All four of your example schemas would allow nulls in this case.

But there are some issues:

• We'd have to change the spec to explain that enum, *Of and other keywords are not equivalent to JSON Schema, because they are subject to modification by nullable.
• If we amend the OpenAPI spec to say that this is the official interpretation, we will break some current implementations. (That's true of any official interpretation, because implementers have been interpreting nullable in different ways.)
• Nullable types can never be true subtypes. We have this very counterintuitive situation where
  allOf: [ {$ref: '#/foo'} ], nullable: true
  is interpreted as
  anyOf: [ {allOf: [ {$ref: '#/foo'} ] }, {type: 'null'} ]
  The allOf gets "wrapped" by this interpretation. The schema is no longer guaranteed to be a valid instance of #/foo, so it can't be treated as a subtype. Code generators will have a hard time with this, and API designers will be unpleasantly surprised to see this interpretation.

[Rinzwind]

Nullable types can never be true subtypes. We have this very counterintuitive situation where {allOf: ['#/foo'], nullable: true} is interpreted as anyOf: [{type: 'null'}, {allOf: ['#/foo']}]

Perhaps I'm missing something, but I don't get why that would be counterintuitive? I opened this issue because I wanted to express the JSON schema { "anyOf": [ { "type": "null" }, { "$ref": "#Fork" } ], and wondered whether the same thing could be expressed by the OpenAPI schema { "nullable": true, "anyOf": [ { "$ref": "#Fork" } ] }. The fact that the latter would by definition be equivalent to the former would not seem counterintuitive to me at all, but exactly what I intended. (Strictly speaking, according to your transformation rule, the latter OpenAPI schema would be equivalent to the JSON schema { "anyOf" : [ { "anyOf" : [ { "$ref": "#Fork" } ] }, { "type" : "null" } ] }, but I don't think that makes a difference, because the ‘inner’ “anyOf” can be simplified to just the reference.)

[tedepstein]

I opened this issue because I wanted to express the JSON schema { "anyOf": [ { "type": "null" }, { "$ref": "#Fork" } ], and wondered whether the same thing could be expressed by the OpenAPI schema { "nullable": true, "anyOf": [ { "$ref": "#Fork" } ] }. The fact that the latter would by definition be equivalent to the former would not seem counterintuitive to me at all, but exactly what I intended.

In your case, you knew exactly what you wanted, and the wrapping of your schema into an anyOf gives you just that. But it's much less intuitive if you're using allOf to define subtypes. Unless you're truly "thinking in JSON schema" and you know exactly how nullable gets interpreted, you wouldn't expect the presence of nullable to break inheritance. But it does.

(Strictly speaking, according to your transformation rule, the latter OpenAPI schema would be equivalent to the JSON schema { "anyOf" : [ { "anyOf" : [ { "$ref": "#Fork" } ] }, { "type" : "null" } ] }, but I don't think that makes a difference, because the ‘inner’ “anyOf” can be simplified to just the reference.)

Yes, that's right.

[Rinzwind]

But it's much less intuitive if you're using allOf to define subtypes.

I'm not quite sure I get what you mean, but are you perhaps talking about an example such as the following? “Override” may have been a bad choice of words with respect to an example like this, but I did not mean to imply that this would allow “foo” to be null in the interpretation of “nullable” I had in mind. There are two schemas within the “allOf”, while the second one allows the instance {"foo": null}, the first one does not; therefore, the schema as a whole does not either.

{ "allOf": [
  { "type": "object", "properties": { "foo": { "type": "integer" } } },
  { "type": "object", "properties": { "foo": { "type": "integer", "nullable": true } } } ] }

This is, as far as I get, in keeping with your transformation rule to JSON schema:

{ "allOf": [
  { "type": "object", "properties": { "foo": { "type": "integer" } } },
  { "type": "object", "properties": { "foo": { "anyOf": [ { "type": "integer" }, { "type": "null" } ] } } } ] }

The same applies: {"foo": null} is valid against the second subschema, but not the first; therefore it is not valid against the schema as a whole.

[tedepstein]

@Rinzwind , I have to get back to my day job. ;-) If I have time, I'll try to parse what you've written here and respond.

But IMO, there is no happy solution that preserves nullable as a keyword, works in harmony with JSON Schema, behaves the way people want and expect it to, and keeps breaking changes to a minimum. The best solution is to deprecate nullable and replace it with type: [null, t].

[Rinzwind]

I have to get back to my day job. ;-) […]
The best solution is to deprecate nullable and replace it with type: [null, t].

Oh sure, no problem, and I would also prefer for OpenAPI to stick to JSON schema's {"type":"null"}. I was just trying to grasp your earlier statement that the OpenAPI schema I posted at the opening of this issue might not allow “resourceFork” to be null as I had intended.

Musings: the only possible benefit I see to “nullable” is that it makes it very easy to determine whether the instance null is allowed by a schema or not, one only needs to look at the schema's “nullable” keyword. Or at least this is the case if “nullable” would be defined according to these two transformation rules from OpenAPI schema to JSON schema:

{ "nullable": true, … } → { "anyOf": [ { … }, { "type": "null" } ] }
{ "nullable": false, … } → { "allOf": [ { … }, { "not": { "type": "null" } } ] }

Keep in mind that “nullable” is implicitly included in every OpenAPI schema with default value false.

This would imply though that the OpenAPI schema {"enum":[null]} would not allow null. It is, by default value, equivalent to {"enum":[null], "nullable": false}. By the above transformation rules, it would then be equivalent to a JSON schema that does not validate the instance null. That probably conflicts with some people's expectations.

We could instead use these two transformation rules:

{ "nullable": true, … } → { "anyOf": [ { … }, { "type": "null" } ] }
{ "nullable": false, … } → { … }

But that would imply losing the possible benefit of “nullable”. A schema like {"enum":[null],"nullable":false} would allow null, despite having the value false for “nullable”. It would not be sufficient to look at the “nullable” keyword's value to know whether the schema validates the instance null or not.

[Rinzwind]

One of the interpretations under discussion is nullable: true --> type: [t, 'null'].

I hadn't fully grasped what you meant by this. I'm not sure I do now; if you find the time to comment, I was wondering whether the following correctly reflects that interpretation w.r.t. to null for the following four OpenAPI schemas?

1. {"nullable": true, "enum":[42, null]}
   ⇒ null is valid
2. {"nullable": true, "enum":[42]}
   ⇒ null is not valid
3. {"nullable": false, "enum":[42, null]}
   ⇒ null is not valid
4. {"enum":[42, null]}
   ⇒ null is not valid

[tedepstein]

I was wondering whether the following correctly reflects that interpretation w.r.t. to null for the following four OpenAPI schemas?

1. {"nullable": true, "enum":[42, null]}
   ⇒ null is valid
2. {"nullable": true, "enum":[42]}
   ⇒ null is not valid
3. {"nullable": false, "enum":[42, null]}
   ⇒ null is not valid
4. {"enum":[42, null]}
   ⇒ null is not valid

Yes, exactly. :-)

In the fourth example, null is not valid because nullable: false is the default, even if type is not specified. If it weren't for that unfortunate design decision, nullable: true --> type: [t, 'null'] would be easy enough to reconcile with JSON Schema. But nullable: false by default blows the whole thing up.

[Rinzwind]

One of the interpretations under discussion is nullable: true --> type: [t, 'null'].

I was wondering whether the following correctly reflects that interpretation w.r.t. to null for the following four OpenAPI schemas?

2. {"nullable": true, "enum":[42]}
   ⇒ null is not valid

Yes, exactly. :-)

Ok, I think I got it then. I think I have to put the question back to @handrews, who was previously the first one to respond to this issue, saying:

I use allOf with one entry regularly (for one entry, the *Of's behave the same and allOf is the simplest). And I do it exactly as you show, with the $ref in the allOf and everything else outside of it.

This is how boolean flags like nullable, readOnly, writeOnly, etc. are intended to be used AFAIK (certainly in JSON Schema for readOnly and as of draft-07 writeOnly).

See also #1389 for nullable vs "type": "null" discussion.

@handrews: I'm not sure you intended to imply that the example I gave (opening comment of this issue) expresses (as intended) that both “dataFork” and “resourceFork” contain a Fork, except that “resourceFork” can also be null. Because, as far as I understand, this would imply that the OpenAPI schema { "nullable": true, "anyOf": [ { "$ref": "#/Fork" } ] } would be equivalent to the regular JSON schema { "anyOf" : [ { "type" : "null" }, { "$ref": "#/Fork" } ] }. You simultaneously seem to be, in the opening post of issue #1389, the originator of the idea that nullable is related to a different transformation of the “type” keyword, which according to @tedepstein implies the example does not express what was intended, as the schema would actually be equivalent to { "type": [ "object", "string", … , "null" ], "anyOf": [ { "$ref": "#/Fork" } ] }. I'm wondering what your take on this is?

Using a simpler example, I think the question is similar to whether this OpenAPI schema:

{ "nullable": true, "type": "integer", "enum": [ 42 ] }

… is equivalent to this JSON schema: (allows null)

{ "anyOf": [ { "type": "null" }, { "type": "integer", "enum": [ 42 ] } ] }

… or to this one: (does not allow null)

{ "type": [ "integer", "null" ], "enum": [ 42 ] }