Override $ref in external reference

[unional]

When I'm writing some json schema and try to organize them, I notice current we cannot override the reference of external references, resulting limitation in extending the schema.

For example, I have define this little functional programming language just-func: https://github.com/justland/just-func-json/blob/main/json-schema/just-func.jsonc

In the schema, one of the definition looks like this:

// just-func.jsonc
    "JustUnaryFunctions": {
      "type": "array",
      "items": [
        {
          "type": "string",
          "enum": ["not","ret"],
          "description": "Basic unary functions:\nnot: logical not\nret: retrieve value defined by `let`"
        },
        {
          "$ref": "#/definitions/JustFunc"
        }
      ],
      "additionalItems": false
    }

Notice I have a "$ref": "#/definitions/JustFunc" here.

When using this language, I would extend it by defining additional functions, e.g.:

// my-app.jsonc
{
  "$schema": "http://json-schema.org/draft-07/schema",
  "description": "Validation schema for my-app written in just-func",
  "definitions": {
    "MyJustFunc": {
      "oneOf": [{
        "$ref": "just-func.jsonc/#/definitions/JustFunc"
      }, {
        "$ref": "#/definitions/MyExtensions"
      }]
    },
    "MyExtensions": {
      // ... snip ...
    }
  }
}

The problem is, if I use one of the default definition provided in just-func.jsonc,
they does not recognize the functions I defined in MyExtensions because their reference is (just-func.jsonc)#/definitions/JustFunc instead of (my-app.jsonc)#/definitions/MyJustFunc.

To solve this, we need a way to override the reference something akin to:

{
  "$override": {
    "just-junc.jsonc": {
      "#/definitions/JustFunc": "/#/definitions/MyJustFunc" // using `/#` to indicate it is a local reference
    }
  }
}

🌷

[gregsdennis]

@unional it looks like you might be using https://github.com/tcorral/JSONC, is that right? I think your question might be more specific to that library than JSON Schema in general.

JSON Schema doesn't define functions or a way to reference them. $ref for JSON Schema is merely a pointer to another schema object.

[unional]

Not using that. I think the question is about json schema. It is about how to override the reference because in these use cases we need to extend or modify the reference.

It's similar to the concept of generics or composition, where the reference needs to be open for extension.

One way to do that is by providing an overriding mechanism.

Hope this helps. 🤗

[unional]

The jsonc in the example is just json with comments. It can be json or yaml.

[unional]

The function in the OP is referring to the function defined in just-func. Just think of it as an array or simply a data structure. What I'm defining is a simple functional programming language similar to clojure and lisp, where code (function) is just data.

E.g.:

['=', 1, 2] evaluates to false

So, say my application defines a function param/get where it will get query param from a web service call, and can be used like this: ['param/get', 'name']

Currently I won't able to use it like ['=', ['param/get', 'name'], 'unional'] because the $ref in JustBinaryFunctions (where = is defined) is pointing to JustFunc thus not recognizing param/get.

[unional]

The current workaround is to copy the content of just-func.jsonc to my-app.jsonc and modify all $ref of JustFunc to MyJustFunc.

That means the just-func.jsonc can only be used as a template.
It cannot be referenced and reused.

[gregsdennis]

It still seems there's some project-specific knowledge that you have that isn't pertinent to JSON Schema.

It's similar to the concept of generics or composition

JSON Schema is not designed for object-oriented concepts. It's a constraints system, which is effectively the opposite of ideals like polymorphism. Composition of schemas creates a more constrained object; it can't extend objects.

Just think of it as an array or simply a data structure.

But as I said, in JSON Schema, $ref can only point to another schema object. It can't point to random data.

I think what you want is to define a new vocabulary with a new keyword that can reference data. (Or better yet, implement mine in JS.)

(Finally, as a side note, you may want to checkout JSON Logic. It does some of the things you're mentioning, but not references.)

[unional]

Thanks for the JSON Logic link.

It is similar concept.
The question is about extensibility.

Another difference is that we are talking about defining a JSON schema that describes the data structure, e.g. { 'var': 'temp' } in JSON logic.

It will have the same problem if it opens the system and make it extensible.

[unional]

But as I said, in JSON Schema, $ref can only point to another schema object. It can't point to random data.

It is pointing to a schema object, as in "$ref": "#/definitions/JustFunc".

The situation I'm describing is that the reference needs to be overridden (or "extends" with oneOf) in order to be reusable.

[unional]

Composition of schemas creates a more constrained object; it can't extend objects.

Maybe this is the limitation.
I can't reuse it by composition and extend it at the same time.

This rules-out a whole class of reusing scenarios.

[unional]

Want to point out that this is not necessary a single edge case.
Today it is not possible to extend schema for plugin configurations due to the same limitation.
For example, babel supports plugins and each plugin can have their addition settings.

{
  "presets": [
    ["@babel/preset-typescript", {
      "allowNamespaces": true // cannot specify schema for this
    }],
  ]
}

Today the only way to do this is to "pollute" the base babel json schema with all the plugin's setting, which is not ideal.

The same thing happens to other configs, such as tsconfig.json for TypeScript, adding IDE specific settings.

[karenetheridge]

Having a need to "override $ref" is something that you might be able to fulfill with the $recursiveRef and $recursiveAnchor keywords in draft 2019-09, or $dynamicRef and $dynamicAnchor in draft 2020-12. The metaschemas themselves use these keywords to dynamically extend the definition of the overall schema (which is recursively referenced by all applicator keywords such as not and allOf).

[unional]

Thanks. That's would be great.
I saw the draft but maybe because it is too new, I can't find any additional examples of it.

I looked at the example there but not sure if I grasp the idea. 😛 .

Do you know of any additional writeup about that feature?

thx

[Relequestual]

Thanks. That's would be great.
I saw the draft but maybe because it is too new, I can't find any additional examples of it.

I looked at the example there but not sure if I grasp the idea. 😛 .

Do you know of any additional writeup about that feature?

thx

It is one of the most tricky aspects of JSON Schema, but once you grasp it, it makes total sense.
It's probably easier to look at the example in the appendix of the specification itself...

https://tools.ietf.org/html/draft-bhutton-json-schema-00#appendix-C

It's likely that you may need to read it 3 or 4 times, and draw diagrams.
We're happy to help on our slack server (or here) should you have follow up questions.

---

As an aside, there was recently a webinar on OpenAPI 3.1 where they showed using $ref and overriding specific properties. Unfortunately, this was inaccurate, and can only be done in OpenAPI 3.1 when using a "Reference Object" and not a "Schema Object". I don't know is this has contributed to your confusion here or not.

[unional]

Thanks. Yes I joined that webinar too.
The $ref override mentioned there is about overriding specific properties in the scope of OpenAPI.
This is about dynamically pointing to a different Schema Object. So I think we are good. :)

https://tools.ietf.org/html/draft-bhutton-json-schema-00#appendix-C

Yes, that's what I saw too.

Let me try to explain it in my own words to see if it make sense.

To simplify things, I refer the strict-tree schema as strict-tree.json and the tree schema as tree.json.

Since strict-tree.json has:

{
  "$ref": "tree",
  "unevaluatedProperties": false
}

It states the whole strict-tree.json is "extending from tree.json while restricting any unevaluated properties.

Then, in the /children/items in tree.json, it has $dynamicRef: #node which is defined by $dynamicAnchor: node in the same document.

If $dynamicAnchor: node was not defined in tree.json and it actually defined like this:

// tree.json
{
  "type": "object",
  "properties": {
    "children": {
      "type": "array",
      "items": {
        "$ref": "#definitions/node"
      }
  },
  "definitions": {
    "node": { "type": "string" } // just for example
  }
}

Than it is referencing the definitions/node schema object.

One question: if the $dynamicAnchor: node was not defined in tree.json but $dynamicRef is used, i.e.:

{
  "properties": {
    "children": {
      "type": "array",
      "items": { "$dynamicRef": "#node" }
    }
  }
}

Would this be invalid due to the missing $dynamicAnchor?

The spec then mentioned:

In this case, the outermost resource also has a "node" fragment defined by "$dynamicAnchor". Therefore instead of resolving the "$dynamicRef" to "https://example.com/tree#node", we resolve it to "https://example.com/strict-tree#node".

So my guess is that the $dynamicAnchor: node defined in strict-tree.json is saying:

"Hey, the tree.json schema I based on has this dynamic anchor node, so as I also defining this anchor here, when tree.json uses $dynamicRef to look up node, use mine instead.

With that said, here is my attempt to implement my use case (simplified to just demonstrate the key points):

// just-func.jsonc
{
  "$dynamicAnchor": "JustFunc",
  "oneOf": [{
    "$ref": "#definitions/JustEq"
  }],
  "definitions": {
    "JustEq": {
      "type": "array",
      "items": [{
         "type": "string",
         "const": "eq"
      }, {
        "$dynamicRef": "#JustFunc"
      }, {
        "$dynamicRef": "#JustFunc"
     }],
    "additionalItems": false
  }
}

And in an application (my-app.jsonc) where I add a new definition called JustParamGet:

// my-just-func.jsonc
{
  "$dynamicAnchor": "JustFunc",
  "oneOf": [{
    "$ref": "#/definitions/JustParamGet"
  }, {
    "$ref": "just-func.jsonc"
  }],
  "definitions": {
    "JustParamGet": {
      "type": "array",
      "items": [{
        "type": "string",
        "const": "param/get"
      }, {
        "$dynamicRef": "#JustFunc"
      }]
    }
  }
}

// my-app-spec.jsonc
{
  "$schema": "http://json-schema.org/draft/2020-12/schema",
  "$id": "my-app",
  "type": "object",
  "properties": {
    "test": {
      "$ref": "my-just-func.jsonc"
    }
  }
}

UPDATE: removed the 3 attempts and only keep the latest attempt I have currently in https://github.com/justland/just-func