Skip to content

Proposal: Remove initial resolution step for dynamic references #1140

Closed
@jdesrosiers

Description

@jdesrosiers
Member

This issue is a spin-off of #1064. That conversation drifted a bit, so I'm moving some of the proposed amendments to that proposal into a separate issue.

Removing the bookending requirement for dynamic references opens the door for some additional considerations. $recrusiveRef didn't allow non-fragment URI parts. $dynamicRef added that in order to allow using $dynamicRef in a non-recursive schema. However, without the bookending requirement, the initial resolution step is no longer necessary (see #1064 (comment)). Since it's not necessary, there exists the possibility that a $dynamicRef doesn't need to be a URI at all. It makes sense for it to just be the dynamic anchor name. There's no reason to encode it into a URI. (Credit to @ssilverman for this idea) (See also #1064 (comment)).

We could take this a step further and make dynamic anchors URIs instead of plain name identifiers. Because dynamic anchors aren't scoped to the local document like traditional anchors, there's a possibility of name conflicts between schemas, so using URI identifiers would be safer. It's a similar concept to using URIs for link relations (for those who are familiar with those).

Activity

jdesrosiers

jdesrosiers commented on Oct 20, 2021

@jdesrosiers
MemberAuthor

Here's an example that sets a dynamic anchor with the identifier https://json-schema.hyperjump.io/anchor/list and dynamically references it.

{
  "$id": "https://json-schema.hyperjump.io/schema/list",
  "$schema": "https://json-schema.org/draft/2020-12/schema",

  "type": "object",
  "properties": {
    "list": {
      "type": "array",
      "items": { "$dynamicRef": "/anchor/llist" }
    },
    "nextPage": { "type": "integer" },
    "previousPage": { "type": "integer" },
    "perPage": { "type": "integer" },
    "page": { "type": "integer" }
  }
}
{
  "$id": "https://json-schema.hyperjump.io/schema/foo-list",
  "$schema": "https://json-schema.org/draft/2020-12/schema",

  "$ref": "/schema/list",

  "$defs": {
    "foo": {
      "$dynamicAnchor": "/anchor/list",
      "$ref": "/schema/foo"
    }
  }
}
jdesrosiers

jdesrosiers commented on Oct 21, 2021

@jdesrosiers
MemberAuthor

I couldn't help writing up what the spec might look like without bookending or initial resolution (#1142). It makes the concept much easier to describe. Personally, I think it's easier to understand as well.

jdesrosiers

jdesrosiers commented on Oct 21, 2021

@jdesrosiers
MemberAuthor

Here is an example using the hyper-schema links schema that shows what is lost by removing the initial resolution step. In this case we want "submissionSchema" to be extendible with a dynamic reference, but we also want it to default to a specific draft. With initial resolution, we can do this.

{
  "$id": "https://json-schema.org/draft/2020-12/links",
  "properties": {
    "submissionSchema": { "$dynamicRef": "/draft/2020-12/hyper-schema#meta" }
  }
}

The dynamic reference first resolves to the 2020-12 dialect where a "meta" dynamic anchor enters the dynamic scope. Since there are no other "meta" dynamic anchors already in this scope, the 2020-12 dialect is used. Without the initial resolution, we can get the same behavior, but it's more verbose.

{
  "$id": "https://json-schema.org/draft/2020-12/links",
  "properties": {
    "submissionSchema": { "$dynamicRef": "meta" }
  },
  "$defs": {
    "defaultDialect": {
      "$dynamicAnchor": "meta",
      "$ref": "/draft/2020-12/hyper-schema"
    }
  }
}

In order for there to be a default place for the dynamic reference to resolve, we have to explicitly set a dynamic anchor somewhere in the schema to define the default.

handrews

handrews commented on May 30, 2022

@handrews
Contributor

I just read back through this and the previous issue, as I never responded to your last example there which you have expanded on here.

I agree that this works!

While it's true that this particular case becomes more verbose, I think that's actually a good thing. The mutual recursion case is very hard to wrap your head around, and having explicit starting points helps make it a lot more clear what is happening.

Here's my last example from the other issue, updated based on your current proposal, just to make sure I'm reading things correctly (We'll just pretend like there's a 2020-12 hyper-schema even though there isn't):

{
  "$id": "https://json-schema.org/draft/2020-12/links",
  "$dynamicAnchor": "links-schema",
  "type": "object",
  "properties": {
    "submissionSchema": { "$dynamicRef": "meta" }
  },
  "$defs": {
    "defaultDialect": {
      "$dynamicAnchor": "meta",
      "$ref": "https://json-schema.og/2020-12/hyper-schema#meta"
    }
  }
}
{
  "$id": "https://json-schema.org/draft/2020-12/hyper-schema",
  "$dynamicAnchor": "meta",
  "type": "object",
  "properties": {
    "items": {"$dynamicRef": "meta"},
    "links": {
      "type": "array",
      "items": {"$dynamicRef": "links-schema"}
    }
  },
  "$defs": {
    "defaultLinksSchema": {
      "$dynamicAnchor": "links-schema",
      "$ref": "https://json-schema.og/2020-12/links#links-schema"
    }
  }
}

@jdesrosiers I see that your solution accomplishes several things compared to 2020-12:

  • No bookending requirement (which I realize was done as a PR for the previous issue)
  • $ref/$anchor/$id and $dynamicRef/$dynamicAnchor are now totally independent sets of functionality
  • It is still possible to make a concrete schema involving $dynamicRef, you just have to explicitly bridge the URI-based static reference and identification system to the non-URI-based dynamic one

I definitely find that last point appealing. TBH I'm not sure why I thought the two systems should interact.

I like it! Definitely an improvement 😃

jdesrosiers

jdesrosiers commented on May 31, 2022

@jdesrosiers
MemberAuthor

@handrews Thanks for looking this over. Your timing is perfect because I was hoping to get back to working on this issue soon. Your example is pretty much what I had in mind except there should be no need for the $refs to include the fragment.

What are your thoughts about making dynamic anchors URIs? My main concern is that people will find it confusing even tho there's precedence in the web space (link relations).

We could take this a step further and make dynamic anchors URIs instead of plain name identifiers. Because dynamic anchors aren't scoped to the local document like traditional anchors, there's a possibility of name conflicts between schemas, so using URI identifiers would be safer. It's a similar concept to using URIs for link relations (for those who are familiar with those).

handrews

handrews commented on May 31, 2022

@handrews
Contributor

there should be no need for the $refs to include the fragment

Oh, right- at least I think I understand. There are actually two things here:

  1. The target of these particular $refs is the resource root, so there's no need for a fragment at all
  2. If we did need a fragment, it would need to be declared with $anchor and not $dynamicAnchor because $dynamicAnchor no longer creates fragments (which I think is a good thing).

Are both of those correct? I think in some circumstances it would be fine to declare both a static $anchor target and a dynamic $dynamicAnchor target in the same schema, but agree it is unnecessary and probably confusing in this example.


What are your thoughts about making dynamic anchors URIs? My main concern is that people will find it confusing even tho there's precedence in the web space (link relations).

I think it will be confusing simply because you write the same dynamic anchor in multiple places, and that's not how URIs are normally used. It's fine with link relations, because you aren't targeting them in those different locations. They serve as references to or identifiers for the semantics of the link relation type. The target that they identify is the specification, and there's only one of those for each link relation URI.

Making $dynamicAnchor a URI makes that URI both a reference (to the semantics of the dynamic anchor) and a target (because you would then use it as the value of $dynamicRef) at the same time. And that's confusing. It's like making the href from <a href="https://example.com/whatever"> a target. The only way you can make an <a> element a target is <a href="https://example.com/whatever" id="thisIdIsAFragment"> But in that case, it's not the href that's the target, it's the id.

I think it's better to keep $dynamicAnchor and $dynamicRef separate from URIs/IRIs entirely. If the concern is uniqueness in an open ecosystem, there are other ways to handle that such as various sorts of namespacing. There's the Java approach of using reversed domain names, for example.

With link relation types, there is a globally shared notion of semantics. That's not really the same for $dynamicAnchor. While we name them after what they do for convenience, the names are just used mechanically. A schema with all of its human-readable $dynamicAnchor values consistently replaced with gibberish hex strings would work just as well. That is not true of link relation types, hence their need for a global semantic registry identified through URIs.

jdesrosiers

jdesrosiers commented on Jun 1, 2022

@jdesrosiers
MemberAuthor
  1. The target of these particular $refs is the resource root, so there's no need for a fragment at all

Yep

  1. If we did need a fragment, it would need to be declared with $anchor and not $dynamicAnchor because $dynamicAnchor no longer creates fragments (which I think is a good thing).

I think we could go either way with this. We could still allow dynamic anchors to be referenced by $ref, but this change would make that unnecessary and it would be cleaner to keep the two concepts separate; you reference anchors with $ref and dynamic anchors with $dynamicRef.

What are your thoughts about making dynamic anchors URIs? My main concern is that people will find it confusing even tho there's precedence in the web space (link relations).

I think it will be confusing ...

Yeah, you make some good points. I think it's not likely to be needed in enough cases to be worth the added complexity and there are plenty of ways to namespace a dynamic anchor that are compatible with plain-name fragments for the cases where it is needed.

handrews

handrews commented on Aug 22, 2022

@handrews
Contributor

I'm in favor of making this change for the next release. We do need to figure out our compatibility story here, as the semantics of $dynamicAnchor change, and both the syntax and semantics of $dynamicRef, even though the core use case remains the same. We could:

  • Just change it — we're not guaranteeing compatibility yet anyway
  • make $dynamicAnchor2 and $dynamicRef2 or some other new pair of names

Just changing it would be more appealing if we can definitely automatically migrate existing uses to a combination of the new syntax and $ref (as some use cases require both a lexical and dynamic reference working together).

Of course, all of this is relevant to / in the context of #1242 .

jdesrosiers

jdesrosiers commented on Aug 22, 2022

@jdesrosiers
MemberAuthor

I think we can just change it. We aren't committed to no backwards compatible changes yet. This is definitely something we want to get in before we consider this feature stable.

7 remaining items

handrews

handrews commented on Sep 23, 2022

@handrews
Contributor

@jdesrosiers thanks for working through my last comment despite the difficulty. I'm sorry it was not more clear or helpful.

Here's a technical walk-through similar to what you presented.

I believe what you're saying is identical to what I laid out in the "What I now think you are proposing", except for implementation details that are unimportant if not irrelevant.

Changing $dynamicAnchor from schema resource scoped to schema object scoped means it doesn't behave like an $anchor anymore, which I think makes the concept more confusing and harder to reason about.

That's a very good point. Arguably, since $dynamicAnchor no longer creates URI fragments with this proposal, it has already diverged in a significant way, in which case perhaps we should consider another set of keyword names. But we can consider whether a name change is necessary or appropriate later.

I especially have no idea what you mean by mixing lexical and dynamic behavior. Your alternate approach just seems to complicate the concept. I can't see the benefit.

If there is a benefit (and TBH there may not be), it's that it makes the inherent complexity of the behavior explicit rather than implicit. I'm going to explain this in the hopes that it fosters understanding of our divergent mental models, rather than as an argument that we need to adopt the "$dynamicRef": ["x", "#thingy"] approach.

To me, there are three aspects of $dynamic* in 2020-12, and it is the combination of them that makes it difficult to understand.

  1. Using URIs, which produces a confusing overlap in functionality and behavior with $ref and $anchor. Your proposal eliminates this aspect.
  2. The actual dynamic behavior of searching based on dynamic scopes, which is relatively (2019-09+) new for JSON Schema to have at all.
  3. The fact that dynamic anchors are resource-scoped (a lexical behavior), meaning that even though $dynamicRef is specified in terms of searching dynamic scopes, you have to consider the dynamic anchors for the entire resource at each dynamic scope rather than just the schema object through which evaluation passed

I personally find the combination of points 2 and 3 to be confusing and unintuitive. I originally thought that you were removing aspect 3, which dramatically simplified the behavior by making $dynamicRef purely dynamic, and relegating all lexical/static behavior to $ref. But that, as noted in prior comments, doesn't really work (and wasn't what you proposed anyway).

The "$dynamicRef": ["x", "#thingy"] syntax separates point 2 (which is handled entirely by the "x" value, which can only resolve to a "$dynamicAnchor": "x" through which evaluation directly passed) from point 3 (which is handled entirely by "#thingy", which is resolved against the base URI of the schema object where the "$dynamicAnchor": "x" was found).

Because what I would like is something like the following (which is not at all possible as written, and which I am not proposing. It is kind of like JSON Schema pseudo-code... psuedo-schema?). I'm going to use $setJump and $longJump for the dynamic side of things because the point here is that, as with C's setjmp, you can only jump to code that has actually been executed once already. This is just to make the difference from both the current $dynamic* and your proposal clear.

{
  "$id": "https://test.json-schema.org/dynamic-ref-leaving-dynamic-scope/main",
  "if": {
    "$id": "first_scope",
    "$setJump": {
        "jumpAnchor": "x",
        "onJump": {"$ref": "#thingy"}
    },
    "$defs": {
      "thingy": {
        "$anchor": "thingy",
        "type": "number"
      }
    }
  },
  "then": {
    "$id": "second_scope",
    "$setJump": {
        "jumpAnchor": "x",
        "onJump": {"$ref": "#thingy"}
    },
    "$ref": "start",
    "$defs": {
      "thingy": {
        "$anchor": "thingy",
        "type": "null"
      }
    }
  },
  "$defs": {
    "start": {
      "$id": "start",
      "$longJump": "x"
    }
  }
}

Here it should be clear that "$longJump": "x" looks through the schema objects (not resources) in the dynamic scope for a "$setJump" with a "jumpAnchor" of "x" (the dynamic behavior). Rather than resolving to the schema object containing the "$setJump", it resolves to the schema under "onJump", which in this case does a normal "$ref" to "#thingy".

Again, I'm not suggesting this - it's even more verbose and requires duplicating the "onJump" schema in every "$setJump".

But it does show how I conceptualize $dynamicRef resolution as a two-step process, the first of which is dynamic, and the second of which is lexical. "$dynamicRef": ["x", "#thingy"] was an attempt to explicitly specify the two steps in a more concise manner.

However, if I'm the only one who finds that implicit behavior combination difficult, then there is no benefit to making it explicit.

added this to the stable-release milestone on Jun 18, 2024
gregsdennis

gregsdennis commented on Jan 31, 2025

@gregsdennis
Member

@jdesrosiers you mentioned in the PR ☝ that you think this section still needs work. I'm happy to discuss here, but I think I'd prefer a clean start. Would you create a new issue describing what you think needs adjustment and then close this, please? (I'm leaving this open for now so that I don't lose track of it.)

jdesrosiers

jdesrosiers commented on Feb 1, 2025

@jdesrosiers
MemberAuthor

It's just some additional clean up that I'll be creating a PR for soon. Trying to discuss what I think needs improvement wasn't working in the PR and there's no reason creating an issue and discussing it there would work any better. I think it's better to just show you what I think could be better.

(I don't know if that means you'd rather keep this open or not, so I'll let you close it when you're ready.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Status

Done

Relationships

None yet

    Participants

    @jdesrosiers@handrews@gregsdennis

    Issue actions

      Proposal: Remove initial resolution step for dynamic references · Issue #1140 · json-schema-org/json-schema-spec