Lexicon vs JSON Schema

[snarfed]

If I remember right, Lexicon started out very close to JSON Schema, maybe even close enough that they were compatible? But it's since diverged more, to handle eg refs, records, tokens, etc.

What's your take on where they are right now? Does current Lexicon feel like modified JSON Schema? Or more like its own thing, with only passing similarities? For someone implementing Lexicon schema validation in a language with mature JSON Schema libraries, would your gut instinct be to customize them to support Lexicon? Or implement Lexicon from scratch, and only maybe borrow bits of the libs where they're useful?

[devinivy]

One of the more awkward parts of using JSON schema (well over a year ago) in atproto was that it was difficult to express open unions of objects and strings. Being able to express this is pretty common/important to us, since we want folks to be able to spec apps that can maintain forward- and backward-compatibility as much as possible, in such a low-coordination environment. Being able to additionally discriminate those open unions in an explicit, namespaced way goes a step further in the same direction. In a way we're sleeping on the value of these areas of lexicon while things are limited to schemas for a single app.

[bnewbold]

I haven't gone through JSON Schema in depth myself, i'd be kind of curious to experiment with anyOf and {"type": "object"} (etc) for open unions.

It is conceivable to me that there might be a path for "compiling Lexicon to JSON Schema" for some use-cases.

Another piece of context is that we are interested in experimenting with alternative syntax for declaring Lexicons. AKA, there is a JSON syntax/format today, but there might be a more terse and understandable syntax (think more like TypeScript declarations and less like YAML).

[devinivy]

The difficulty in doing it that way, if I recall, is that the validation in such an open union effectively becomes {"type": "object"}. You often want the validation to be based on $type when the $type is understood, and only otherwise fall down to the generic {"type": "object"}. There is a way to express such an open union with some duplication of the types, not, and oneOf, but it's just gnarly doing this all over the place and is a bit leaky with lex semantics. Here's what it looked like once tidied-up:

atproto/schemas/todo.social/getFeed.json

Lines 32 to 38 in c70911a

	"embed": {
	"oneOf": [
	{"$ref": "#/$defs/recordEmbed"},
	{"$ref": "#/$defs/externalEmbed"},
	{"$ref": "#/$defs/unknownEmbed"}
	]
	},

atproto/schemas/todo.social/getFeed.json

Lines 64 to 93 in c70911a

	"recordEmbed": {
	"type": "object",
	"required": ["type", "author", "record"],
	"properties": {
	"type": {"const": "record"},
	"author": {"$ref": "#/$defs/user"},
	"record": {"type": "object"}
	}
	},
	"externalEmbed": {
	"type": "object",
	"required": ["type", "uri", "title", "description", "imageUri"],
	"properties": {
	"type": {"const": "external"},
	"uri": {"type": "string"},
	"title": {"type": "string"},
	"description": {"type": "string"},
	"imageUri": {"type": "string"}
	}
	},
	"unknownEmbed": {
	"type": "object",
	"required": ["type"],
	"properties": {
	"type": {
	"type": "string",
	"not": {"enum": ["record", "external"]}
	}
	}
	}

The good news is that this shows it is possible to compile lexicons into json schemas in a generic way despite our open unions, which I agree may be a useful capability!