Refactor #Lineage to support reverse lens-mappings between schema within a sequence

[sdboyer]

Currently, lenses exist solely at the #Sequence level, and only for sequences after the first:

thema/lineage.cue

Lines 69 to 80 in b7b4330

	// seqs is the list of sequences of schema that comprise the overall
	// lineage, along with the lenses that allow translation back and forth
	// across sequences.
	seqs: [
	{
	schemas: #Sequence
	},
	...{
	schemas: #Sequence
	lens: #Lens
	}
	]

This is a relic of older thinking, where i was trying to convince myself that it's sufficient to guarantee translatability to the latest schema. It's not - if the same guarantee doesn't hold in the reverse direction, then a familiar evolutionary coupling results for communicating systems: it's not safe to start sending a new schema version until all receivers are known to have updated to accept it. It's arguably not as bad as the primary case, where receivers can't update until senders are known to support it, as receivers are often the hubs - but it's still an unacceptable degree of coupling.

Backward compatibility does not imply forwards compatibility/is a non-invertible relation, so introducing a requirement for reverse translatability means we have to change how lineages and sequences themselves are declared in order to introduce a place for the in-sequence reverse lenses.

Fixing this requires changing something fundamental about how lineages are declared. The simplest answer would be introducing an additional reverse-lens list onto #Sequence that's len(schemas)-1, though i haven't actually tried that to see what other weird warts that produces.

Even if that does work, i'm not sure i'm happy with the ergonomics - lineages are already a deeply intermeshed datastructure, and it's hard enough to keep track of everything going on in them. It's possible that we should use this additional requirement to trigger revisiting the way lineages are structured (in a way that preserves key invariants, obviously). I need to put up a discussion around that.

---

Although...

At least...i think we have to do this, but i'm not actually sure. I have this nagging voice in the back of my mind, saying that the set of changes allowable under backwards compatibility/subsumption rules actually do imply a lens morphism that can be inferred, at least in almost all cases:

• If a new field was added, drop the field in the translated instance
  • This is the approach irrespective of whether the added field is optional or required+default
• If a field contains a disjunction with a concrete default (e.g. field: *"foo" | "bar:), and new branches were added to the disjunction (field: *"foo" | "bar" | "baz"), backwards compatibility rules already dictate that the default (*"foo") must exist in prior versions. If an instance containing a value from a newly-added branch ("baz") is reverse-translated, fall back to the default ("foo")
  • If a field contains a disjunction with a non-concrete default or no default at all, and the newer instance falls outside the acceptable value space for the older schema, we cannot reach a resolution.

If the above is correct, it's not complete enough to obviate the need for a changed structure - though it could mean that automated logic is applied in the absence of a declaration. I'm loathe to get too far into lens generation right away, as it's probably a big rabbit hole, but i'd be remiss to not at least mention the possibility.

[sdboyer]

We've landed pretty firmly against any kind of lens generation. Instead, the appropriate path is providing helper pseudofuncs (akin to things in hof) that make it easy to handle e.g. the "drop a field" logic.