MSC4107: Feature-focused versioning #4107
Conversation
turt2live
changed the title
turt2live
added
proposal
|
|
||
| **TODO**: There's probably other issues. They should be documented. | ||
|
|
||
| ## Alternatives |
There was a problem hiding this comment.
Raised by @ara4n : Another option is to promote the use of unstable_features more, saying that if implementations haven't yet filled the delta of $current_version and $target_version, they continue to use (un)stable prefixed feature flags that clients should expect.
I haven't really thought about consequences here, but we may need to adjust the process to encourage servers to drop unstable flags after 2 months from spec release, but inform clients of a different timeline (>2 months, potentially significantly so).
| then a new feature ID would be recommended. The SCT is responsible for determining when/if a new feature | ||
| ID should be used. | ||
|
|
||
| ## Potential issues |
There was a problem hiding this comment.
A possible concern is when features overlap and depend on each other. For example, threads requiring sync changes. Representing this in unstable_features isn't impossible (see https://github.com/matrix-org/matrix-spec-proposals/pull/4107/files#r1495129299 ), though I think this MSC makes it clearer/different.
Threads being assigned msc3440 as an example, and /sync being considered core, a client would look for a compatible version of each. If something like MSC4103 comes along, the server would be required to implement msc3440: ["org.matrix.msc4103"], core: ["org.matrix.msc4103"] before clients will use it in earnest.
| then a new feature ID would be recommended. The SCT is responsible for determining when/if a new feature | ||
| ID should be used. | ||
|
|
||
| ## Potential issues |
There was a problem hiding this comment.
Mentioned by @tulir : A potential benefit or issue, depending on how you look at it, would be that a server could support v1.3 of everything except appservice, where it deliberately downgrades to v1.1 due to default config or something. This would match Synapse's current behaviour.
There was a problem hiding this comment.
To clarify, Synapse defaults to only sending the Authorization header in appservice transactions, which means it's not compatible with v1.1, v1.2 and v1.3 of the appservice spec (which required the access_token query param). Synapse still advertises support for all spec versions even though it's not compatible with the older AS spec versions with the default configuration. With this MSC, it could accurately advertise what it supports (e.g. "appservice": ["v1.4", "v1.5", ...] and "core": ["v1.1", ...])
There was a problem hiding this comment.
Synapse still advertises support for all spec versions even though it's not compatible with the older AS spec versions with the default configuration.
Synapse still advertises support for all client spec versions. It isn't really clear if /_matrix/client/versions applies to other aspects of the spec or not (e.g. appservices, federation, etc.).
There was a problem hiding this comment.
The appservice API extends the client-server API, and so it should be included in /versions (appservices are clients by definition). Federation would be out of scope here.
| and specification releases. The feature ID is typically the MSC number, though for historical modules | ||
| in the Client-Server API some are named features. If a given feature spans multiple MSCs, it is expected | ||
| that an MSC to allocate a feature ID is created. |
There was a problem hiding this comment.
Using MSC numbers feels really awkward from a client developer point of view. You really need to understand the whole way the spec changes over time, the MSC process, etc. when (as a client developer) you might wish to just check "are threads supported?". It feels like process leaking into the protocol.
There was a problem hiding this comment.
Yea, I'm not sure what the best balance is here. In the draft that wasn't published the feature ID was actually named "threading" and such, but then we either require an MSC to assign the feature ID or an unstable prefix, at which point we've fragmented the purpose of this system back into unstable/stable/different-stable.
There was a problem hiding this comment.
My initial thought was that when a feature passed FCP you would get a stable name, but as you said that kind of has the same issue where you have to check multiple spots. 😞
| This approach could fragment implementation slightly, allowing servers to selectively implement a given | ||
| MSC up to the "latest" spec release, but exactly zero other features receive the same treatment. This | ||
| situation is how XMPP largely works: implementations are a mix of core functionality and whatever | ||
| desirable XEPs are needed. We at Matrix want to avoid this. Though the MSC doesn't go into detail about | ||
| it currently, the author's theory is that a `core` feature ID could be assigned to most functions in | ||
| the spec, and clients would in practice be checking `core + msc4107` support of the server. This still | ||
| encourages the server to keep up to date on the core functionality, and push for complete features as | ||
| well. |
There was a problem hiding this comment.
I think this will cause a lot of fragmentation. Maybe that's OK, but it definitely needs to be acknowledged.
Would it make sense to consider the versions in light of the feature profiles instead? Does that help at all?
There was a problem hiding this comment.
Instead of individual "features" you could say you support "profiles of features", it is a larger grouping of things to support.
| Another possible way to ensure full compliance with the specification is The Matrix.org Foundation C.I.C. | ||
| running a "certification" program to verify a given implementation's level of spec support. |
There was a problem hiding this comment.
This seems like it would just be another hurdle, i.e. making it harder to solve the problem this MSC outlines.
There was a problem hiding this comment.
We've always kinda wanted to do this in some capacity, but yea: the alternatives are not fully considered here.
| In a related concern, server implementations *often* lag behind the spec release cycle due to the size | ||
| and complexity of the spec. The spec itself is very monolithic, and that causes friction when a server | ||
| wishes to support a given subset or limited scope, or when a server implementation is trying to get | ||
| started and immediately falls behind on client support. Unfortunately, reducing the release cadence | ||
| from quartly to biannually does not have an effect of easier releases to implement - the releases just | ||
| become chunkier (or don't happen at all because it doesn't feel "worth it" to release). |
There was a problem hiding this comment.
I would personally be very interested in understanding why this is -- are there particular features that are extremely hard to implement? Is there just a lot of historical features to catch-up on? Is this due to mostly chasing down bugs instead of implementing new features?
There was a problem hiding this comment.
From my perspective it's complexity in the spec. Clients naturally work at a cadence which is significantly faster than the server, but servers which don't have full time engineering teams behind them can't advertise feature support until they've implemented the entire specification at that version. Clients then move forward and start dropping old versions, which breaks compatibility and affects user experience of the protocol.
This MSC aims to solve the last part of that without introducing a ton of fragmentation: servers already have support for plenty of features, though not necessarily enough to advertise full compatibility with a given spec version. Clients can then use this to utilize the features the server supports without arbitrarily having to say that the whole server is incompatible because an unrelated section of the spec hasn't been addressed yet in the server.
Overall, the goal is to get features in front of users sooner, which is part of the reason why unstable implementation exists in the process. This is an evolution of that to allow those features to stay in front of users without having to rely on unstable prefixes for an eternity (as that's the current workaround).
There was a problem hiding this comment.
This MSC aims to solve the last part of that without introducing a ton of fragmentation
I fail to see how it aims to protect against fragmentation. It seems each individual feature could be decided to be supported?
There was a problem hiding this comment.
The specific mechanic isn't yet figured out. The MSC presents making core a thing as the likely mechanism, but other options could include a compliance program run by the Foundation. Personally, I prefer the core library being larger and therefore an active dependency in most features, thus requiring servers to implement a modern version of the spec before clients turn on the feature, but still being careful to avoid making the requirement constantly be $latest.
There was a problem hiding this comment.
Largely reiterating clokep's question here. It seems, from discussion elsewhere, that we aren't sure whether the desired situation is:
- Servers implement a given spec version (say, v1.1) with a few extra features on top (say, private read receipts), or:
- Servers implement 90% of a given spec version (say, v1.8), but find a few features hard to implement (say, edits), which means they can't advertise support for v1.8.
This MSC appears to assume the former, but @ara4n has assumed the latter in conversation. I am therefore worried we are solving the wrong problem, and I would like to see input from maintainers of the affected servers (Dendrite and Conduit primarily, I assume?) before discussing further.
Rendered
In line with matrix-org/matrix-spec#1700, the following disclosure applies:
I am Director of Standards Development at The Matrix.org Foundation C.I.C., Matrix Spec Core Team (SCT) member, employed by Element, and operate the t2bot.io service. This proposal is written and published as a member of the SCT.
Implementation requirements: