Representing specification versions

[wbamberg]

(This is part of #2822)

We need to have a way to represent different versions of the Matrix spec in Hugo-land. This issue is to discuss how we will do that. This whole discussion presupposes that we're moving to a global version number as proposed in #2844.

---

At any time there are three "statuses" for different versions of the spec.

• "unstable": the spec that we're currently working on, that changes are getting merged into.
• "current": the current release of the spec
• "historical": old versions of the spec that have been superseded by "current"

It's important to represent these, because we want to be able to tell readers "this is an old version of the spec, you probably want the latest one" or "this is the unstable version of the spec, you might want the latest released one".

Given that according to global versioning proposal, "Implementations do not need to worry about the patch version." it feels like it makes sense to manage these statuses at the level of major.minor versioning only - so for instance the current version is "1.1" and stays "1.1" through all the patch versions of 1.1, until 1.2 is released. We should still represent patch versions of course, but maybe just in the changelog and the version name.

So that would mean:

• "historical" versions are frozen
• "current" may be updated by patch versions
• "unstable" may be updated with any changes

Overview of this proposal

The specification lives inside a static site generator (specifically, Hugo) in a Git repo.

This proposal uses Git branches to represent different versions of the specification. Each branch (and therefore each release) is a separate deployment of the site and is really a self-contained world. The "unstable" version is built from the main branch.

Repository organization

In Hugo, by default, the /content/$language directory is used to contain the pages that will be built. In this repo our /content directory is like:

content/en/_index.md                           <- overview (like https://matrix.org/docs/spec/)
            /client-server                     <- client-server
            /server-server                     <- etc
            /rooms
                /v1                            <- room version 1
                /v2                            <- etc

In Hugo, site-wide settings live in a config.toml file. Our config.toml will contain:

• a setting saying whether this spec version is: "unstable", "current" or "historical"
• a setting which is a URL pointing to the current release, like "https://spec.matrix.org/latest"

We will use this to show the appropriate banners for unstable and historical pages, and to link to “latest”.

Our config.toml will also probably want to contain settings saying which version it is, including patch version.

Deployed

The deployed specification could look like this:

spec.matrix.org
            unstable                        <- deployed from matrix-doc HEAD
                /                           <- overview
                        /client-server      <- client-server
                         /server-server     <- etc
                         /rooms
                            /v1             <- room version 1
                            /v2             <- etc

            1.1                             <- deployed from matrix-doc 1.1 branch
                /                           <- overview
                        /client-server      <- client-server
                        /server-server      <- etc
                        /rooms
                          /v1               <- room version 1
                          /v2               <- etc

            1.0                             <- deployed from matrix-doc 1.0 branch
                /                           <- overview
                        /client-server      <- client-server
                        /server-server      <- etc
                        /rooms
                          /v1               <- room version 1
                          /v2               <- etc

spec.matrix.org is a hardcoded homepage (not in Hugo/matrix-doc), and links to spec documents using URLs like spec.matrix.org/latest/client-server. These links redirect to the actual latest version.

Under spec.matrix.org, we deploy the main branch and all release branches of matrix-doc, independently.

Changelogs

In the repo, have a /changelogs directory. In the main branch, this could contain a newsfragments directory containing new changes since the last release:

/changelogs
    /newsfragments

In release branches this could contain at least one changelog named something like “release.md”, which is the changelog for the first version of this release. It may also contain one or more additional one or more built changelogs, which are named after the patch version that contains this set of changes:

/changelogs
    /release.md
    /1.md
    /2.md

The behaviour of the SSG is then:

• If there are built changelogs, use them to construct a complete changelog for this major.minor release. It should list the release.md changes, followed by all patch versions.
• Otherwise, if there are newsfragments, use them to build a changelog for the unstable version of the spec

Making a release

To make a new major/minor release:

• decide on the new release version (say, "1.1")
• generate the release.md changelog for the new release
• make a new branch from main, giving it whatever name and tag (e.g. "1.1")
• set the config.toml for this release branch to say it's the new current release
• set the config.toml for the last release branch (say, "1.0") to say it's the historical release
• set the config.toml to contain the actual release number including patch version (0)
• on the server, update the "latest" redirect for the spec.matrix.org to point to the "1.1"
• deploy the new release and redeploy the previous release

To make a patch release:

• generate the new $patch.md changelog for this patch
• merge changes from main into the latest release branch
• increment the patch version number in config.toml.

"Releases" dropdown

It would be nice to have a dropdown on each page listing all releases of the spec. That's tricky in this model because historical releases don't know about subsequent releases. One suggestion Travis made is: we could populate the dropdown dynamically, reading a releases.json file that's maintained in the repo.

[clokep]

"Releases" dropdown

It would be nice to have a dropdown on each page listing all releases of the spec. That's tricky in this model because historical releases don't know about subsequent releases. One suggestion Travis made is: we could populate the dropdown dynamically, reading a releases.json file that's maintained in the repo.

Just to chip in that I find the ability to easily switch between releases this way essential. It also really helps with deep links to old versions of the spec as you can quickly see "hey you're on an old version!". Django even throws up a banner for old versions when they're obsolete (which might not make sense for the spec for a while, but is nice):

The rest of this proposal pretty much sounds like what ReadTheDocs does for versioning: https://docs.readthedocs.io/en/stable/versions.html Pretty much the repo only knows about a single version, but the generated static assets have different versions by putting them at different paths.

[wbamberg]

Thanks for looking @clokep !

Just to chip in that I find the ability to easily switch between releases this way essential. It also really helps with deep links to old versions of the spec as you can quickly see "hey you're on an old version!". Django even throws up a banner for old versions when they're obsolete

There are a couple of different things here:

• showing a banner linking to the latest release, if you're on an old version. This seems like the most common use case and is handled by the proposal already (you just need a link that will always point to "latest").

• showing a dropdown of all versions: that's what the "Releases dropdown" bit is referring to, and is trickier because historical releases don't know about more recent releases (e.g. 1.1 doesn't know if there will be a 1.2).

But maybe it's not necessary to show all versions, and it would be OK for the dropdown to show: (1) the unstable version (2) the latest release (3) and historical releases that existed at the time this version was made. That's what the docs do currently actually, compare 0.6.1: https://matrix.org/docs/spec/client_server/r0.6.1#other-versions-of-this-specification to 0.4.0: https://matrix.org/docs/spec/client_server/r0.4.0#other-versions-of-this-specification.

[nadonomy]

Discussing this on a call with @wbamberg & @harrisondean and slightly concerned we'd end up with a strange interstitial UX where you'd land on spec.matrix.org and have to deliberately click/link through to view the latest spec.

Exploring an alternative option by storyboarding wireframes where:

• The latest spec would live at spec.matrix.org, with links to historical versions & unstable
• Versioned spec would live in paths
• Unstable would live in a path

[clokep]

where you'd land on spec.matrix.org and have to deliberately click/link through to view the latest spec

Most docs sites I've seen do a redirect from / -> /latest or whatever to solve this issue.

[nadonomy]

where you'd land on spec.matrix.org and have to deliberately click/link through to view the latest spec

Most docs sites I've seen do a redirect from / -> /latest or whatever to solve this issue.

@clokep can you point us to some references to benchmark? Just want to double check a few implementation details.

[clokep]

where you'd land on spec.matrix.org and have to deliberately click/link through to view the latest spec

Most docs sites I've seen do a redirect from / -> /latest or whatever to solve this issue.

@clokep can you point us to some references to benchmark? Just want to double check a few implementation details.

• https://docs.djangoproject.com
• https://docs.celeryproject.org/
• https://tox.readthedocs.io (this is just a ReadTheDocs project, I think most projects are setup to redirect this way, although I found at least one counter-example: https://six.readthedocs.io/)

All of the above are built with Sphinx though so maybe it is just due to that. 😄

[wbamberg]

So we should amend the "Deployment" section above to be like:

spec.matrix.org                             <- redirect to spec.matrix.org/1.1

            unstable                        <- deployed from matrix-doc HEAD
                /                           <- overview
                        /client-server      <- client-server
                         /server-server     <- etc
                         /rooms
                            /v1             <- room version 1
                            /v2             <- etc

            1.1                             <- deployed from matrix-doc 1.1 branch
                /                           <- overview
                        /client-server      <- client-server
                        /server-server      <- etc
                        /rooms
                          /v1               <- room version 1
                          /v2               <- etc

            1.0                             <- deployed from matrix-doc 1.0 branch
                /                           <- overview
                        /client-server      <- client-server
                        /server-server      <- etc
                        /rooms
                          /v1               <- room version 1
                          /v2               <- etc

...and as part of deploying a new major.minor release we update the spec.matrix.org redirect?

[wbamberg]

I've been experimenting with ways to represent versions and changelogs. It's easier to show than tell, so:

• https://adoring-einstein-5ea514.netlify.app/ is built from the main branch, which represents the "unstable" version of the spec
• https://upbeat-swirles-af35c7.netlify.app/ is built from the 1.1 branch, which represents the "current" version of the spec, which is a hypothetical "1.1.2" version.

Content differences

The content differences between these two versions are:

config settings

• main has config settings like:
  version.status="unstable"

• 1.1 has config settings like:
  version.status="current"
major_version = 1
minor_version = 1
patch_version = 2

That is, if you have a setting other than "unstable", then we expect to find values for the major, minor, patch components of a version.

changelog directory

Both versions have a "changelog" directory.

• For main this contains a directory "newsfragments" containing news fragments.
• For 1.1 this contains a directory "releases" that contains a subdirectory for every patch version (so in this case three: "0", "1", "2"). Each of these subdirectories contains a "release.yaml" file, which lists the Git tag for this release and the release date. It also contains a directory "newsfragments" that contains newsfragments for this release.

So again: if you have a setting other than "unstable", then we expect to find a "changelog/releases" directory with the structure described.

Rendered differences

First, the navbar at the top now includes the version, taken from the config settings.

Second, unstable releases get a banner on each page pointing to the current release (athough the link doesn't actually go to https://upbeat-swirles-af35c7.netlify.app/ at the moment).

Third, we get different changelogs of course:

• unstable: https://adoring-einstein-5ea514.netlify.app/changelog/ - lists all changes found in "changelog/newsfragments".
• 1.1.2: https://upbeat-swirles-af35c7.netlify.app/changelog/ - includes a section for each directory it finds in "changelog/releases".

Version X.Y.Z sections

Each "version" section includes a table built from the release.yaml, with link to the Git tag and release date. We don't represent release date in the existing spec, but it seems useful enough to me.

About Git tags: I wasn't sure why we don't currently refer to Git tags to point people at the commit for this release. Perhaps I'm missing something?

Historical releases

Both "unstable" and "1.1.2" include a section about "Historical releases" - that is, versions that preceded the global versioning system. When we start having not-current releases in the global versioning system (i.e. when "1.2" is released, so 1.1 becomes not current) we will also have a "Previous releases" section that will point to them.

[wbamberg]

I ran this past Travis and had some feedback which I've copied below:

The overall proposal looks good, though it is a bit unfortunate that we don't have a releases dropdown - I'm happy to keep this squarely in the "someone else figure this out" bucket though.

It might also be beneficial to keep the version numbering together simply for the purposes of us maybe needing -beta, -demo, etc on the fly. Keeping version.status makes sense, though I'd add version.number="1.1.2" in place of the major_version and similar fields.

About Git tags: I wasn't sure why we don't currently refer to Git tags to point people at the commit for this release. Perhaps I'm missing something?

I think this is mostly a case of no one really bothering to promote the system better. We seem to have had intentions of using the commit version for something, but ultimately didn't do anything productive with it and started using tags. Might have been carryover from the pre-git days of Matrix.

[wbamberg]

For the "releases dropdown". To be clear: this is a bit of UI that enables a user to switch between this version of the spec and some other version.

I think it's easy to make a "releases dropdown" that includes links to:

• the current release version of the spec
• the current unstable version of the spec
• any old released versions before this one

What's trickier is making a version of the spec include links to old released versions made after this one.

For example: suppose we have shipped versions 1.1, 1.2, 1.3, 1.4 of the spec, and are looking at the 1.2 version. It's easy for this dropdown to contain:

• current (which will redirect to 1.4)
• unstable
• 1.1

...but it's not so easy for it to contain 1.3, because 1.3 didn't exist at the time 1.2 shipped.

If this is an important thing to support, one way to do it (h/t Travis) is to maintain a "releases.json" in the repo, and populate the releases dropdown dynamically using JS.

I don't think this is a very important thing to support, and suggest we do the static/easy thing for now. Note that in the current spec we omit links to obsolete versions which are newer than the current one. For example, https://matrix.org/docs/spec/client_server/r0.2.0#other-versions-of-this-specification doesn't link to any released versions after 0.2.0.

[richvdh]

Is this fixed? I'm not really sure what it's asking for.

[turt2live]

I think this is fixed for the time being. There's an action of how we want to show older versions of the spec in the platform, but we have a somewhat natural way of conveying that for the moment - if we end up needing a dropdown, we can add one with a new issue.