Versioned Build/Release Process

[ptrus]

We need to come up with a process for oasis-core that will properly tag, build, and release software with some kind of versioning scheme (semantic, date based, etc).

Details

• Any versioning scheme should work as long as it's documented
• We would want to keep a changelog that we can point to so that operators know what kind of changes happen between releases
• All published artifacts should be have proper versions associated

Acceptance Criteria

• We create a design doc for handling the versioned tag, build, release process for our repos
• Create any issues to handle any automation for the build/release process

[kostko]

The process should also take care of bumping various protocol versions. This means that we shouldn't bump the versions in each PR that changes something, but only when a release is made.

So the process needs a way to keep track of all the breaking changes.

[matevz]

Regarding the changelog, maybe we should have a special changelog: section for end user at the end of git commit message which goes into changelog. So, something like:

commit 3e8e8cdad4afc113683bc04f3772311a6de593f9 (origin/matevz/feature/oasis-node-version, matevz/feature/oasis-node-version)
Author: Matevž Jekovec <matevz.jekovec@oasislabs.com>
Date:   Tue Nov 26 11:09:18 2019 +0100

    oasis-node/cmd: append git HEAD commit to oasis-node version
    
    Appends git commit to Software version for easier debugging for
    node operators.
    
    changelog:
    When running `oasis-node --version`, the exact git commit is now
    appended at the end of the `Software version:
    ```bash
    $ oasis-node/oasis-node --version
    Software version: 0.2.0-alpha+abcdef
    Runtime protocol version: 0.9.0
    Consensus protocol version: 0.20.0
    Committee protocol version: 0.6.0
    Tendermint core version: 0.32.7
    ABCI library version: 0.16.1
    ```

Would produce an entry in the changelog document

• oasis-node/cmd: append git HEAD commit to oasis-node version

  When running oasis-node --version, the exact git commit is now
  appended at the end of the Software version:

  $ oasis-node/oasis-node --version
  Software version: 0.2.0-alpha+abcdef
  Runtime protocol version: 0.9.0
  Consensus protocol version: 0.20.0
  Committee protocol version: 0.6.0
  Tendermint core version: 0.32.7
  ABCI library version: 0.16.1

We could even shave off the oasis-node/cmd: from the title, capitalize it and add an icon of the author ;)

[tjanez]

Regarding the changelog, maybe we should have a special changelog: section for end user at the end of git commit message which goes into changelog.

Huh, I would be reluctant to adapt such an ad-hoc approach:

• Markdown in git's commit messages would not be rendered by any standard git tooling (including GitHub).
• It would be impossible to correct/improve such changelog entries before preparing a release.

Also, can you show an example of a prominent open-source project that uses that?

[matevz]

Also, can you show an example of a prominent open-source project that uses that?

Not really... release notes are usually done manually with a short git log just dumped at the end. But it just seems an overhead for us writing all those every week or so. Let alone error prone by missing something important out.

[kostko]

We could require each PR to also update the "pending changelog" which is then used when preparing the final one?

[matevz]

We could require each PR to also update the "pending changelog" which is then used when preparing the final one?

In this case I would vote for changelog.md file with sections for each release in oasis-core repo. And the guy making a release would just wrap that for the release notes.

[tjanez]

release notes are usually done manually with a short git log just dumped at the end.

That is a really poor approach and such a changelog has little value. I'm more in favor of a curated changelog. See https://keepachangelog.com/ for more details.

We could require each PR to also update the "pending changelog" which is then used when preparing the final one?

Yes, that is one possibility and quite an established one. One issue I have with it is that there are a lot of rebase/merge conflicts since people generally add things to the same part of the changelog file 🙂.

The other possibility which is emerging recently is to have changelog fragments in separate files which are then automatically assembled into the changelog when a release is prepared. This approach resolves the constant rebase/merge conflicts.

[kostko]

is to have changelog fragments in separate files

Any examples to link to?

[matevz]

release notes are usually done manually with a short git log just dumped at the end.

That is a really poor approach and such a changelog has little value. I'm more in favor of a curated changelog. See https://keepachangelog.com/ for more details.

Is there a place where keepachangelog.com defines where to put stuff like upgrade instructions for our node operators like we did it here:
https://docs.oasis.dev/operators/current-testnet-parameters.html#deployment-change-log

Maybe I'm mixing two things - release notes and a changelog. Upgrade instructions and special stuff the node operators (users) should be careful about are probably in release notes. Changelog is shorter and more "technical" I guess. Libreoffice's release notes: https://wiki.documentfoundation.org/ReleaseNotes/6.3

[tjanez]

is to have changelog fragments in separate files
Any examples to link to?

I'm most familiar with such tools in the Python ecosystem.

One quite popular is towncrier, which is used by (among others):

• pip: https://github.com/pypa/pip/tree/db1a2754bb2aa4f92db781933661c170100f154f/news
• Pipenv: https://github.com/pypa/pipenv/tree/5b6bfc96b5e50011a796ea6c3cc476b7909f38b1/news
• Buildbot: https://github.com/buildbot/buildbot/tree/d2aa5a659305562bc9742afdd03c291fc83e468e/master/buildbot/newsfragments
• Trio: https://github.com/python-trio/trio/tree/master/newsfragments
• Telepresence: https://github.com/telepresenceio/telepresence/tree/bb2eb584cc25deba7e5920938d576693c847bed0/newsfragments

Although it is geared towards Python projects and works with rST by default, it can be made to work with Markdown and for non-Python projects.

Another one is OpenStack's reno which is more complex and uses rST wrapped in YAML, so I think less interesting for us.

[kostko]

Nice, so I assume this news directory gets cleaned on each release?

[tjanez]

Nice, so I assume this news directory gets cleaned on each release?

Yes, correct.

[kostko]

This looks like the best solution I've seen so far, so I would be in favor of using it.

[tjanez]

Maybe I'm mixing two things - release notes and a changelog. Upgrade instructions and special stuff the node operators (users) should be careful about are probably in release notes. Changelog is shorter and more "technical" I guess.

Yes, release notes and a changelog are two different things.

From my experience, release notes typically highlight important changes in a release and talk about upgrading from previous releases.

[matevz]

In addition we could have .release_notes files in news/ folder which would contain upgrade information for node operators. So the release notes document would wrap the changelog document + prepend upgrade instructions.

UPDATE: Looking at pip "relase notes" here https://pip.pypa.io/en/stable/news/ I think we should just combine both documents into one (and call it whatever) and have instructions for node operators inside the ordinary .bugfix or .feature files and rendered as bullets in the final document.

[tjanez]

Filled/updated the following issues:

• Document Versioning scheme used for Oasis Core and Protocols #2457
• Use a versioned git tag for each release of Oasis Core #2458
• Automate build and release process for oasis-node #2456
• Automatically generate the changelog #759 (comment)
• Set up automatic packaging of Rust libraries and publishing to crates.io #2460
• Sign Oasis Core release artifacts #2461