Release notes

[bgilbert]

In Container Linux we never really got release notes right:

• The notes for a release are manually constructed, which can be time-consuming.
• To correctly determine the complete set of changes after a cross-channel promotion, users must understand the CL branching model, which is not obvious or documented (Container Linux release notes are not usable bugs#2134).
• Release notes are available on a web page, and in a JSON doc which requires custom tooling to read (i.e., is not RSS or Atom). There's no good way to receive notifications when new release notes are available (except for subscribing to the repo where the release notes are pushed).
• Documentation does not explicitly recommend reading release notes, but a prudent user will do so anyway.
• Release notes are generally considered immutable, and are not changed even if incorrect or incomplete.
• The docs site takes half an hour to build, imposing a 30- to 90-minute delay between release of artifacts and publication of release notes.

We should design the release notes for Fedora CoreOS.

• Should they be automatically generated? From what inputs?
• Should we provide tools for showing all changes between two versions?
• Should notes be revised if later determined to be incorrect? (e.g., a fix was ineffective)
• Should notes be revised if a release is later determined to introduce an important bug?
• Where should release notes be published? Presumably a web page. RSS/Atom feed? Mailing list? Release and/or stream metadata?
• Where should the authoritative copy of release notes be stored, and how should it be propagated to other copies?

[jlebon]

Just a random note: rpm-ostree has some nice tooling, albeit targeted for the client side, surrounding the question "what changed?".

1. it can download and display security errata for packages and extract CVE information from their synopses
2. rpm-ostree db diff --changelogs gives you a full summary of what changed in packages between two versions

Definitely open to rework those to make them more accessible if they'd be useful in whatever workflow we establish.

[bgilbert]

Discussed at last week's meeting.

FAH release notes consist of an automated email and a followup. The latter can have handwritten content. These messages are sent only for stable releases, and aren't archived anywhere other than the mailing lists.

Our release notes could have several sections:

• A manually-curated set of important information: major changes, notable security or bug fixes
• Perhaps a list of package versions for important packages, as CL does
• Automatically generated list of CVEs
• Something like rpm-ostree db diff --changelogs, which lists the relevant parts of package changelogs. We could filter the package set, or float important packages to the top. For new upstream releases of relevant packages, we could automatically link to the upstream release notes.

We should have a permalink for each set of release notes. We might want to use several publication methods, as different users may have different preferences: mailing list, RSS, Twitter, etc.

[dhawal55]

Any update on this? What are the plan for release notes? Can we at least start with something simple and build upon it?

[jlebon]

Re. authoritative source, one simple idea to get started is just a release.md in https://github.com/coreos/fedora-coreos-config production branches. I.e. this naturally ties the release notes to the content promotion process and we can have it be generated at that time, and reviewed, and merged atomically with the content, etc... (Guess it could be as the commit message too, though GitHub doesn't make reviewing commit messages easy, and it's less flexible in case we want to fix up a typo or something).

[jlebon]

Where should release notes be published? Presumably a web page.

Not sure if we want to maintain a separate site from https://getfedora.org/en/coreos, so my vote is to keep it there. This argues for putting it in the stream metadata of course, but we're not necessarily tied to that either and could have the notes fetched from somewhere else.

[mariusgrigoriu]

Until we figure out how changelogs are published, what is the recommended way to get a sense of what's new with any given release prior to taking an update?

[jlebon]

Until we figure out how changelogs are published, what is the recommended way to get a sense of what's new with any given release prior to taking an update?

@zonggen is working on a redesign of the download page that will show the package list and package diff at least until we enhance it with nicer release notes.

For now, you can do something like:

# ostree pull fedora:fedora/x86_64/coreos/stable
# rpm-ostree db diff fedora:fedora/x86_64/coreos/stable --changelogs

[fifofonix]

The newly re-worked release pages with the summaries of changed packages are a definite plus but I think the 'next' stream should be calling out the docker version change to 19.03.8 from 18.09.8 as a top-level change right now. Just like there are call outs for versions of ignition/podman etc.

I'm not sure how this all comes together but I did do an rpm-ostree db list on next and grepped for docker to no avail to confirm why the docker change isn't even actually listed in the full listing. I guess it is layered into the build in a way that I'm not appreciating? Perhaps these additional pieces need to be scanned somehow for versions when building auto release highlight pages to give them visibility.

[bgilbert]

@fifofonix The package you're looking for is moby-engine.

[fifofonix]

Got it. And yes of course that is in the full list of packages. But I still think it deserves a more obvious call out.

[dustymabe]

@zonggen - can we add moby-engine to the short list of packages that get highlighted for each release of FCOS on the release notes page?

[zonggen]

@dustymabe Sure. For reference, we have a list of important packages to show in the build browser and the download page:

• https://github.com/coreos/fedora-coreos-browser/blob/master/index.html#L194
• https://pagure.io/fedora-web/websites/blob/master/f/sites/static/js/coreos-release-notes.js#_21

Filed https://pagure.io/fedora-web/websites/pull-request/118

EDIT:
Also as a followup on the moby-engine, filed coreos/fedora-coreos-browser#14 to add moby-engine into the build browser important package list.

[dustymabe]

We discussed this in the meeting today.

In general I think we agreed that we could use a file (probably yaml) in the tip of each production stream (stable, testing, next) to collect release note information about each release in that stream. How we plumb that through to get to the release notes on the website is still up for discussion.

Another idea was that we come up with a way to add release notes as we merge in PRs. So something like a release-notes.d/ directory in the testing-devel branch where each new PR could add content there (unique file names avoid merge conflicts) and then it gets collated together when we do a release.

[zonggen]

Does the following design sound feasible:

• Include a changelog.yml file inside each of the build directory (or changelog.d/ that contains the yaml files)
• merge the yaml files from last release build upto the next release builds into a single release changelog file, and also de-dup the changelog
• release page use the final changelog yaml file to display desired information

Though this sounds like a naive solution..

[dustymabe]

Does the following design sound feasible:

• Include a changelog.yml file inside each of the build directory (or changelog.d/ that contains the yaml files)

What do you mean by "build directory" here? Are you referring to having a file or directory in each branch of the fcos configs repo?

• merge the yaml files from last release build upto the next release builds into a single release changelog file, and also de-dup the changelog

Yep. I'm thinking that when we take the contents from the /.d directory on the testing-devel stream and merge it together and put it under a nextrelease entry in the yaml.

Once we do the release we update the entry in the yaml to be the version number from the build we just put out.

Example toplevel yaml:

releases:
  - nextrelease:
    - removed package xyz
        - This package was causing problems. Workaround by foo foo foo.
    - added package abc
    - fixed [bug3](https://example.com/bug3)
  - 32.20200629.3.0:
    - removed package xyz
    - added package abc
    - fixed [bug4](https://example.com/bug3)

nextrelease would then get replaced when we put out the release. There are a lot of mechanics here and a lot of ways to do this. I think the important part is that we get something in place and then start ot iterate on it.

• release page use the final changelog yaml file to display desired information

👍

Another thing to think about is the format of the yaml file. Would we accept markdown and then try to render it as html?

[zonggen]

What do you mean by "build directory" here? Are you referring to having a file or directory in each branch of the fcos configs repo?

Yes, I was thinking about adding /.d or a .yaml in each branch of fcos config repo.

the important part is that we get something in place and then start ot iterate on it.

👍 I will start experimenting from the basic approach and see how it looks.

Would we accept markdown and then try to render it as html?

Parsing markdown into html and inject into Vue.js code is feasible however may require additional dependencies, but I do think markdown is much more readable so will experiment with the dependencies. Yaml and json parser is built into javascript so reading them should be straightforward.

[zonggen]

Further discussion with @dustymabe leads to the following design.

The reason for separating out the release notes creation from FCOS pipeline is that when we want to make adjustment to the old release notes we've already published, we can manually update the release-notes.yaml entry for that release and update the website, independent of release cycles.

Developement situation:

1. Developer A opens PR to testing-devel repo to fedora-coreos-config repo
   • adds release note to release-notes.d/fixed-ignition-bug.yaml
     • fixed bug 77 in ignition all nodes now use DHCP
   • PR gets merged
2. Developer B opens PR to testing-devel repo to fedora-coreos-config repo
   • adds release note to release-notes.d/fixed-afterburn-bug.yaml
     • fixed bug 87[https://link/to/bug/87] in afterburn all nodes now use ecdsa SSH keys
   • PR gets merged
3. We want to do a testing release
   • merge testing-devel into testing
   • run a build through the build pipeline
     • this is the step where we learn what the version number is (i.e. 32.20200629.3.0)
   • we do some final checks (mostly just little inspections)
   • then we run the release pipeline
     • replicates to aws regions
     • copies ostree content to prod repo
   • then we run fedora-coreos-stream-generator
     • metadata processing
   • copy metadata to S3
     • Update server (cincinatti) picks up new metadata from s3, serves updates
     • Website picks up new metadata from s3, serves updates

Note: In the case of overgrowing release-notes.yaml, do mv releasenotes.yaml 2020-releasenotes.yaml annually (or periodically)

Proposal:

Have a tool that runs independently of the pipeline like bump-lockfile and sync-stream-metadata jenkins jobs. This tool will trigger based on a new release of FCOS or can be triggered manually. The responsibility of this tool includes:

Git repository manipulation

• look at the release-notes.yaml file for the related branch (i.e. testing)
  • if an entry for that release exists, it does nothing to the git repo
  • if an entry for that release does not exist
    • create a new entry in release-notes.yaml for that release version
      • merge the release-notes.d/* files into release-notes.yaml
      • delete release-notes.d/* files from that branch
      • delete corresponding release-notes.d/* files from testing-devel branch

Example release-notes.yaml after the merge:

    releases:
      - 32.20200629.3.0:
        - inline: |
            - fixed bug [77](https://link/to/bug/77) in ignition all nodes now use DHCP
            - fixed bug [87](https://link/to/bug/87) in afterburn all nodes now use ecdsa SSH keys
      - 32.20200619.3.0:
        - DISCLAIMER: There was a serious flaw affecting this release that causes AWS nodes to reboot after 4 hours of uptime.
        - old release note 1
        - old release note 2

Intermediate processing to get into ideal form for website code consumption

• process release-notes.yaml (maybe convert to JSON or other formats)

Upload to place where website can consume

• upload to s3
• release-notes.yaml picked up by FCOS release-notes.js for rendering

[jlebon]

Re. CVEs, there's a draft PR in coreos/rpm-ostree#2695 which will make this much easier for whatever release notes tooling to query.

[travier]

In order to make progress on this issue, I'm proposing that we start by solving a smaller subset of it: linking solved issues from the tracker to releases on the release page.

To give an idea about how this would look like for the last couple of releases, you can look at: https://travier.github.io/fedora-coreos-tracker/.

The general idea would be to create three JSON files in this repo, one for each stream, and list the issues referenced for each releases:

$ cat testing.json
[
  {
    "release": "XXXX.Y.Z",
    "issues": [
      "X", <-- issue numbers only or title + issue number
      "Y",
      "Z",
    ]
  },
  ...
]

The release page could then lazily load those JSON documents and display the issue numbers and titles, fetched from GitHub, or directly included in the JSON files. Having the full titles directly in the JSON means that we do less network calls on the release page but it makes updating them less easy.

This is partially inspired by https://www.flatcar-linux.org/releases/.

[travier]

Examples:

• https://github.com/travier/fedora-coreos-tracker/blob/main/docs/stable.json
• https://github.com/travier/fedora-coreos-tracker/blob/main/docs/testing.json
• https://github.com/travier/fedora-coreos-tracker/blob/main/docs/next.json

[dustymabe]

We discussed this in the community meeting yesterday.

There was no opposition to the idea considering anything is an improvement over what we currently have. There was some interest in the finer details of how to achieve the goal with the least maintenance burden. @travier can you lead the conversation about those details going forward?

[travier]

The general idea is to dynamically load the JSON with the list of issues on the release page: https://getfedora.org/en/coreos?stream=stable
The main questions are:

• Where do we store those JSON files? Storing them in this repo might keep things easier as we can quickly link them into a PR.
• Who's in charge of updating them? I can keep doing it as before or share with @dustymabe as he does the #label cleanup for each release. Or maybe we add that to the release work at the end. It's fairly easy and won't be too long for those doing the releases.

[travier]

PRs for this work:

• Done:

  • release-notes: Initial import for all streams fedora-coreos-streams#477
  • release-notes: Update for new releases fedora-coreos-streams#507
  • jobs/sync-stream-metadata: also sync release notes fedora-coreos-pipeline#501
  • https://pagure.io/fedora-web/websites/pull-request/246

• Dropped:

  • templates: Add release notes instructions fedora-coreos-streams#493

[dustymabe]

Website PR merged:

Nice work @travier @mohelt !

[travier]

Thanks a lot @mohelt for the good work!

[travier]

I'm closing this one given that we now have release notes available on the website!

I've created #1227 to track future enhancements.