Release notes

[bgilbert]

In Container Linux we never really got release notes right:

• The notes for a release are manually constructed, which can be time-consuming.
• To correctly determine the complete set of changes after a cross-channel promotion, users must understand the CL branching model, which is not obvious or documented (coreos/bugs#2134).
• Release notes are available on a web page, and in a JSON doc which requires custom tooling to read (i.e., is not RSS or Atom). There's no good way to receive notifications when new release notes are available (except for subscribing to the repo where the release notes are pushed).
• Documentation does not explicitly recommend reading release notes, but a prudent user will do so anyway.
• Release notes are generally considered immutable, and are not changed even if incorrect or incomplete.
• The docs site takes half an hour to build, imposing a 30- to 90-minute delay between release of artifacts and publication of release notes.

We should design the release notes for Fedora CoreOS.

• Should they be automatically generated? From what inputs?
• Should we provide tools for showing all changes between two versions?
• Should notes be revised if later determined to be incorrect? (e.g., a fix was ineffective)
• Should notes be revised if a release is later determined to introduce an important bug?
• Where should release notes be published? Presumably a web page. RSS/Atom feed? Mailing list? Release and/or stream metadata?
• Where should the authoritative copy of release notes be stored, and how should it be propagated to other copies?

[jlebon]

Just a random note: rpm-ostree has some nice tooling, albeit targeted for the client side, surrounding the question "what changed?".

1. it can download and display security errata for packages and extract CVE information from their synopses
2. rpm-ostree db diff --changelogs gives you a full summary of what changed in packages between two versions

Definitely open to rework those to make them more accessible if they'd be useful in whatever workflow we establish.

[bgilbert]

Discussed at last week's meeting.

FAH release notes consist of an automated email and a followup. The latter can have handwritten content. These messages are sent only for stable releases, and aren't archived anywhere other than the mailing lists.

Our release notes could have several sections:

• A manually-curated set of important information: major changes, notable security or bug fixes
• Perhaps a list of package versions for important packages, as CL does
• Automatically generated list of CVEs
• Something like rpm-ostree db diff --changelogs, which lists the relevant parts of package changelogs. We could filter the package set, or float important packages to the top. For new upstream releases of relevant packages, we could automatically link to the upstream release notes.

We should have a permalink for each set of release notes. We might want to use several publication methods, as different users may have different preferences: mailing list, RSS, Twitter, etc.

[dhawal55]

Any update on this? What are the plan for release notes? Can we at least start with something simple and build upon it?

[jlebon]

Re. authoritative source, one simple idea to get started is just a release.md in https://github.com/coreos/fedora-coreos-config production branches. I.e. this naturally ties the release notes to the content promotion process and we can have it be generated at that time, and reviewed, and merged atomically with the content, etc... (Guess it could be as the commit message too, though GitHub doesn't make reviewing commit messages easy, and it's less flexible in case we want to fix up a typo or something).

[jlebon]

Where should release notes be published? Presumably a web page.

Not sure if we want to maintain a separate site from https://getfedora.org/en/coreos, so my vote is to keep it there. This argues for putting it in the stream metadata of course, but we're not necessarily tied to that either and could have the notes fetched from somewhere else.

[mariusgrigoriu]

Until we figure out how changelogs are published, what is the recommended way to get a sense of what's new with any given release prior to taking an update?

[jlebon]

Until we figure out how changelogs are published, what is the recommended way to get a sense of what's new with any given release prior to taking an update?

@zonggen is working on a redesign of the download page that will show the package list and package diff at least until we enhance it with nicer release notes.

For now, you can do something like:

# ostree pull fedora:fedora/x86_64/coreos/stable
# rpm-ostree db diff fedora:fedora/x86_64/coreos/stable --changelogs

[fifofonix]

The newly re-worked release pages with the summaries of changed packages are a definite plus but I think the 'next' stream should be calling out the docker version change to 19.03.8 from 18.09.8 as a top-level change right now. Just like there are call outs for versions of ignition/podman etc.

I'm not sure how this all comes together but I did do an rpm-ostree db list on next and grepped for docker to no avail to confirm why the docker change isn't even actually listed in the full listing. I guess it is layered into the build in a way that I'm not appreciating? Perhaps these additional pieces need to be scanned somehow for versions when building auto release highlight pages to give them visibility.

[bgilbert]

@fifofonix The package you're looking for is moby-engine.

[fifofonix]

Got it. And yes of course that is in the full list of packages. But I still think it deserves a more obvious call out.

[dustymabe]

@zonggen - can we add moby-engine to the short list of packages that get highlighted for each release of FCOS on the release notes page?

[zonggen]

@dustymabe Sure. For reference, we have a list of important packages to show in the build browser and the download page:

• https://github.com/coreos/fedora-coreos-browser/blob/master/index.html#L194
• https://pagure.io/fedora-web/websites/blob/master/f/sites/static/js/coreos-release-notes.js#_21

Filed https://pagure.io/fedora-web/websites/pull-request/118

EDIT:
Also as a followup on the moby-engine, filed coreos/fedora-coreos-browser#14 to add moby-engine into the build browser important package list.

[dustymabe]

We discussed this in the meeting today.

In general I think we agreed that we could use a file (probably yaml) in the tip of each production stream (stable, testing, next) to collect release note information about each release in that stream. How we plumb that through to get to the release notes on the website is still up for discussion.

Another idea was that we come up with a way to add release notes as we merge in PRs. So something like a release-notes.d/ directory in the testing-devel branch where each new PR could add content there (unique file names avoid merge conflicts) and then it gets collated together when we do a release.

[zonggen]

Does the following design sound feasible:

• Include a changelog.yml file inside each of the build directory (or changelog.d/ that contains the yaml files)
• merge the yaml files from last release build upto the next release builds into a single release changelog file, and also de-dup the changelog
• release page use the final changelog yaml file to display desired information

Though this sounds like a naive solution..

[dustymabe]

Does the following design sound feasible:

• Include a changelog.yml file inside each of the build directory (or changelog.d/ that contains the yaml files)

What do you mean by "build directory" here? Are you referring to having a file or directory in each branch of the fcos configs repo?

• merge the yaml files from last release build upto the next release builds into a single release changelog file, and also de-dup the changelog

Yep. I'm thinking that when we take the contents from the /.d directory on the testing-devel stream and merge it together and put it under a nextrelease entry in the yaml.

Once we do the release we update the entry in the yaml to be the version number from the build we just put out.

Example toplevel yaml:

releases:
  - nextrelease:
    - removed package xyz
        - This package was causing problems. Workaround by foo foo foo.
    - added package abc
    - fixed [bug3](https://example.com/bug3)
  - 32.20200629.3.0:
    - removed package xyz
    - added package abc
    - fixed [bug4](https://example.com/bug3)

nextrelease would then get replaced when we put out the release. There are a lot of mechanics here and a lot of ways to do this. I think the important part is that we get something in place and then start ot iterate on it.

• release page use the final changelog yaml file to display desired information

👍

Another thing to think about is the format of the yaml file. Would we accept markdown and then try to render it as html?

[zonggen]

What do you mean by "build directory" here? Are you referring to having a file or directory in each branch of the fcos configs repo?

Yes, I was thinking about adding /.d or a .yaml in each branch of fcos config repo.

the important part is that we get something in place and then start ot iterate on it.

👍 I will start experimenting from the basic approach and see how it looks.

Would we accept markdown and then try to render it as html?

Parsing markdown into html and inject into Vue.js code is feasible however may require additional dependencies, but I do think markdown is much more readable so will experiment with the dependencies. Yaml and json parser is built into javascript so reading them should be straightforward.

[zonggen]

Further discussion with @dustymabe leads to the following design.

The reason for separating out the release notes creation from FCOS pipeline is that when we want to make adjustment to the old release notes we've already published, we can manually update the release-notes.yaml entry for that release and update the website, independent of release cycles.

Developement situation:

1. Developer A opens PR to testing-devel repo to fedora-coreos-config repo
   • adds release note to release-notes.d/fixed-ignition-bug.yaml
     • fixed bug 77 in ignition all nodes now use DHCP
   • PR gets merged
2. Developer B opens PR to testing-devel repo to fedora-coreos-config repo
   • adds release note to release-notes.d/fixed-afterburn-bug.yaml
     • fixed bug 87[https://link/to/bug/87] in afterburn all nodes now use ecdsa SSH keys
   • PR gets merged
3. We want to do a testing release
   • merge testing-devel into testing
   • run a build through the build pipeline
     • this is the step where we learn what the version number is (i.e. 32.20200629.3.0)
   • we do some final checks (mostly just little inspections)
   • then we run the release pipeline
     • replicates to aws regions
     • copies ostree content to prod repo
   • then we run fedora-coreos-stream-generator
     • metadata processing
   • copy metadata to S3
     • Update server (cincinatti) picks up new metadata from s3, serves updates
     • Website picks up new metadata from s3, serves updates

Note: In the case of overgrowing release-notes.yaml, do mv releasenotes.yaml 2020-releasenotes.yaml annually (or periodically)

Proposal:

Have a tool that runs independently of the pipeline like bump-lockfile and sync-stream-metadata jenkins jobs. This tool will trigger based on a new release of FCOS or can be triggered manually. The responsibility of this tool includes:

Git repository manipulation

• look at the release-notes.yaml file for the related branch (i.e. testing)
  • if an entry for that release exists, it does nothing to the git repo
  • if an entry for that release does not exist
    • create a new entry in release-notes.yaml for that release version
      • merge the release-notes.d/* files into release-notes.yaml
      • delete release-notes.d/* files from that branch
      • delete corresponding release-notes.d/* files from testing-devel branch

Example release-notes.yaml after the merge:

    releases:
      - 32.20200629.3.0:
        - inline: |
            - fixed bug [77](https://link/to/bug/77) in ignition all nodes now use DHCP
            - fixed bug [87](https://link/to/bug/87) in afterburn all nodes now use ecdsa SSH keys
      - 32.20200619.3.0:
        - DISCLAIMER: There was a serious flaw affecting this release that causes AWS nodes to reboot after 4 hours of uptime.
        - old release note 1
        - old release note 2

Intermediate processing to get into ideal form for website code consumption

• process release-notes.yaml (maybe convert to JSON or other formats)

Upload to place where website can consume

• upload to s3
• release-notes.yaml picked up by FCOS release-notes.js for rendering

[zonggen]

• Create a release-notes.d directory that stores the pre-mature release note yaml snippets (coreos/fedora-coreos-config#550)
• Create a release-note-generator script that merges the pre-mature yaml snippets and writes to the next-release field of release-notes.yaml (coreos/fedora-coreos-releng-automation#119)
• Enable release-note-generator script to replace the next-release key with the corresponding build id (coreos/fedora-coreos-releng-automation#119)
• Add a separate job in fedora-coreos-pipeline that automates release note generation and cleanup step for yaml snippets as well as the step to push to S3 bucket (coreos/fedora-coreos-pipeline#261)
• Pick up release-note.yaml from Fedora CoreOS website and nicely render the release notes

[egeturgay]

It would be great if the team consider some improvements on the release notes, particularly on the CVEs. Visibility in CVEs allow us to choose to upgrade or hold on to a specific release. It's an important set of information for various compliance requirements.
We currently workaround it by running the below and looking up CVE severity manually on the web.
ostree pull fedora:fedora/x86_64/coreos/stable && rpm-ostree db diff fedora:fedora/x86_64/coreos/stable --changelogs | grep -A 1 -B 3 CVE

[bgilbert]

Visibility in CVEs allow us to choose to upgrade or hold on to a specific release. It's an important set of information for various compliance requirements.

@egeturgay Could you expand on that a bit? Any CVEs that apply to a new release are likely to apply to previous releases as well, so when would CVE metadata cause you not to upgrade?

[egeturgay]

@bgilbert we wouldn't upgrade to a newer release if the CVEs are attributed with a low score (low, moderate etc) and if they don't affect the services we run (e.g. cups, samba), we may choose to not upgrade until we come across a high risk fix.
We replace the instances with the new releases based on these after testing them rather than using the in-place auto update to ensure we don't experience any regressions.

Sometimes regressions can be side effects of a newer version of the OS components, for example the recent systemd version upgrade broke fluentd systemd integration which stopped all logging.

We may choose to test our dev systems for a longer period to ensure we don't experience such issues if the CVEs are low enough and reduce the number of re-deployments across our stacks.

[travier]

For the CVE case more specifically, support may come with coreos/rpm-ostree#1696

[jlebon]

Re. CVEs, there's a draft PR in coreos/rpm-ostree#2695 which will make this much easier for whatever release notes tooling to query.