Automatic changelog in book

[Tom-van-Woudenberg]

Use tags and releases in github to point to specific commits like this one: https://github.com/TeachBooks/bridging_mechanics/releases/tag/2024_week1-7
Each release on a branch should end up on a changelog in the book of that branch containing the description of the release and a link to the changelog, now the example has a manual changelog: https://github.com/TeachBooks/bridging_mechanics/releases/tag/2024_week1-7 https://teachbooks.github.io/bridging_mechanics/2024/changelog.html

Does anyone has further ideas how this could work? @rlanzafame @jialeidng @TeachBooks/tu-delft-cs-tas @TeachBooks/escience-center-rses

[Tom-van-Woudenberg]

Preferably, you shold be able to use markdown relative links to book pages in the release description.

[rlanzafame]

I think the tag is a nice way to go for authors, and could be useful for some subset of readers. however, i think for most authors the changelog in the book itself will be more important than that in the source code/repo. Regardless, seems like the big question is how to easily and efficiently keep tags and book changelogs in sync (without writing a lot of hard to maintain scripts or developing a massive software package to visualize changes).

I think the tag and link to a branch comparison is pretty good. then its on the author team to make branch and commit messages that are informative

[Tom-van-Woudenberg]

I think the script to generate the changelog can be small (checks git releases, summarizes in markdown, add to TOC). Visualization of changes is more substantial and could be done at later stage but might have overlap with the personal book project: we need some kind of rendered html diff tool.

[Tom-van-Woudenberg]

Furthermore, I think the official publishing process of the library would like specific releases (more similar to editions of written books)

[BSchilperoort]

In our projects we generally keep a changelog manually. By working with PR's and using squash-merge the commit history is generally pretty clear so even if we forget something, finishing up the changelog when we prepare for a release is quite straightforward.
However, this does require a structured way of working and some experience in writing changelogs, doing pull/merge requests, etc.

Github itself does already has a button "Generate release notes" which pulls info from the commits since the last release to create an automatic overview. However, that depends on how well you've written your commit messages. And, it would have all changes in there (not necessarily relevant for readers):

i think for most authors the changelog in the book itself will be more important than that in the source code/repo.

we need some kind of rendered html diff tool.

Seems indeed like you would want a visual-diff of the rendered HTML pages. And manually writing changelogs is undesired 😉

Assuming no one messes with the structure of files (names, location) a workflow to create an html diff could be;

• checkout the previous tag, build html
• checkout current repo, build html
• generate diff of all html files (with diff: builtin linux command).
  • Or alternatively; https://diff2html.xyz/#cli

[Tom-van-Woudenberg]

Thanks for your contribution @BSchilperoort! Do you know of diffs of HTML pages which actually show the diff of the rendered pages instead of the diff of the source code?

[BSchilperoort]

This is often called a "rich" diff. Generating these for HTML is not easy, especially with pages less-linear than the teachbook chapters. I did find a BSc thesis on the topic.

Generating rich diffs for markdown is easier (as it's closer to plain text). Rich diffs for notebooks are also difficult, which is why Github's rich notebook diff has been in preview for 1.5 years already. It uses nbdime.

Using nbdime's nbdiff-web command might be the easiest way to get a rich diff for the notebook-based content.

[Tom-van-Woudenberg]

Ah, so ideally the changelog would create those visual diffs for all changed markdowns and notebook files (which won't porperly visualize all book-specific myst / interactive stuff) and put them in the book so it's visible for readers

[BSchilperoort]

Yes, I think that would be the most viable way forward.

[Tom-van-Woudenberg]

Made into a project here: Automatic changelog (view)

[Tom-van-Woudenberg]

@rlanzafame , @astoeriko

[rlanzafame]

Tom and I were talking. @BSchilperoort , you see any major flaws?

1. We discussed releases versus other options and realized releases is the obvious place to start.
2. Is release information in a GitHub repo accessible from all parts of the repo, specifically branches? For example, if we have releases on different branches, that should not affect release-based changelogs, right? So like release-3 on branch-1 and release-4 on branch-2 works same as if they were on same branch.
3. If so we think the next step is to use the teachbooks package to automatically generate a changelog (we start with creating the log then worry about the fancy stuff like rich diffs later). We could then set up a reusable workflow that would simply use the teachbooks package to execute the changelog generation in the pre-processing phase of the book build.
4. the teachbooks package would basically just need to know the github repo where the releases are defined (config file). let's now worry about private repo's (until someone requests it). we envision the changelog to generate automatically, if releases are used in the repo, unless turned off (in config file).

[BSchilperoort]

Is release information in a GitHub repo accessible from all parts of the repo, specifically branches? For example, if we have releases on different branches, that should not affect release-based changelogs, right? So like release-3 on branch-1 and release-4 on branch-2 works same as if they were on same branch.

Releases are a GitHub only thing, not visible from anywhere but github. You create a release from a certain branch. When you create a release you also need to create a "tag". Tags are git things, on a repository level. You can checkout a specific tag with git checkout tags/<tag_name>.

If so we think the next step is to use the teachbooks package to automatically generate a changelog (we start with creating the log then worry about the fancy stuff like rich diffs later). We could then set up a reusable workflow that would simply use the teachbooks package to execute the changelog generation in the pre-processing phase of the book build.

I would separate changelogs and diffs. A changelog is generally human readable and an overview of what changed. Generating these automatically is difficult. Ideally you keep track of changes in a CHANGELOG.md file (example)

Diffs (even fancy rich ones) can be difficult to see things in, and take too much effort/time to compare.

[Tom-van-Woudenberg]

We came to the conclusion that we should just have a markdown file changelog.md inside the book directory which can also be rendered in the book. People can use releases to refer to specific versions and link to these from the changelog

[Tom-van-Woudenberg]

Covered in TeachBooks/manual#114