Catalog: bootstrap the change log

[nl0]

No description provided.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 38.43%. Comparing base (be8f2b2) to head (1d08abf).
Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff            @@
##           master    #4112    +/-   ##
========================================
  Coverage   38.43%   38.43%            
========================================
  Files         724      724            
  Lines       33256    33256            
  Branches     4715     4889   +174     
========================================
  Hits        12783    12783            
+ Misses      19849    19324   -525     
- Partials      624     1149   +525     

Flag	Coverage Δ
api-python	90.85% <ø> (ø)
catalog	11.43% <ø> (ø)
lambda	87.95% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[fiskus]

I understand, that we need to separate Catalog changelog from quilt3 and lambdas. Mostly, I understand that quilt3 changelog shouldn't contain Catalog's changes.

But why do we need a Changelog in the Catalog? Won't it be the same as a list of commits?

[nl0]

I understand, that we need to separate Catalog changelog from quilt3 and lambdas. Mostly, I understand that quilt3 changelog shouldn't contain Catalog's changes.

But why do we need a Changelog in the Catalog? Won't it be the same as a list of commits?

pls refer to https://keepachangelog.com/en/1.1.0/ for some context.
specifically:

Can changelogs be bad?

Yes. Here are a few ways they can be less than useful.

Commit log diffs

Using commit log diffs as changelogs is a bad idea: they're full of noise.
Things like merge commits, commits with obscure titles, documentation changes, etc.
The purpose of a commit is to document a step in the evolution of the source code.
Some projects clean up commits, some don't.
The purpose of a changelog entry is to document the noteworthy difference, often across multiple commits, to communicate them clearly to end users.

also, commit log would list all the commits across the repo, not only the ones related to the catalog

[fiskus]

I know that, I read it. But we squash commits and use PR's only to merge into master.
So, git log ./catalog shows almost the same as changelog.md.

We need to cut corners somewhere, because we have huge amount of manual work. So, one thing to consider is to optimize creating changelogs. We log every feature multiple times. I think, creating changelog for catalog doesn't worth it. Especially, because we don't have changelog versions/releases.

[fiskus]

In other words, I don't say we should use commits log as changelog. I'm asking why do we need a changelog for catalog? We have a changelog for catalog features released to customers, that's understandable. But why here?

[nl0]

I know that, I read it. But we squash commits and use PR's only to merge into master. So, git log ./catalog shows almost the same as changelog.md.

well, almost is the key word here. there are also dependency updates, for example.

we don't have changelog versions/releases.

that's something we should address soon-ish

I'm asking why do we need a changelog for catalog?

to keep the log of the notable changes readily available

We have a changelog for catalog features released to customers, that's understandable. But why here?

because this is the source of truth for the catalog.
we (me, at least) take changelog entries from this repo when creating a changelog entry for the deployment.

also, it becomes trivial to see the changes to a specific component as a diff to the specific changelog file (catalog, in this case, but this is also true for the lambdas, for example).

example: try figuring out what's changed in the s3hash lambda by looking at this commit log.
then navigate to the lambdas/s3hash/CHANGELOG.md in the list of changed files and there it is -- that makes the whole thing worthwile for me.

a nice step forward, i believe, would be adopting changesets or something similar to automate the bulk of the release/changelog maintenance work, but, well, that requires some work

[nl0]

also, i don't remember you complaining while recording the changes in the main quilt3 changelog.
why the change of heart after this seemingly minor adjustment?

[fiskus]

I don't complain. I saw an opportunity to cut duplicated work, and raised a discussion about that

[nl0]

I don't complain. I saw an opportunity to cut duplicated work, and raised a discussion about that

ok, tho i don't see a lot of duplication here (the only "true duplicate" i see is the changelog entry usually repeats the PR title, but quite often the wording may be a little different).
the work of describing changes is happening here in this repo (and in other components' respective repos),
and then the entries get transplanted / aggregated into the main changelog in deployment that gives a holistic picture of the evolution of the platform (and it's also used as a basis for the customer-facing updates)

[fiskus]

we don't have changelog versions/releases.

that's something we should address soon-ish

Ok, changelog makes sense for releases. But I'm not aware of these plans. Could you elaborate? I think, Slack "#engeneering" channel would be a better place for this.

[fiskus]

example: try figuring out what's changed in the s3hash lambda by looking at this commit log.

It seems, that this log is more relevant https://github.com/quiltdata/quilt/commits/master/lambdas/s3hash

[nl0]

we don't have changelog versions/releases.

that's something we should address soon-ish

Ok, changelog makes sense for releases. But I'm not aware of these plans. Could you elaborate?

there are no specific plans, that's just something that makes sense and we were thinking about (low key) for a long time,
but there is always something more important than this.

tho i want to stress that having labelled releases per se does not make a huge difference in this case,
see my example above showing the value of having per-component changelogs in the current setup
(our platform changelog links to the component changes using the exact same format of github compare links)

[nl0]

example: try figuring out what's changed in the s3hash lambda by looking at this commit log.

It seems, that this log is more relevant https://github.com/quiltdata/quilt/commits/master/lambdas/s3hash

more relevant how?

compared to the manually curated changelog this:

1. cannot show the difference between two commits
2. has a lot of noise
3. may miss some changes from other related files not in this folder (e.g. gh workflows)

[fiskus]

Oh, I see. You, meant the diff 57095a7...a1cf658#files_bucket, not the list of commits