Skip to content

Automated Testing: Use three-dot diff comparison for changelog checks#79548

Merged
ciampo merged 3 commits into
trunkfrom
fix/pr-changelog-check-three-dot
Jun 26, 2026
Merged

Automated Testing: Use three-dot diff comparison for changelog checks#79548
ciampo merged 3 commits into
trunkfrom
fix/pr-changelog-check-three-dot

Conversation

@aduth

@aduth aduth commented Jun 25, 2026

Copy link
Copy Markdown
Member

What?

Fixes an issue where the "Check Package Changelogs" CI check may produce false positives.

Related: #79460 (comment)

Why?

Our tests should be accurate to reflect their expected requirements, lest false positive cause confusion or decrease trust in the tests.

How?

Before these changes, the changelog check was effectively doing git diff $BASE_SHA HEAD, which is a "two dot" operation (ref).

For pull request changes, three-dot notation should be used, and is consistent with how GitHub presents pull request changes:

Since the three-dot comparison compares with the merge base, it is focusing on "what a pull request introduces."

For the false positive flagged in #79460, you can see the specific difference in behavior with how the PR's last commit compares against trunk as it exists at the point of the check being run:

Testing Instructions

Observe that the sample failure commit triggers test failure for "Check changelog" test.

Use of AI Tools

Used Cursor + Composer for research, changes were applied myself.

@aduth aduth requested review from a team and desrosj as code owners June 25, 2026 14:03
@github-actions

github-actions Bot commented Jun 25, 2026

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: aduth <aduth@git.wordpress.org>
Co-authored-by: ciampo <mciampini@git.wordpress.org>
Co-authored-by: jsnajdr <jsnajdr@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

@github-actions github-actions Bot added the [Package] Theme /packages/theme label Jun 25, 2026
@aduth aduth added the [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. label Jun 25, 2026
@github-actions

Copy link
Copy Markdown

Size Change: 0 B

Total Size: 7.51 MB

compressed-size-action

@ciampo ciampo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

I never really considered this aspect of git diffing.

@github-actions github-actions Bot removed the [Package] Theme /packages/theme label Jun 25, 2026
@aduth

aduth commented Jun 25, 2026

Copy link
Copy Markdown
Member Author

I never really considered this aspect of git diffing.

Yeah I learned a lot today 😄 Also relevant is that "base SHA" is a moving target in the context of these tests and depends on when the checks are run, where typically we're thinking in terms of the merge-base commit.

@github-actions

Copy link
Copy Markdown

Flaky tests detected in 7d3b4d0.
Some tests passed with failed attempts. The failures may not be related to this commit but are still reported for visibility. See the documentation for more information.

🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/28177258096
📝 Reported issues:

@jsnajdr jsnajdr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense 👍

@ciampo ciampo merged commit 5da7d3f into trunk Jun 26, 2026
42 checks passed
@ciampo ciampo deleted the fix/pr-changelog-check-three-dot branch June 26, 2026 15:08
@github-actions github-actions Bot added this to the Gutenberg 23.6 milestone Jun 26, 2026
SainathPoojary pushed a commit to SainathPoojary/gutenberg that referenced this pull request Jun 29, 2026
…WordPress#79548)

* Automated Testing: Use three-dot diff comparison for changelog checks

* Temporary: Sample failure

* Revert "Temporary: Sample failure"

This reverts commit 1cd3d89.

---

Co-authored-by: aduth <aduth@git.wordpress.org>
Co-authored-by: ciampo <mciampini@git.wordpress.org>
Co-authored-by: jsnajdr <jsnajdr@git.wordpress.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

[Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants