Automated Testing: Use three-dot diff comparison for changelog checks#79548
Conversation
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
|
Size Change: 0 B Total Size: 7.51 MB |
This reverts commit 1cd3d89.
Yeah I learned a lot today 😄 Also relevant is that "base SHA" is a moving target in the context of these tests and depends on when the checks are run, where typically we're thinking in terms of the merge-base commit. |
|
Flaky tests detected in 7d3b4d0. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/28177258096
|
…WordPress#79548) * Automated Testing: Use three-dot diff comparison for changelog checks * Temporary: Sample failure * Revert "Temporary: Sample failure" This reverts commit 1cd3d89. --- Co-authored-by: aduth <aduth@git.wordpress.org> Co-authored-by: ciampo <mciampini@git.wordpress.org> Co-authored-by: jsnajdr <jsnajdr@git.wordpress.org>
What?
Fixes an issue where the "Check Package Changelogs" CI check may produce false positives.
Related: #79460 (comment)
Why?
Our tests should be accurate to reflect their expected requirements, lest false positive cause confusion or decrease trust in the tests.
How?
Before these changes, the changelog check was effectively doing
git diff $BASE_SHA HEAD, which is a "two dot" operation (ref).For pull request changes, three-dot notation should be used, and is consistent with how GitHub presents pull request changes:
For the false positive flagged in #79460, you can see the specific difference in behavior with how the PR's last commit compares against trunk as it exists at the point of the check being run:
packages/themechanges which is what triggered the test failure)packages/themechanges)Testing Instructions
Observe that the sample failure commit triggers test failure for "Check changelog" test.
Use of AI Tools
Used Cursor + Composer for research, changes were applied myself.