Add Behat scenarios testing SHA-512 enforcement and MD5 handling during update - #1098
Merged
Conversation
swissspidy
added
scope:testing
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe CLI feature suite adds scenarios for SHA-512 metadata failures, SHA-512 mismatches, MD5 fallback warnings, and MD5 mismatches. The scenarios verify output, errors, warnings, and exit codes. ChangesCLI update verification
Estimated code review effort: 2 (Simple) | ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@features/cli.feature`:
- Around line 195-215: Extend the SHA-512 mismatch scenario in
features/cli.feature lines 195-215 to assert that {PHAR_PATH} --version still
reports WP-CLI 2.8.0. Add the same post-failure version assertion to the SHA-512
metadata access failure scenario in features/cli.feature lines 176-193 and the
MD5 mismatch scenario in features/cli.feature lines 244-264, confirming each
rejected update leaves the original Phar installed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
Comment on lines
+195
to
+215
| Scenario: Fail update when sha512 hash mismatches | ||
| Given an empty directory | ||
| And a new Phar with version "2.8.0" | ||
| And that HTTP requests to https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli-nightly.phar.sha512 will respond with: | ||
| """ | ||
| HTTP/1.1 200 OK | ||
| Content-Type: text/plain | ||
|
|
||
| invalidsha512hash | ||
| """ | ||
|
|
||
| When I try `{PHAR_PATH} cli update --nightly --yes` | ||
| Then STDERR should contain: | ||
| """ | ||
| Error: sha512 hash for download | ||
| """ | ||
| And STDERR should contain: | ||
| """ | ||
| is different than the release hash (invalidsha512hash). | ||
| """ | ||
| And the return code should be 1 |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Verify that a rejected update does not replace the Phar.
An error and exit code of 1 do not prove that verification prevented installation. A faulty implementation can replace {PHAR_PATH} and then report the verification failure.
features/cli.feature#L195-L215: After the SHA-512 mismatch, assert{PHAR_PATH} --versionstill returnsWP-CLI 2.8.0.features/cli.feature#L176-L193: After SHA-512 metadata access fails, assert{PHAR_PATH} --versionstill returnsWP-CLI 2.8.0.features/cli.feature#L244-L264: After the MD5 mismatch, assert{PHAR_PATH} --versionstill returnsWP-CLI 2.8.0.
Proposed assertion
+ When I run `{PHAR_PATH} --version`
+ Then STDOUT should be:
+ """
+ WP-CLI 2.8.0
+ """📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| Scenario: Fail update when sha512 hash mismatches | |
| Given an empty directory | |
| And a new Phar with version "2.8.0" | |
| And that HTTP requests to https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli-nightly.phar.sha512 will respond with: | |
| """ | |
| HTTP/1.1 200 OK | |
| Content-Type: text/plain | |
| invalidsha512hash | |
| """ | |
| When I try `{PHAR_PATH} cli update --nightly --yes` | |
| Then STDERR should contain: | |
| """ | |
| Error: sha512 hash for download | |
| """ | |
| And STDERR should contain: | |
| """ | |
| is different than the release hash (invalidsha512hash). | |
| """ | |
| And the return code should be 1 | |
| Scenario: Fail update when sha512 hash mismatches | |
| Given an empty directory | |
| And a new Phar with version "2.8.0" | |
| And that HTTP requests to https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli-nightly.phar.sha512 will respond with: | |
| """ | |
| HTTP/1.1 200 OK | |
| Content-Type: text/plain | |
| invalidsha512hash | |
| """ | |
| When I try `{PHAR_PATH} cli update --nightly --yes` | |
| Then STDERR should contain: | |
| """ | |
| Error: sha512 hash for download | |
| """ | |
| And STDERR should contain: | |
| """ | |
| is different than the release hash (invalidsha512hash). | |
| """ | |
| And the return code should be 1 | |
| When I run `{PHAR_PATH} --version` | |
| Then STDOUT should be: | |
| """ | |
| WP-CLI 2.8.0 | |
| """ |
📍 Affects 1 file
features/cli.feature#L195-L215(this comment)features/cli.feature#L176-L193features/cli.feature#L244-L264
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@features/cli.feature` around lines 195 - 215, Extend the SHA-512 mismatch
scenario in features/cli.feature lines 195-215 to assert that {PHAR_PATH}
--version still reports WP-CLI 2.8.0. Add the same post-failure version
assertion to the SHA-512 metadata access failure scenario in
features/cli.feature lines 176-193 and the MD5 mismatch scenario in
features/cli.feature lines 244-264, confirming each rejected update leaves the
original Phar installed.
There was a problem hiding this comment.
Pull request overview
Adds Behat coverage to the WP-CLI bundle to validate the self-update integrity behavior introduced in wp-cli/wp-cli#6370, specifically enforcing SHA-512 verification while treating MD5 as a secondary/optional check.
Changes:
- Added a scenario to ensure
wp cli update --nightlyfails when the SHA-512 hash cannot be retrieved. - Added a scenario to ensure
wp cli update --nightlyfails when the SHA-512 hash mismatches. - Added scenarios to ensure MD5 retrieval failures don’t block updates (when SHA-512 matches) and that MD5 mismatches are surfaced as failures.
Suppressed comments (1)
features/cli.feature:252
- The mocked MD5 hash value (
invalidmd5hash) is not a valid MD5 hex digest length. If the updater validates hash format, this scenario could fail due to format handling instead of the intended mismatch path. Use a valid 32-character hex string and update the expected error text accordingly.
"""
HTTP/1.1 200 OK
Content-Type: text/plain
invalidmd5hash
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to wp-cli/wp-cli#6370
Summary by CodeRabbit