Skip to content

Add Behat scenarios testing SHA-512 enforcement and MD5 handling during update #1098

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
swissspidy merged 3 commits into from
Aug 3, 2026
+90 −2
Merged

Add Behat scenarios testing SHA-512 enforcement and MD5 handling during update #1098

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5f68db8
Add Behat scenarios testing SHA-512 enforcement and MD5 handling duri…
swissspidy Aug 3, 2026
8b84fc1
Merge branch 'main' into add/sha-tests
swissspidy Aug 3, 2026
c2875f4
Merge main into add/sha-tests resolving cli.feature conflicts
swissspidy Aug 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 90 additions & 2 deletions features/cli.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,95 @@ Feature: `wp cli` tasks
WP-CLI {UPDATE_VERSION}
"""

Scenario: Fail update when sha512 hash cannot be accessed
Given an empty directory
And a new Phar with version "2.8.0"
And that HTTP requests to https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli-nightly.phar.sha512 will respond with:
"""
HTTP/1.1 404 Not Found
Content-Type: text/plain

Not Found
"""

When I try `{PHAR_PATH} cli update --nightly --yes`
Then STDERR should contain:
"""
Error: Couldn't access sha512 hash for release (HTTP code 404).
"""
And the return code should be 1

Scenario: Fail update when sha512 hash mismatches
Given an empty directory
And a new Phar with version "2.8.0"
And that HTTP requests to https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli-nightly.phar.sha512 will respond with:
"""
HTTP/1.1 200 OK
Content-Type: text/plain

invalidsha512hash
"""

When I try `{PHAR_PATH} cli update --nightly --yes`
Then STDERR should contain:
"""
Error: sha512 hash for download
"""
And STDERR should contain:
"""
is different than the release hash (invalidsha512hash).
"""
Comment thread
swissspidy marked this conversation as resolved.
And the return code should be 1
Comment on lines +195 to +215

@coderabbitai coderabbitai Bot Aug 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Verify that a rejected update does not replace the Phar.

An error and exit code of 1 do not prove that verification prevented installation. A faulty implementation can replace {PHAR_PATH} and then report the verification failure.

  • features/cli.feature#L195-L215: After the SHA-512 mismatch, assert {PHAR_PATH} --version still returns WP-CLI 2.8.0.
  • features/cli.feature#L176-L193: After SHA-512 metadata access fails, assert {PHAR_PATH} --version still returns WP-CLI 2.8.0.
  • features/cli.feature#L244-L264: After the MD5 mismatch, assert {PHAR_PATH} --version still returns WP-CLI 2.8.0.
Proposed assertion 
+    When I run `{PHAR_PATH} --version`
+    Then STDOUT should be:
+      """
+      WP-CLI 2.8.0
+      """
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
Scenario: Fail update when sha512 hash mismatches
Given an empty directory
And a new Phar with version "2.8.0"
And that HTTP requests to https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli-nightly.phar.sha512 will respond with:
"""
HTTP/1.1 200 OK
Content-Type: text/plain
invalidsha512hash
"""
When I try `{PHAR_PATH} cli update --nightly --yes`
Then STDERR should contain:
"""
Error: sha512 hash for download
"""
And STDERR should contain:
"""
is different than the release hash (invalidsha512hash).
"""
And the return code should be 1
Scenario: Fail update when sha512 hash mismatches
Given an empty directory
And a new Phar with version "2.8.0"
And that HTTP requests to https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli-nightly.phar.sha512 will respond with:
"""
HTTP/1.1 200 OK
Content-Type: text/plain
invalidsha512hash
"""
When I try `{PHAR_PATH} cli update --nightly --yes`
Then STDERR should contain:
"""
Error: sha512 hash for download
"""
And STDERR should contain:
"""
is different than the release hash (invalidsha512hash).
"""
And the return code should be 1
When I run `{PHAR_PATH} --version`
Then STDOUT should be:
"""
WP-CLI 2.8.0
"""
📍 Affects 1 file
  • features/cli.feature#L195-L215 (this comment)
  • features/cli.feature#L176-L193
  • features/cli.feature#L244-L264
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@features/cli.feature` around lines 195 - 215, Extend the SHA-512 mismatch
scenario in features/cli.feature lines 195-215 to assert that {PHAR_PATH}
--version still reports WP-CLI 2.8.0. Add the same post-failure version
assertion to the SHA-512 metadata access failure scenario in
features/cli.feature lines 176-193 and the MD5 mismatch scenario in
features/cli.feature lines 244-264, confirming each rejected update leaves the
original Phar installed.


Scenario: Update succeeds when sha512 matches even if md5 cannot be accessed
Given an empty directory
And a new Phar with version "2.8.0"
And that HTTP requests to https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli-nightly.phar.md5 will respond with:
"""
HTTP/1.1 404 Not Found
Content-Type: text/plain

Not Found
"""

When I run `{PHAR_PATH} cli update --nightly --yes`
Then STDOUT should contain:
"""
sha512 hash verified:
"""
And STDOUT should contain:
"""
Couldn't access md5 hash for release (HTTP code 404).
"""
And STDOUT should contain:
"""
Success: Updated WP-CLI to the latest nightly release.
"""
And STDERR should be empty
And the return code should be 0

Scenario: Fail update when md5 hash mismatches
Given an empty directory
And a new Phar with version "2.8.0"
And that HTTP requests to https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli-nightly.phar.md5 will respond with:
"""
HTTP/1.1 200 OK
Content-Type: text/plain

invalidmd5hash
"""

When I try `{PHAR_PATH} cli update --nightly --yes`
Then STDERR should contain:
"""
Error: md5 hash for download
"""
And STDERR should contain:
"""
is different than the release hash (invalidmd5hash).
"""
And the return code should be 1

Scenario: Prevent stable update when PHP version requirement is not met
Given an empty directory
And a new Phar with version "2.8.0"
Expand Down Expand Up @@ -212,5 +301,4 @@ Feature: `wp cli` tasks
"""
The requested update requires PHP 99.0.0 or higher.
"""
And the return code should be 1

And the return code should be 1
Loading