Issuing HEAD requests conflicts with signed S3 requests #15604

Arkelenia · 2023-06-27T19:03:44Z

`brew doctor` output

brew doctor
Your system is ready to brew.

Verification

My "brew doctor output" above says Your system is ready to brew. and am still able to reproduce my issue.
I ran brew update twice and am still able to reproduce my issue.
This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

`brew config` output

HOMEBREW_VERSION: 4.0.24
ORIGIN: https://github.com/Homebrew/brew
HEAD: 54c8876dc39047c04de15e7d212979ae8d98cf1c
Last commit: 8 days ago
Core tap JSON: 26 Jun 17:42 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: ["--require-sha"]
HOMEBREW_MAKE_JOBS: 10
HOMEBREW_NO_INSECURE_REDIRECT: set
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: 10-core 64-bit arm_firestorm_icestorm
Clang: 14.0.0 build 1400
Git: 2.40.0 => /opt/homebrew/bin/git
Curl: 7.79.1 => /usr/bin/curl
macOS: 12.6.1-arm64
CLT: 14.1.0.0.1.1666437224
Xcode: N/A
Rosetta 2: false

What were you trying to do (and why)?

Trying to install a formula from a custom tap

What happened (include all command output)?

> brew install fabric-cli
==> Fetching datadog/tap/fabric-cli
==> Downloading https://artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz
==> Downloading from https://s3.amazonaws.com/artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA****************%2F20230627%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=202
######################################################################################################################################################################################################################################################## 100.0%curl: (22) The requested URL returned error: 403

Error: fabric-cli: Failed to download resource "fabric-cli"
Download failed: https://artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz

What did you expect to happen?

the download should succeed.

Step-by-step reproduction instructions (by running `brew` commands)

# This is difficult to reproduce without the company's VPN but I'll include as much debug information below. Our setup is as follows:
# - An artifact repository is available at artifacts.ddci.com.
# - When the repository receives an HTTP request, it returns a 302 status code with an authenticated and signed redirect URL to S3.
# - The redirect is followed and the artifact is downloaded by S3.

# The signature includes the HTTP method. brew emits a HEAD request to the repository which answers with a redirect for a HEAD request to S3. Later, brew emits a GET request using the redirect location to S3. Since the S3 request is signed using a HEAD HTTP method, the request is rejected by S3.

> HOMEBREW_CURL_VERBOSE=1 brew install --debug fabric-cli
/opt/homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /opt/homebrew/Library/Taps/datadog/homebrew-tap/Formula/fabric-cli.rb
==> Fetching datadog/tap/fabric-cli
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.0.26\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 12.6.1\)\ curl/7.79.1 --header Accept-Language:\ en --retry 3 --fail --location --silent --head https://artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz
==> Downloading https://artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz
==> Downloading from https://s3.amazonaws.com/artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA****************%2F20230627%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=202
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.0.26\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 12.6.1\)\ curl/7.79.1 --header Accept-Language:\ en --retry 3 --fail --location --silent --head --location https://s3.amazonaws.com/artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz\?X-Amz-Algorithm=AWS4-HMAC-SHA256\&X-Amz-Credential=ASIA****************\%2F20230627\%2Fus-east-1\%2Fs3\%2Faws4_request\&X-Amz-Date=20230627T173407Z\&X-Amz-Expires=900\&X-Amz-Security-Token=<amazon-security-token>&X-Amz-SignedHeaders=host\&X-Amz-Signature=<amazon-signature>
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.0.26\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 12.6.1\)\ curl/7.79.1 --header Accept-Language:\ en --fail --progress-bar --verbose --retry 3 --remote-time --output /Users/frederic.hemery/Library/Caches/Homebrew/downloads/1fe7ae07946d49445d4ac272a38b5dff4b6a77d83ae5d857a65c43c3a3bbb048--fabric_1.37.0_darwin_arm64.tar.gz.incomplete --continue-at - --location https://s3.amazonaws.com/artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz\?X-Amz-Algorithm=AWS4-HMAC-SHA256\&X-Amz-Credential=ASIA****************\%2F20230627\%2Fus-east-1\%2Fs3\%2Faws4_request\&X-Amz-Date=20230627T173407Z\&X-Amz-Expires=900\&X-Amz-Security-Token=<amazon-security-token>&X-Amz-SignedHeaders=host\&X-Amz-Signature=<amazon-signature>
######################################################################################################################################################################################################################################################## 100.0%*   Trying 52.217.91.6:443...
* Connected to s3.amazonaws.com (52.217.91.6) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
} [321 bytes data]
* (304) (IN), TLS handshake, Server hello (2):
{ [106 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [5486 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=s3.amazonaws.com
*  start date: Apr 11 00:00:00 2023 GMT
*  expire date: Dec 20 23:59:59 2023 GMT
*  subjectAltName: host "s3.amazonaws.com" matched cert's "s3.amazonaws.com"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
*  SSL certificate verify ok.
> GET /artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA****************%2F20230627%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230627T173407Z&X-Amz-Expires=900&X-Amz-Security-Token=<amazon-security-token>&X-Amz-SignedHeaders=host&X-Amz-Signature=<amazon-signature> HTTP/1.1
> Host: s3.amazonaws.com
> Range: bytes=266-
> User-Agent: Homebrew/4.0.26 (Macintosh; arm64 Mac OS X 12.6.1) curl/7.79.1
> Accept: */*
> Accept-Language: en
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< x-amz-request-id: <amazon-request-id>
< x-amz-id-2: <amazon-id-2>
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Tue, 27 Jun 2023 17:34:07 GMT
< Server: AmazonS3
* The requested URL returned error: 403
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
} [2 bytes data]
curl: (22) The requested URL returned error: 403

Error: fabric-cli: Failed to download resource "fabric-cli"
Download failed: https://artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz

# Issuing a HEAD request on the redirect url from the repository succeeds:

> curl --verbose --head 'https://s3.amazonaws.com/artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA****************%2F20230627%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230627T173407Z&X-Amz-Expires=900&X-Amz-Security-Token=<amazon-security-token>&X-Amz-SignedHeaders=host&X-Amz-Signature=<amazon-signature>'
*   Trying 52.216.178.13:443...
* Connected to s3.amazonaws.com (52.216.178.13) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=s3.amazonaws.com
*  start date: Apr 11 00:00:00 2023 GMT
*  expire date: Dec 20 23:59:59 2023 GMT
*  subjectAltName: host "s3.amazonaws.com" matched cert's "s3.amazonaws.com"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
*  SSL certificate verify ok.
> HEAD /artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA****************%2F20230627%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230627T173407Z&X-Amz-Expires=900&X-Amz-Security-Token=<amazon-security-token>&X-Amz-SignedHeaders=host&X-Amz-Signature=<amazon-signature> HTTP/1.1
> Host: s3.amazonaws.com
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< x-amz-id-2: <amazon-id-2>
x-amz-id-2: <amazon-id-2>
< x-amz-request-id: <amazon-request-id>
x-amz-request-id: <amazon-request-id>
< Date: Tue, 27 Jun 2023 17:34:07 GMT
Date: Tue, 27 Jun 2023 17:34:07 GMT
< Last-Modified: Wed, 31 May 2023 19:46:21 GMT
Last-Modified: Wed, 31 May 2023 19:46:21 GMT
< ETag: "18937274ed1ac07cb0681eabbe163407-3"
ETag: "18937274ed1ac07cb0681eabbe163407-3"
< x-amz-server-side-encryption: AES256
x-amz-server-side-encryption: AES256
< Content-Disposition: attachment; filename=fabric_1.37.0_darwin_arm64.tar.gz
Content-Disposition: attachment; filename=fabric_1.37.0_darwin_arm64.tar.gz
< x-amz-version-id: EFr9ttVIIGFSqXvR29244Uh7c_JFagVa
x-amz-version-id: EFr9ttVIIGFSqXvR29244Uh7c_JFagVa
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Content-Type: application/x-gzip
Content-Type: application/x-gzip
< Server: AmazonS3
Server: AmazonS3
< Content-Length: 14191772
Content-Length: 14191772

<
* Connection #0 to host s3.amazonaws.com left intact

# On the other hand, emitting a GET request fails because of a signature mismatch

> curl --verbose 'https://s3.amazonaws.com/artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA****************%2F20230627%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230627T173407Z&X-Amz-Expires=900&X-Amz-Security-Token=<amazon-security-token>&X-Amz-SignedHeaders=host&X-Amz-Signature=<amazon-signature>'
*   Trying 52.216.59.112:443...
* Connected to s3.amazonaws.com (52.216.59.112) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=s3.amazonaws.com
*  start date: Apr 11 00:00:00 2023 GMT
*  expire date: Dec 20 23:59:59 2023 GMT
*  subjectAltName: host "s3.amazonaws.com" matched cert's "s3.amazonaws.com"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
*  SSL certificate verify ok.
> GET /artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA****************%2F20230627%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230627T173407Z&X-Amz-Expires=900&X-Amz-Security-Token=<amazon-security-token>&X-Amz-SignedHeaders=host&X-Amz-Signature=<amazon-signature> HTTP/1.1
> Host: s3.amazonaws.com
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< x-amz-request-id: <amazon-request-id>
< x-amz-id-2: <amazon-id-2>
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Tue, 27 Jun 2023 18:58:17 GMT
< Server: AmazonS3
<
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<redacted details about the signature>

# After investigation, the signature mismatch is caused by the HTTP method mismatch. The redirect url from the repository was built from a HEAD request and later used by brew in a GET request.

The text was updated successfully, but these errors were encountered:

MikeMcQuaid · 2023-06-27T19:47:20Z

CC @reitermarkus for help with the HEAD/GET stuff here.

reitermarkus · 2023-06-27T20:46:38Z

From how I understand this, I think the only way to fix this is not to reuse the resolved URL, i.e. getting rid of “Downloading from”.

MikeMcQuaid · 2023-06-28T10:34:35Z

@reitermarkus Is this something you'd be willing or able to make a PR for? No worries if not.

Arkelenia · 2023-07-17T18:03:39Z

I wanted to follow up on this issue. Can I help in any way on this? Would it be useful if I made a PR?

MikeMcQuaid · 2023-07-17T18:09:47Z

Can I help in any way on this? Would it be useful if I made a PR?

Yes please!

This document should help and we're happy to walk you through anything else.

Thanks!

github-actions · 2023-08-08T00:06:47Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

Arkelenia added the bug Reproducible Homebrew/brew bug label Jun 27, 2023

github-actions bot added the stale No recent activity label Aug 8, 2023

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Aug 16, 2023

github-actions bot added the outdated PR was locked due to age label Sep 15, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issuing HEAD requests conflicts with signed S3 requests #15604

Issuing HEAD requests conflicts with signed S3 requests #15604

Arkelenia commented Jun 27, 2023 •

edited

MikeMcQuaid commented Jun 27, 2023

reitermarkus commented Jun 27, 2023

MikeMcQuaid commented Jun 28, 2023

Arkelenia commented Jul 17, 2023

MikeMcQuaid commented Jul 17, 2023

github-actions bot commented Aug 8, 2023

Issuing HEAD requests conflicts with signed S3 requests #15604

Issuing HEAD requests conflicts with signed S3 requests #15604

Comments

Arkelenia commented Jun 27, 2023 • edited

brew doctor output

Verification

brew config output

What were you trying to do (and why)?

What happened (include all command output)?

What did you expect to happen?

Step-by-step reproduction instructions (by running brew commands)

MikeMcQuaid commented Jun 27, 2023

reitermarkus commented Jun 27, 2023

MikeMcQuaid commented Jun 28, 2023

Arkelenia commented Jul 17, 2023

MikeMcQuaid commented Jul 17, 2023

github-actions bot commented Aug 8, 2023

Arkelenia commented Jun 27, 2023 •

edited

`brew doctor` output

`brew config` output

Step-by-step reproduction instructions (by running `brew` commands)