Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

svt-av1: fix sha256 checksum #127055

Closed
wants to merge 1 commit into from
Closed

svt-av1: fix sha256 checksum #127055

wants to merge 1 commit into from

Conversation

qianbinbin
Copy link
Contributor

  • Have you followed the guidelines for contributing?
  • Have you ensured that your commits follow the commit style guide?
  • Have you checked that there aren't other open pull requests for the same formula update/change?
  • Have you built your formula locally with brew install --build-from-source <formula>, where <formula> is the name of the formula you're submitting?
  • Is your test running fine brew test <formula>, where <formula> is the name of the formula you're submitting?
  • Does your build pass brew audit --strict <formula> (after doing brew install --build-from-source <formula>)? If this is a new formula, does it pass brew audit --new <formula>?

@carlocab carlocab added the checksum mismatch SHA-256 doesn't match the download label Mar 30, 2023
@qianbinbin qianbinbin closed this Mar 30, 2023
@qianbinbin
Copy link
Contributor Author

It's weird that curl and Chrome gives different checksums:

❯ curl -L https://gitlab.com/AOMediaCodec/SVT-AV1/-/archive/v1.4.1/SVT-AV1-v1.4.1.tar.bz2 | sha256sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 9222k    0 9222k    0     0   394k      0 --:--:--  0:00:23 --:--:--  531k
2ddef549e1eaeecc1fc48f0d8332ea3545809e46509db69beb3a0a4bf19ef906  -
❯ sha256sum ~/Downloads/SVT-AV1-v1.4.1.tar.bz2  # downloaded by Chrome
0e988582f315fe76c909accf5e7f81b975c5bd2b850ee760d8e9fac297f70b5d  /Users/username/Downloads/SVT-AV1-v1.4.1.tar.bz2

Seems brew uses the former, is the commit okay?

@qianbinbin qianbinbin reopened this Mar 30, 2023
@qianbinbin
Copy link
Contributor Author

I tried different versions of curl,

when downloading from gitlab.com they behaves differently:

❯ /usr/bin/curl -V
curl 7.64.1 (x86_64-apple-darwin19.0) libcurl/7.64.1 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.39.2
Release-Date: 2019-03-27
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets
❯ /usr/bin/curl -L https://gitlab.com/AOMediaCodec/SVT-AV1/-/archive/v1.4.1/SVT-AV1-v1.4.1.tar.bz2 | sha256sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 9222k    0 9222k    0     0   489k      0 --:--:--  0:00:18 --:--:--  706k
2ddef549e1eaeecc1fc48f0d8332ea3545809e46509db69beb3a0a4bf19ef906  -

❯ /usr/local/opt/curl/bin/curl -V
curl 8.0.1 (x86_64-apple-darwin19.6.0) libcurl/8.0.1 (SecureTransport) OpenSSL/1.1.1t zlib/1.2.11 brotli/1.0.9 zstd/1.5.4 libidn2/2.3.4 libssh2/1.10.0 nghttp2/1.52.0 librtmp/2.3
Release-Date: 2023-03-20
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd
❯ /usr/local/opt/curl/bin/curl -L https://gitlab.com/AOMediaCodec/SVT-AV1/-/archive/v1.4.1/SVT-AV1-v1.4.1.tar.bz2 | sha256sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 9218k    0 9218k    0     0   344k      0 --:--:--  0:00:26 --:--:--  574k
0e988582f315fe76c909accf5e7f81b975c5bd2b850ee760d8e9fac297f70b5d  -

but for github.com they just behaves the same:

❯ /usr/bin/curl -L https://github.com/Homebrew/homebrew-core/raw/master/Formula/svt-av1.rb | sha256sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
100  1995  100  1995    0     0    227      0  0:00:08  0:00:08 --:--:--   557
a95e6e7106cd49e8478b32b17faa39c3d812d257e474159a83a9880b7b6c6933  -
❯ /usr/local/opt/curl/bin/curl -L https://github.com/Homebrew/homebrew-core/raw/master/Formula/svt-av1.rb | sha256sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
100  1995  100  1995    0     0    536      0  0:00:03  0:00:03 --:--:--  4182
a95e6e7106cd49e8478b32b17faa39c3d812d257e474159a83a9880b7b6c6933  -

Hi @carlocab , is this a problem by curl or gitlab? Thanks in advance.

@carlocab
Copy link
Member

Seems like a configuration issue on your end:

❯ /usr/local/opt/curl/bin/curl -fSsL https://gitlab.com/AOMediaCodec/SVT-AV1/-/archive/v1.4.1/SVT-AV1-v1.4.1.tar.bz2 | sha256sum
2ddef549e1eaeecc1fc48f0d8332ea3545809e46509db69beb3a0a4bf19ef906  -
❯ /usr/bin/curl -fSsL https://gitlab.com/AOMediaCodec/SVT-AV1/-/archive/v1.4.1/SVT-AV1-v1.4.1.tar.bz2 | sha256sum
2ddef549e1eaeecc1fc48f0d8332ea3545809e46509db69beb3a0a4bf19ef906  -

@qianbinbin
Copy link
Contributor Author

qianbinbin commented Mar 30, 2023 •
edited

Edit:

I found that gitlab made different tarballs somehow:

❯ /usr/bin/curl -fsSL https://gitlab.com/AOMediaCodec/SVT-AV1/-/archive/v1.4.1/SVT-AV1-v1.4.1.tar.bz2 >old
❯ /usr/local/opt/curl/bin/curl -fsSL https://gitlab.com/AOMediaCodec/SVT-AV1/-/archive/v1.4.1/SVT-AV1-v1.4.1.tar.bz2 >new
❯ tar xf old
❯ ls
SVT-AV1-v1.4.1
❯ tar xf new
❯ ls # the directory name is totally different!!
SVT-AV1-018276d714ce65d9b586f6205ee016cbd8d5425d
SVT-AV1-v1.4.1
❯ diff SVT-AV1-v1.4.1 SVT-AV1-018276d714ce65d9b586f6205ee016cbd8d5425d -r
❯ echo $? # but the contents are exactly the same
0

For reference: old-and-new.zip

@carlocab
Copy link
Member

No, it is not. The issue you linked is about the generation of gzip archives. The file you are trying to download is a bz2 archive.

The pattern in the hexdump described in the issue doesn't even match the hexdump that you're seeing, precisely because the hexdump has to do with the header of the gzip archive. The format of bz2 headers is different.

@qianbinbin
Copy link
Contributor Author

Thank you @carlocab , sorry I made a mistake, but it still seems a problem of gitlab, what should we do with that?

@carlocab
Copy link
Member

I don't think it's a problem with GitLab. It seems to me more like a configuration problem on your machine.

@qianbinbin
Copy link
Contributor Author

qianbinbin commented Mar 30, 2023 •
edited

No, I can confirm that and I have no reason to waste our time here. I'm stuck by this for hours.

For some reason gitlab returns 2 different files with exactly the same URL, I can reproduce in macOS and 4 different Linux servers.
In my Linux servers, even with the same curl but at different times, gitlab may return different files.

root@vultr:~# curl -L https://gitlab.com/AOMediaCodec/SVT-AV1/-/archive/v1.4.1/SVT-AV1-v1.4.1.tar.bz2 | sha256sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 9222k    0 9222k    0     0  22.0M      0 --:--:-- --:--:-- --:--:-- 22.1M
2ddef549e1eaeecc1fc48f0d8332ea3545809e46509db69beb3a0a4bf19ef906  -
# wait for some time...
root@vultr:~# curl -L https://gitlab.com/AOMediaCodec/SVT-AV1/-/archive/v1.4.1/SVT-AV1-v1.4.1.tar.bz2 | sha256sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 9218k    0 9218k    0     0  1786k      0 --:--:--  0:00:05 --:--:-- 1474k
0e988582f315fe76c909accf5e7f81b975c5bd2b850ee760d8e9fac297f70b5d  -

Maybe I should give up, feel free to close the PR. But I doubt someone may encounter similar problem too.
Thanks for your reply.

@carlocab
Copy link
Member

carlocab commented Mar 30, 2023 •
edited

Ok, then it's probably best to report this upstream to see what they have to say about it. We'd need to do that here before updating the checksum anyway.

@qianbinbin
Copy link
Contributor Author

True that it's not a good idea to simply update the checksum..

@qianbinbin
Copy link
Contributor Author

Follow up: https://forum.gitlab.com/t/different-checksums-for-the-same-archive/84143

@qianbinbin
Copy link
Contributor Author

Closing due to Gitlab's lack of interest.

For now the bug cannot be reproduced, hope nobody would encounter the same problem.

@qianbinbin qianbinbin closed this Mar 31, 2023
@cho-m cho-m mentioned this pull request Apr 2, 2023
Closed
3 tasks
@qianbinbin
Copy link
Contributor Author

The issue should be fixed in https://gitlab.com/gitlab-org/gitlab/-/issues/402616 , brief explanation from a maintainer of Gitlab:

The root cause was discussed in a confidential issue, but I can share a general idea what went wrong.

One of deployed fixes had a side-effect of changing the hash sum of the archived files.
When it was discovered, we reverted the code.
However, Cloudflare still responded with old archive files for some regions.
We updated the cache and that resolved the problem.

But frotz still seems affected, see #127298

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
checksum mismatch SHA-256 doesn't match the download
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@qianbinbin @carlocab