SHA256 hash mismatches (deno cache and curl | sha256sum disagree)

[fasterthanlime]

This is kind of a weird problem. When nix building one of my projects with deno2nix, I get errors like:

$ nix build '.#executable'
warning: Git tree '/Users/amos/bearcove/keysmash2' is dirty
error: hash mismatch in file downloaded from 'https://esm.sh/preact@10.11.0/hooks':
         specified: sha256:0dpp3ypl1j86v8whp3yqlx7i3aam3pzsqx7hccb511wbxdaw33ib
         got:       sha256:049xy78c6l4i6c6cwppgpgyrilcz2n78ydvjppv8rr8gifklzz4n
(use '--show-trace' to show detailed location information)

I'm not sure how those hashes are formatted (they don't look like hex, there's w in there), but out of curiosity, I checked what was in deno.lock and compared it with curl url | gsha256sum, and they disagree:

The deno.lock file has 2b8ec155eb8b87501663f074acff1d55a9114fa7d88f0b39da06c940af1ff736, but:

$ curl -s https://esm.sh/preact@10.11.0/hooks | gsha256sum
96fc4fa78b0fe58cf6bd72378f8e159fd198fdbbef5ece0c339150c3d0f13d11  -

This might be a bug in deno cache (1.30.3), or esm.sh might be serving something different for different user-agents (and in this case it can be fixed in deno2nix?). All I know right now is that it's weird and I'm wondering if you have any idea where this could come from.

(My best idea of a workaround right now is to write a tool that updates deno.lock with the proper SHA256 sums before calling nix build, but that's not great).

[vielmetti]

There is a (lengthy) discussion of hash mismatches here

NixOS/nix#1885

which might be useful (and it's certainly extensive enough to suggest you're not the only one).

[fasterthanlime]

It looks like esm.sh returns something different based on the User-Agent, so deno2nix should set it: https://mspsocial.net/users/ironiridis/statuses/109866537157919405

[ironiridis]

@fasterthanlime Note that my instance doesn't permit unauthenticated access due to an upstream Mastodon change so here's the text of that post:

You were right, it checks the UA. See the ESBuild Options section on this page: https://esm.sh/#docs

Luckily you can override it.

curl -s 'https://esm.sh/preact@10.11.0/hooks?target=deno' | sha256sum
2b8ec155eb8b87501663f074acff1d55a9114fa7d88f0b39da06c940af1ff736 -

This is in the context of Amos's post.

[fasterthanlime]

I tried passing curlOpts with a User-Agent: Deno/1.0 (which in my local testing should give the right hash) in this branch but I'm now stuck on another nix-related error that may be another bug entirely / me holding it wrong:

nix build --show-trace '.#executable'
warning: Git tree '/Users/amos/bearcove/keysmash2' is dirty
error: builder for '/nix/store/ly08n3gzfqcxy6akl5ci0l3s8hplsgdw--preact-render-to-string-5.2.4.drv' failed to produce output path for output 'out' at '/nix/store/bdfx0mf5i8hf1axk832f42wvhn8krji8--preact-render-to-string-5.2.4'
error: 1 dependencies of derivation '/nix/store/qvq7hwmvcq1yg49pyf1d0gidi3d5v154-deps.drv' failed to build
error: 1 dependencies of derivation '/nix/store/yb0n6sg94cpi68m2mdyh14aibzpv3wfh-simple-executable-0.1.0.drv' failed to build

[fasterthanlime]

Oh, this is a duplicate of #6! Closing.

[voidcontext]

I tried passing curlOpts with a User-Agent: Deno/1.0 (which in my local testing should give the right hash) in this branch but I'm now stuck on another nix-related error that may be another bug entirely / me holding it wrong:

nix build --show-trace '.#executable'
warning: Git tree '/Users/amos/bearcove/keysmash2' is dirty
error: builder for '/nix/store/ly08n3gzfqcxy6akl5ci0l3s8hplsgdw--preact-render-to-string-5.2.4.drv' failed to produce output path for output 'out' at '/nix/store/bdfx0mf5i8hf1axk832f42wvhn8krji8--preact-render-to-string-5.2.4'
error: 1 dependencies of derivation '/nix/store/qvq7hwmvcq1yg49pyf1d0gidi3d5v154-deps.drv' failed to build
error: 1 dependencies of derivation '/nix/store/yb0n6sg94cpi68m2mdyh14aibzpv3wfh-simple-executable-0.1.0.drv' failed to build

@fasterthanlime The reason it fails is because curlOpts doesn't like spaces, you should use curlOptsList = [ "-H" "User-Agent: Deno/1.0" ];. Unfortunately after this we're back to the checksum issue. No matter what I use: nix-prefetch-url, curl + target=deno + sha256sum, curl + deno user agent + sha256, I am just not able to get the checksum that is in deno.lock.

My changes: voidcontext@3f2577b

[ironiridis]

@voidcontext Can you try -A instead? https://everything.curl.dev/http/requests/user-agent

[voidcontext]

@voidcontext Can you try -A instead? https://everything.curl.dev/http/requests/user-agent

@ironiridis Yes, I can try that a little bit later, but the problem is this:

❯ curl -s 'https://esm.sh/v90/graphql@16.5.0' -A "Deno/1.0"| sha256sum 
8fb8d3d42518420220907239e5eb6b4c822d18638051afe734016e080fda54c8  -

❯ curl -s 'https://esm.sh/v90/graphql@16.5.0' -H "User-Agent: Deno/1.0"| sha256sum
8fb8d3d42518420220907239e5eb6b4c822d18638051afe734016e080fda54c8  -

❯ curl -s 'https://esm.sh/v90/graphql@16.5.0?target=deno' | sha256sum
8fb8d3d42518420220907239e5eb6b4c822d18638051afe734016e080fda54c8  -

❯ curl -s 'https://esm.sh/v90/graphql@16.5.0' | sha256sum 
c351b3f437d7882de2bd8c2baa92d9e0f4a9405f3dbae8a435e831a185f1a45e  -

And tha sha256sum in the lock file is c7decbea396688889b13d21aa0859a5df441e1ab86280ec98c480ff1e8d85f7d

[voidcontext]

@voidcontext Can you try -A instead? https://everything.curl.dev/http/requests/user-agent

@ironiridis yup, same issue with this: voidcontext@ab9ac6e

[stepbrobd]

curlOptsList = [ "-H" "User-Agent: Deno/1.0" ];

Deno/1.0 UA still causes hash mismatch, I changed it to Deno/${deno.version} and everything seems working now