rust-analyzer failing with a bad hash

[spikespaz]

Not sure why, but rust-lang.rust-analyzer fails with a bad hash.

error: hash mismatch in fixed-output derivation '/nix/store/1magwzvshwqzxv4npv0m185jfxczmxqn-rust-analyzer-0.4.1381.zip.drv':
         specified: sha256-4+fS2fKhkYuPWfp1nrHwkNK7XU9kvs0iB/RtAzBH0sY=
            got:    sha256-z9pcu0rVMuwYJpJrI1l5YaT+TxlQo762j8RP+MGW2kQ=
error: 1 dependencies of derivation '/nix/store/klyv1hqbx1d4ym4qxidsl8j9352kv2fv-vscode-extension-rust-lang-rust-analyzer-0.4.1381.drv' failed to build

I would like to know why the hash is wrong in this repository, I don't know how the automation/crawler works, but is it even possible for the hash to change on the VSCode Marketplace?

[deemp]

Hi, @spikespaz !
The crawler uses nix store prefetch-file extension_url (see this line). You may see the extension_url in the generated nix files. E.g., in data/generated/vscode-marketplace.nix

BTW, I checked with the latest version and it worked

nix-repl> :lf github:nix-community/nix-vscode-extensions/56efe16c813eef6328812856a704a41af7645925
Added 14 variables.
nix-repl> :b extensions.x86_64-linux.vscode.rust-lang.rust-analyzer

This derivation produced the following outputs:
  out -> /nix/store/alawnirlr33pl73yrylacpf6x7cyvvh2-vscode-extension-rust-lang-rust-analyzer-0.4.1385

[AmeerTaweel]

We had this problem before. It seems that the API is not always deterministic. I was able to get different hashes for the same version of the same extension on the same machine after trying to request it enough times.

[spikespaz]

@AmeerTaweel Can the generator be adjusted to verify the hashes to prevent this mistake from affecting the end-user? I don't think it is a great thing to require the user to write correcting overlays for the faulty fetchers.

[AmeerTaweel]

@spikespaz the problem is not with the fetcher. The API does not always provide the same file. So even if you write an overlay with the hash you are getting now, you might still get a file with another hash later.

And in the same way, even if the generator verified the hash and it was correct, you might still get a wrong hash later.

As mentioned in the issue in the original repo, we used curl to fetch files from the API and checked the hashes manually. So the issue is with the API itself.

[anilanar]

Curious, I've been using vscode with nix for years and had my own list of extensions that I maintained and used with vscode-utils.extensionFromVscodeMarketplace. Now I decided to switch to using nix-vscode-extensions and I get mismatching hash the second day.

I guess I was just lucky before?

[twz123]

I also noticed some problems when I tried to fetch redhat-java via nix from the VSCode marketplace some time ago. In my case, I sometimes got the Linux build of the extension, sometimes I got the OSX build when using the exact same URL. That made me think that maybe only those extensions are affected that have different assets for different architectures, and there's some query parameter or HTTP header involved that needs to be set? E.g. to select the architecture? The absence of that parameter could mean "don't care"... (In my case case, I've opted to use the assets from the GitHub releases page instead of the marketplace.)

[anilanar]

I also noticed some problems when I tried to fetch redhat-java via nix from the VSCode marketplace some time ago. In my case, I sometimes got the Linux build of the extension, sometimes I got the OSX build when using the exact same URL. That made me think that maybe only those extensions are affected that have different assets for different architectures, and there's some query parameter or HTTP header involved that needs to be set? E.g. to select the architecture? The absence of that parameter could mean "don't care"... (In my case case, I've opted to use the assets from the GitHub releases page instead of the marketplace.)

Could that be targetPlatform query param used by vscode-utils.extensionFromVscodeMarketplace?

That would explain why vscode-utils.extensionFromVscodeMarketplace never caused hash mismatch for me in the past.

[AmeerTaweel]

I did some experiments and it seems true.

When I run the following:

$ curl "https://redhat.gallery.vsassets.io/_apis/public/gallery/publisher/redhat/extension/vscode-yaml/1.10.20220805/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage" -o extension.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1108k  100 1108k    0     0  1402k      0 --:--:-- --:--:-- --:--:-- 1403k
$ nix hash to-sri --type sha256 $(nix-hash --flat --base32 --type sha256 extension.zip)
sha256-Q1R+2t/tWI3haoF5fW7k2413BLIMf1ImM4J9CUrtjnQ=

Then I tried different types of architectures, and this is the one that matched:

$ curl "https://redhat.gallery.vsassets.io/_apis/public/gallery/publisher/redhat/extension/vscode-yaml/1.10.20220805/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage?targetPlatform=alpine-arm64" -o extension.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1108k  100 1108k    0     0  1319k      0 --:--:-- --:--:-- --:--:-- 1319k
$ nix hash to-sri --type sha256 $(nix-hash --flat --base32 --type sha256 extension.zip)
sha256-Q1R+2t/tWI3haoF5fW7k2413BLIMf1ImM4J9CUrtjnQ=

So I was getting the alpine-arm64 version by default. However, I can obtain different versions at will now:

$ curl "https://redhat.gallery.vsassets.io/_apis/public/gallery/publisher/redhat/extension/vscode-yaml/1.10.20220805/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage?targetPlatform=win32-arm64" -o extension.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1108k  100 1108k    0     0  1042k      0  0:00:01  0:00:01 --:--:-- 1043k
$ nix hash to-sri --type sha256 $(nix-hash --flat --base32 --type sha256 extension.zip)
sha256-1aEyB3BTe8S0OGpk3JJLi6Zq6gQLPacOk7Om5suYTr8=

[AmeerTaweel]

Interestingly, when I ask for the darwin-x64 version or the darwin-arm64 version, I get an error saying that there is no support for the requested target platform.

So we need to put some thought into this to fix our fetcher:

• Like how do we know if an extension has platform-specific versions.
• And what should we do if there are platform-specific versions but we don't find one for the current platform.

[AmeerTaweel]

Also, here is a list of supported target platforms if it helps anyone.

[twz123]

The JSON API to find out version/targetPlatform combinations:

cat <<'EOF' | curl -sSL -H 'Accept: application/json;api-version=6.1-preview.1' -H 'Content-Type: application/json' https://marketplace.visualstudio.com/_apis/public/gallery/extensionquery -d @- | jq --raw-output '.results[0].extensions[0].versions[] | .version  + " " + .targetPlatform'
{
    "assetTypes": [],
    "filters": [
        {
            "criteria": [
                {
                    "filterType": 8,
                    "value": "Microsoft.VisualStudio.Code"
                },
                {
                    "filterType": 7,
                    "value": "redhat.vscode-yaml"
                },
                {
                    "filterType": 12,
                    "value": "4096"
                }
            ],
            "pageNumber": 1,
            "pageSize": 2
        }
    ],
    "flags": 946
}
EOF

yields

1.11.10112022
1.10.20220805 alpine-x64
1.10.20220805 alpine-arm64
1.10.20220805 linux-armhf
1.10.20220805 win32-arm64
1.10.20220805 win32-ia32

[AmeerTaweel]

@twz123 thanks for your help!

Correct me if I'm wrong. Are they using alpine and linux interchangeably?

[twz123]

I'd guess that alpine ships (probably statically linked) musl binaries, whereas Linux ships usual glibc binaries.

[spikespaz]

Any news on getting that extra query information working so the hashes can match?

[ejpcmac]

Hello, I am facing this issue today, on rust-lang.rust-analyzer as well. Other extensions have the proper hash.

[deemp]

Hmm, this works for me

nix-repl> :lf github:nix-community/nix-vscode-extensions/d5a069fccf424791a45722805f5d99cbbe1ae264
Added 15 variables.

nix-repl> :b extensions.x86_64-linux.vscode-marketplace.rust-lang.rust-analyzer                   

This derivation produced the following outputs:
  out -> /nix/store/63ihxcwcaj7s4wra8z27wwai55r5cgd8-vscode-extension-rust-lang-rust-analyzer-0.4.1505

[anilanar]

@deemp Because it's random.

[ejpcmac]

Indeed, I’m now able to build rust-lang.rust-analyzer without updating nix-vscode-extensions. This is annoying that the VSCode Marketplaces has a non-deterministic API. Has it been reported upstream?

[deemp]

I suggest we continue this discussion in #20