incorrectly reported signature validation failure (Rust 1.8.0 to Rust 1.21.0, some nightlies)

[ChaiTRex]

Problem

Older versions of Rust (on x86_64-unknown-linux-gnu) report a signature verification failure, so I cannot trust them, so I can't check which of those versions of Rust my code works on.

Steps

1. rustup toolchain install 1.8

Possible Solution(s)

No response

Notes

This applies to more than just 1.8, but here is the output for 1.8, which I stopped early.

$ rustup toolchain install 1.8
info: syncing channel updates for '1.8.0-x86_64-unknown-linux-gnu'
 65.3 KiB /  65.3 KiB (100 %)  20.0 KiB/s in  8s ETA:  0s
warning: Signature verification failed for 'https://static.rust-lang.org/dist/channel-rust-1.8.0.toml'
info: latest update on 2016-04-12, rust version 1.8.0 (db2939409 2016-04-11)
info: downloading component 'cargo'
167.5 KiB /   3.9 MiB (  4 %)   0 B/s in  2s ETA: Unknown^C

Rustup version

rustup 1.25.2 (17db695f1 2023-02-01)

Installed toolchains

Default host: x86_64-unknown-linux-gnu
rustup home:  /home/chaitrex/.rustup

installed toolchains
--------------------

stable-x86_64-unknown-linux-gnu (default)
beta-x86_64-unknown-linux-gnu
nightly-x86_64-unknown-linux-gnu
1.0.0-x86_64-unknown-linux-gnu
1.1.0-x86_64-unknown-linux-gnu
1.2.0-x86_64-unknown-linux-gnu
1.3.0-x86_64-unknown-linux-gnu
1.4.0-x86_64-unknown-linux-gnu
1.5.0-x86_64-unknown-linux-gnu
1.6.0-x86_64-unknown-linux-gnu
1.7.0-x86_64-unknown-linux-gnu

active toolchain
----------------

stable-x86_64-unknown-linux-gnu (default)
rustc 1.67.1 (d5a82bbd2 2023-02-07)

[pietroalbini]

The actual signatures do match, this is yet another problem of the pgp library used by rustup rejecting still valid signatures (like the issue that prompted rustup 1.25.2 to be released).

You can manually verify it matches with:

curl https://raw.githubusercontent.com/rust-lang/rustup/master/src/rust-key.pgp.ascii | gpg --import
curl -OL https://static.rust-lang.org/dist/channel-rust-1.8.0.toml
curl -OL https://static.rust-lang.org/dist/channel-rust-1.8.0.toml.asc
gpg --verify channel-rust-1.8.0.asc

[pietroalbini]

rustup mistakenly says signatures are not verified from Rust 1.8.0 to Rust 1.20.0.

[ChaiTRex]

Found out that 1.14 is fine and 1.21 is not.

Failures list

warning: Signature verification failed for 'https://static.rust-lang.org/dist/channel-rust-1.8.0.toml'
warning: Signature verification failed for 'https://static.rust-lang.org/dist/channel-rust-1.9.toml'
warning: Signature verification failed for 'https://static.rust-lang.org/dist/channel-rust-1.10.toml'
warning: Signature verification failed for 'https://static.rust-lang.org/dist/channel-rust-1.11.toml'
warning: Signature verification failed for 'https://static.rust-lang.org/dist/channel-rust-1.12.toml'
warning: Signature verification failed for 'https://static.rust-lang.org/dist/channel-rust-1.13.toml'
warning: Signature verification failed for 'https://static.rust-lang.org/dist/channel-rust-1.15.toml'
warning: Signature verification failed for 'https://static.rust-lang.org/dist/channel-rust-1.16.toml'
warning: Signature verification failed for 'https://static.rust-lang.org/dist/channel-rust-1.17.toml'
warning: Signature verification failed for 'https://static.rust-lang.org/dist/channel-rust-1.18.toml'
warning: Signature verification failed for 'https://static.rust-lang.org/dist/channel-rust-1.19.toml'
warning: Signature verification failed for 'https://static.rust-lang.org/dist/channel-rust-1.20.toml'
warning: Signature verification failed for 'https://static.rust-lang.org/dist/channel-rust-1.21.toml'

[dcow]

This is happening on 1.67.1 and 1.69.0-nightly for me so I don't think it's limited to 1.8 to 1.20.

❯ rustup update
info: syncing channel updates for 'stable-aarch64-apple-darwin'
warning: Signature verification failed for 'https://static.rust-lang.org/dist/channel-rust-stable.toml'
info: latest update on 2023-02-09, rust version 1.67.1 (d5a82bbd2 2023-02-07)
...
info: syncing channel updates for 'nightly-aarch64-apple-darwin'
warning: Signature verification failed for 'https://static.rust-lang.org/dist/channel-rust-nightly.toml'
info: latest update on 2023-03-02, rust version 1.69.0-nightly (f77bfb733 2023-03-01)

Just today on macOS.

❯ rustup self update
info: checking for self-updates
  rustup unchanged - 1.25.2

I'm surprised this was just a warning and it didn't bail out and require me to manually override. Seems like if this was an actual mismatch, I could be in some trouble and have just downloaded untrusted/unverified code. I do understand that TLS mitigates this; I just kinda expect that if a project does elect to use the extra layer of signature checks, that they serve their purpose and aren't mere theatre. I could be misunderstanding this warning though.

[rbtcollins]

We haven't been confident enough in the system in place to promote the signature validation from warning to blocking. There's also been discussion about TUF but I don't think anyone is actively pushing on those concerns currently.

[mberry]

Getting this warning updating to 1.68:

info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
warning: Signature verification failed for 'https://static.rust-lang.org/dist/channel-rust-stable.toml'
info: latest update on 2023-03-09, rust version 1.68.0 (2c8cc3432 2023-03-06)

Agree with @dcow on this that the default behaviour is surprising. The enduser could always be prompted to use --force to continue with installation.

[nwalfield]

this is yet another problem of the pgp library used by rustup rejecting still valid signatures

@pietroalbini, I'd appreciate it if you could describe what valid signatures sequoia is rejecting so that we can fix sequoia.

(like the issue that prompted rustup 1.25.2 to be released).

Can you explain what you mean here. I'm a bit confused about the comparison to the 1.25.2 issue: My recollection of that issue is that the security of the certificate used to create the signatures relied on SHA-1. SHA-1 is broken, so those signatures were not valid. Are you saying those signatures were (are?) valid?

[rbtcollins]

@nwalfield
rustup toolchain install --force-non-host 1.8.0-x86_64-unknown-linux-gnu reproduces the reported problem

I suspect we have multiple different cases though, as the reported nightly issue above doesn't reproduce for me:
rustup toolchain install --force-non-host nightly-2023-03-02-aarch64-apple-darwin

I'm not sure what infra's take on the current situation is.

Mine (a rustup team-lead) is: we don't currently depend on the PGP signature validity for our security: our primary security mechanism today is transport security and trusted mirrors. We have stalled work in progress towards TUF or some such thing, and we have a PGP validation layer that is warn-only (though users can opt into strict checks).

I would like us to have PGP or TUF or some similar form of transport-independent verification of the distributions we're installing, because the sheer number of corporate MITM proxies makes depending solely on transport security iffy. There are a number of things we likely need to do to get to that point, though what the minimum work is is currently unclear.

HTH

[hi-rustin]

I suspect we have multiple different cases though, as the reported nightly issue above doesn't reproduce for me:
rustup toolchain install --force-non-host nightly-2023-03-02-aarch64-apple-darwin

I can not reproduce it on my M1 Macbook Pro too.

[rbtcollins]

Discussion on zulip and discord seems to have consensus that there is no harm, and some benefit just removing the gpg code for now.

• the foundation's new security engineer Walter Pearce is working on a new system, not based around GPG, for validation of distributions
• we don't rely on the signatures today - these warnings are not errors by default
• sustained ignored, unfixable signature errors will teach folk to ignore them, which is harmful to everyone
• we could do streaming unpacking (with some changes) if we trust the transport rather than the current monolithic signature validation, which could improve performance

I think (will have to check) we have a regular checksum in the channels that we can use to detect bit-errors and short-reads without GPG. So this should be fairly easy, though its not going to be a super-tiny patch.

The other detail to consider is users that have opted into strict gpg validation. Should we make their config files error (to surface their config not applying anymore), or just ignore the key? I think erroring is marginally better, but welcome input!

[nwalfield]

The signature for the 1.8.0 channel uses SHA-1:

$ wget -q -O- https://static.rust-lang.org/dist/channel-rust-1.8.0.toml.asc | sq packet dump
Signature Packet, old CTB, 540 bytes
    Version: 4
    Type: Binary
    Pk algo: RSA
    Hash algo: SHA1
    Hashed area:
      Signature creation time: 2016-04-12 08:50:58 UTC
    Unhashed area:
      Issuer: 5CB4A9347B3B09DC
    Digest prefix: 6DAA
    Level: 0 (signature over data)

As @pietroalbini points out, gpg accept this:

$ gpg --version
gpg (GnuPG) 2.2.40
...
$ gpg  --verify channel-rust-1.8.0.toml.asc channel-rust-1.8.0.toml
gpg: Signature made Tue 12 Apr 2016 10:50:58 AM CEST
gpg:                using RSA key 0x5CB4A9347B3B09DC
gpg: Good signature from "Rust Language (Tag and Release Signing Key) <rust-key@rust-lang.org>" [unknown]

That's because gpg allows SHA-1 by default. gpg can be set to reject signatures that rely on SHA-1 by using the --weak-digest algorithm. It then also rejects the signature:

$ gpg --weak-digest SHA1 --verify channel-rust-1.8.0.toml.asc channel-rust-1.8.0.toml
gpg: Signature made Tue 12 Apr 2016 10:50:58 AM CEST
gpg:                using RSA key 0x5CB4A9347B3B09DC
gpg: Note: signatures using the SHA1 algorithm are rejected
gpg: Can't check signature: Invalid digest algorithm

FWIW, I also can't reproduce the nightly issue either. And, looking at the signature, it seems fine:

$ wget -q -O- https://static.rust-lang.org/dist/2023-03-02/channel-rust-nightly.toml.asc | sq packet dump
Signature Packet, new CTB, 540 bytes
    Version: 4
    Type: Binary
    Pk algo: RSA
    Hash algo: SHA512
    Hashed area:
      Signature creation time: 2023-03-02 00:41:18 UTC
      Issuer: 85AB96E6FA1BE5FE
    Digest prefix: 4C03
    Level: 0 (signature over data)
  

@pietroalbini says:

The actual signatures do match, this is yet another problem of the pgp library used by rustup rejecting still valid signatures (like the issue that prompted rustup 1.25.2 to be released).

Right now, I don't see any evidence that the pgp library used by rustup is rejecting still valid signatures. The policy explicitly opted into by rustup is to reject signatures over data that use SHA-1. If you want something else, you need to configure something else.

That said, I'd strongly recommend not allowing the use of SHA-1 for signatures. To address this specific issue, my recommendation is to resign the files that use signatures that rely on SHA-1.

It appears that at least the following signatures are affected:

$ for i in $(seq 1 68); do if wget -q -O- https://static.rust-lang.org/dist/channel-rust-1.$i.0.toml.asc | sq packet dump 2>/dev/null | grep -q SHA1; then echo channel-rust-1.$i.0.toml.asc uses SHA1; fi; done
channel-rust-1.8.0.toml.asc uses SHA1
channel-rust-1.9.0.toml.asc uses SHA1
channel-rust-1.10.0.toml.asc uses SHA1
channel-rust-1.11.0.toml.asc uses SHA1
channel-rust-1.12.0.toml.asc uses SHA1
channel-rust-1.13.0.toml.asc uses SHA1
channel-rust-1.15.0.toml.asc uses SHA1
channel-rust-1.16.0.toml.asc uses SHA1
channel-rust-1.17.0.toml.asc uses SHA1
channel-rust-1.18.0.toml.asc uses SHA1
channel-rust-1.19.0.toml.asc uses SHA1
channel-rust-1.20.0.toml.asc uses SHA1
channel-rust-1.21.0.toml.asc uses SHA1

[pietroalbini]

The other detail to consider is users that have opted into strict gpg validation. Should we make their config files error (to surface their config not applying anymore), or just ignore the key? I think erroring is marginally better, but welcome input!

Is that actually possible?

[rbtcollins]

The other detail to consider is users that have opted into strict gpg validation.

Is that actually possible?

No, my fleshy ChatGPT-4 implementation was hallucinating :) . Daniel and I discussed it at one point I think.

[hi-rustin]

@rbtcollins @pietroalbini Is there any problem with accepting sha-1 completely for now, as opposed to removing the entire signature check?

[pietroalbini]

I don't think keeping gpg support while accepting sha-1 has any value tbh.