migrate bootstrapper away from RSA

[marten-seemann]

Checklist

• My issue is specific & actionable.
• I am not suggesting a protocol enhancement.
• I have searched on the issue tracker for my issue.

Description

After benchmarking libp2p handshakes, it turns out that verifying signatures is roughly equally fast for RSA and for Ed25519 keys. Where performance differs significantly is while signing. For CPU performance, it therefore doesn't really matter which key type your peer uses, it only matter which key type you use.

The conclusion can be summarized: If you care about your CPU load, you really don't want to use an RSA host key.
This is why generating Ed25519 on ipfs init makes sense, and adding a warning to nudge people to migrate away from RSA (#9506) does too.

libp2p planning to drop OpenSSL support

OpenSSL can be used to speed up RSA operations. There's some numbers in libp2p/go-libp2p#1686 (comment), due to a bug (libp2p/go-libp2p#1951) only the number for signing are valid.

While OpenSSL helps bring down the cost of RSA signing from 1.2ms to 0.5ms, it is still vastly more expensive than Ed25519 signing, which sits at around 22µs.

libp2p is planning to drop OpenSSL support soon (libp2p/go-libp2p#1952), since it has been creating a disproportionate amount of maintenance burden with unclear benefit. As argued above, the only benefit is that it provides is slightly reducing the amount of self-inflicted pain from using RSA keys.

Bootstrappers

The only problem with this advice ("just don't use RSA keys) is that Kubo has hardcoded peer IDs of bootstrappers, and those still use RSA keys for historic reasons. As the IPFS network is notoriously slow to upgrade to new Kubo versions, at least some of these nodes will probably need to keep running for years. As libp2p is dropping OpenSSL support, this means that these bootstrappers would not be able to update to Kubo versions that use those newer go-libp2p versions.

This problem does not occur with Hydra nodes, as those don't use hard-coded peer IDs (as far as I can tell), which should make it trivial to switch them to Ed25519, in case they were deployed with RSA.

Assuming that handshakes are a significant component of the CPU load of bootstrappers , significant savings of CPU time can be expected from switching them to Ed25519.
I don't have access to them, but it should be easy to confirm or disprove by getting a pprof. I took some pprofs from a Kubo node that I set up which runs the Accelerated DHT client, and it looks like other code paths are still more expensive, but that might not translate well to what the bootstrappers are doing.
It might be interesting to convert these CPU savings into $$$ by scaling down the hardware these nodes run on.

I'm opening this issue to explore a transition path to use a new set of bootstrap nodes that posses Ed25519 keys. One option I can think of would be to locate them on a new subdomain. We need to pay attention to keeping the DNS message size small due to libp2p/go-libp2p#1837.
Arguably, IPFS bootstrappers should live on bootstrap.ipfs.io, not bootstrap.libp2p.io. New versions of Kubo would then (exclusively?) ship the list of new bootstrap nodes, which would reduce the load on the old bootstrappers as nodes upgrade. This would allow us to gradually scale down the hardware they run on, and decommission them eventually, probably after a few years.

[lidel]

• Adjusting default bootstrappers in new Kubo releases sounds doable, we can write config migration that switches preexisting users to ed25519 ones too (like we did for /quic → /quic-v1 in 0.18)
• Another way of decreasing bootstraper load is decreasing reliance on them: Saving previously seen nodes for later bootstrapping #3926
• When it comes to a domain, sgtm too, but I'd like to use something other than ipfs.io as this domain is tainted by Gateway feature and under risk of being blocked due to malware/phishing.

[marten-seemann]

Adjusting default bootstrappers in new Kubo releases sounds doable, we can write config migration that switches preexisting users to ed25519 ones too (like we did for /quic → /quic-v1 in 0.18)

Great, thank you for confirming.

Another way of decreasing bootstraper load is decreasing reliance on them: #3926

That's a good idea, but it won't help with this issue, right? It requires people to upgrade their nodes, so it doesn't reduce the load on the RSA bootstrappers being hit by legacy nodes.

---

If I understand correctly, all that's would be needed is spinning up a few ed25519-powered bootstrap nodes, and a config migration. That sounds like go-libp2p could drop OpenSSL support in the next release, without causing too many problems? Given the recently discovered bugs (this and this one, specifically) in the implementation, we'd like to get rid of this code path asap.

[BigLep]

@marten-seemann : I'm catching up on the conversation here, in libp2p/go-libp2p#1952, and in FIL slack.

A couple things:

1. General statement: I'm fully supportive of the push for go-libp2p to reduce its maintenance burden.
2. If go-libp2p drops OpenSSL in v0.25 (next go-libp2p version), Kubo 0.19 (next version) uses go-libp2p v0.25, and the legacy hardcoded bootstrappers are deployed with Kubo 0.19, is the only ramification some CPU hit on the bootstrapers? Everything will still work but there will be some minor performance degradation? Is that right? (I think this is the same question as what @hsanjuan is asking in OpenSSL vs. BoringSSL vs. Go standard library crypto libp2p/go-libp2p#1952 (comment) . ).

If everything will still work, then this seems like an easy decision. IPFS/Kubo/Bifrost can handle migrating the bootstrappers away from RSA on a separate timetable. (I'm supportive of the migration from RSA, but it would be nice to have as little coupling as possible here.)

[marten-seemann]

2. If go-libp2p drops OpenSSL in v0.25 (next go-libp2p version), Kubo 0.19 (next version) uses go-libp2p v0.25, and the legacy hardcoded bootstrappers are deployed with Kubo 0.19, is the only ramification some CPU hit on the bootstrapers? Everything will still work but there will be some minor performance degradation? Is that right? (I think this is the same question as what @hsanjuan is asking in OpenSSL vs. BoringSSL vs. Go standard library crypto libp2p/go-libp2p#1952 (comment) . ).

Correct.
I'm still wondering where the narrative "OpenSSL is crucial for our performance" came from in the first place, but the recently collected profiles from Mars suggest that the performance impact will be minimal, if at all.

If everything will still work, then this seems like an easy decision. IPFS/Kubo/Bifrost can handle migrating the bootstrappers away from RSA on a separate timetable. (I'm supportive of the migration from RSA, but it would be nice to have as little coupling as possible here.)

Agreed, looking at the new profiles, it seems like dropping OpenSSL won't necessitate migrating to new bootstrappers. Migrating away from RSA (at the very least, diversifying the bootstrapper setup) is probably still a good idea.