Port to Hyper 1.0

[slinkydeveloper]

Tasks

Beta Give feedback

1. Investigate feasibility of Hyper 1.0 usage for invoker #100
   invoker +1
   
2. Wait for hyper 1.0 to be released
3. Wait for hyper-rustls to support hyper 1.0
4. Port invoker to hyper 1.0 #1060
   +0
5. Port admin API to hyper 1.0 #1727
   +0
   
6. Port ingress to hyper 1.0 #1059
   +0
   
7. Bump Lambda smithy runtime in service-client #1729
   +0
   
8. Bump Otel to latest version #1730
   +0
   
9. Bump hyper 1.0 in CLI tunnel and cloud login code #1731
   +0
   
10. Bump metadata-store to latest hyper/tonic/tower/http crates #1732
    +0
    
11. Bump benchmarks to hyper 1.0 #1737
    +0
    
12. Bump restatectl, restate-admin, restate-core, restate-types, restate-service-protocol to use latest prost and tonic dependencies #1745
    +0
    

[slinkydeveloper]

When porting to hyper 1.0, we should remove the task spawn within the InvocationTask that drives the connection forward.

[slinkydeveloper]

Also need to keep track of rustls/hyper-rustls#192

[slinkydeveloper]

Very relevant PR here: hyperium/hyper#3155

[slinkydeveloper]

With version 0.14.25 we can start using the new 1.0 APIs, which hopefully will make the transition smoother: https://github.com/hyperium/hyper/releases/tag/v0.14.25

[slinkydeveloper]

hyper 1.0 has been released, and hyper-util as well. We need the hyper-rustls and then we can start

[tillrohrmann]

In my latest performance measurements, I noticed that the main bottleneck seems to be the invoker currently (the ingress is also causing high CPU load but does not seem be the limiting factor). One thing I noticed in the flamegraphs is that hyper requires a lot of CPU cycles when polling the Body. I haven't found the reason for this problem but it could be that there is some contention on the single TCP connection that the invoker currently uses when talking to the service deployment.

I hope that by upgrading to Hyper 1.1 we will manage to remove the bottleneck from the server.

[slinkydeveloper]

To upgrade to hyper 1.0 we still need hyper-rustls to be released with rustls/hyper-rustls#232

[tillrohrmann]

completed 🎉