Running rustls example results in panic on the server side #1183

andreiltd · 2022-12-09T19:51:34Z

Bug Report

Version

master

Platform

Linux pop-os 6.0.6-76060006-generic x86_64

Description

Running tls_rustls example results in panic on the server side:

server: cargo run --bin tls-server-rustls
client: cargo run --bin tls-client-rustls

then on server:

Got a request from: [::1]:59296 with certs: []
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: hyper::Error(Io, Kind(UnexpectedEof))', examples/src/tls/server_rustls.rs:88:52
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Expected behavior: example is working without errors.

The text was updated successfully, but these errors were encountered:

LucioFranco · 2022-12-12T19:08:53Z

It works for me locally on my mac so not sure why its failing for you

server:

     Running `target/debug/tls-server-rustls`                                                                                                                                         │
Got a request from: [::1]:60564 with certs: []

client:

RESPONSE=Response { metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Mon, 12 Dec 2022 19:07:24 GMT", "grpc-status": "0"} }, message: EchoResponse { message: "hello" }, extensions: Extensions }

LucioFranco · 2022-12-12T19:09:13Z

You could try enabling logging to see if it tells you anything.

andreiltd · 2022-12-12T19:14:15Z

Hey! Thanks for the reply, I can retry with logs enabled. The other problem I had is that the list of client certs is empty: with certs: [], shouldn't this contain client certificates?

andreiltd · 2022-12-12T19:19:39Z

Here is the output log, but I don't see any obvious problem 🤔

     Running `target/debug/tls-server-rustls`
2022-12-12T19:17:43.697143Z DEBUG rustls::server::hs: decided upon suite TLS13_AES_256_GCM_SHA384    
2022-12-12T19:17:43.697577Z DEBUG rustls::server::hs: Chosen ALPN protocol [104, 50]    
2022-12-12T19:17:43.699470Z DEBUG h2::codec::framed_write: send frame=Settings { flags: (0x0), initial_window_size: 1048576, max_frame_size: 16384, max_header_list_size: 16777216 }
2022-12-12T19:17:43.699741Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Settings { flags: (0x0), enable_push: 0, initial_window_size: 2097152, max_frame_size: 16384 }
2022-12-12T19:17:43.699770Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Settings { flags: (0x1: ACK) }
2022-12-12T19:17:43.699789Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=WindowUpdate { stream_id: StreamId(0), size_increment: 5177345 }
2022-12-12T19:17:43.699812Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=WindowUpdate { stream_id: StreamId(0), size_increment: 983041 }
2022-12-12T19:17:43.699904Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Settings { flags: (0x1: ACK) }
2022-12-12T19:17:43.699921Z DEBUG Connection{peer=Server}: h2::proto::settings: received settings ACK; applying Settings { flags: (0x0), initial_window_size: 1048576, max_frame_size: 16384, max_header_list_size: 16777216 }
2022-12-12T19:17:43.700171Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(1), flags: (0x4: END_HEADERS) }
2022-12-12T19:17:43.700226Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(1) }
2022-12-12T19:17:43.700246Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(1), flags: (0x1: END_STREAM) }
Got a request from: [::1]:46330 with certs: []
2022-12-12T19:17:43.700508Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(1), flags: (0x4: END_HEADERS) }
2022-12-12T19:17:43.700567Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Data { stream_id: StreamId(1) }
2022-12-12T19:17:43.700594Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(1), flags: (0x5: END_HEADERS | END_STREAM) }
2022-12-12T19:17:43.701357Z DEBUG Connection{peer=Server}: h2::proto::connection: Connection::poll; IO error error=UnexpectedEof
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: hyper::Error(Io, Kind(UnexpectedEof))', examples/src/tls/server_rustls.rs:92:52
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

LucioFranco · 2022-12-13T16:38:29Z

Can you tell me what commit you're on? And verify you've changed nothing? Maybe this has to do with linux kernel 6

andreiltd · 2022-12-13T17:38:53Z

I'm on d00fd08. I have no local changes other than logging enabled (but initially checked without logging)

diff --git a/examples/src/tls/server_rustls.rs b/examples/src/tls/server_rustls.rs
index 8985cf0..e2371e4 100644
--- a/examples/src/tls/server_rustls.rs
+++ b/examples/src/tls/server_rustls.rs
@@ -12,9 +12,13 @@ use tokio_rustls::{
 };
 use tonic::{transport::Server, Request, Response, Status};
 use tower_http::ServiceBuilderExt;
+use tracing_subscriber::EnvFilter;
 
 #[tokio::main]
 async fn main() -> Result<(), Box<dyn std::error::Error>> {
+    let filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("debug"));
+    tracing_subscriber::fmt().with_env_filter(filter).init();
+
     let certs = {
         let fd = std::fs::File::open("examples/data/tls/server.pem")?;
         let mut buf = std::io::BufReader::new(&fd);

andreiltd · 2022-12-13T17:40:32Z

Let me know what else I can check. I'm happy to dig into this :)

LucioFranco · 2022-12-13T18:09:05Z

Could you clarify the full version of your OS and linux kernel? I am having trouble finding out what version you are using etc.

andreiltd · 2022-12-13T18:21:27Z

OS: Pop!_OS 22.04 LTS
Linux version: 6.0.6-76060006-generic (jenkins@warp.pop-os.org)

It is based on Ubuntu 22.04 LTS, but they provide updated kernel, I guess.

crisidev · 2022-12-13T18:58:46Z

I can reproduce on ubuntu 22.04 with 6.0.0-1007-oem.

LucioFranco · 2022-12-13T19:09:31Z

Same here I can reproduce it on debian with kernel 6. I also updated my code locally on my mac and was not able to reproduce with the same version of rustls. Additionally, this only happens with the rustls and not the general helloworld.

LucioFranco · 2022-12-14T18:25:07Z

Ok so its not a linux kernel 6+ issue, I am seeing it on 5.4 as well.

Related to hyperium/tonic#1183

LucioFranco · 2022-12-14T19:42:03Z

Okay so I've been digging some more, beyond the UnexpectedEof I am also seeing NotConnected errors crop up. I wrote a fix for it here hyperium/h2#654.

As for the eof part of this, I am still debugging it but it doesn'thappen

@quinner do you have any idea? When I get eof I see there is around 8k bytes in the buf and thats why its returning eof but I don't know beyond that why its failing.

dinoling · 2023-02-23T02:03:21Z

Platform
server : almalinux 9.1
client : windows 11

the same happens

The client does not send out "warning alert Close_Notify", not every time

Nehliin · 2023-11-23T10:01:22Z

@LucioFranco we've experienced this as well a lot seemingly randomly in prod and I wonder if you think these two issues could be related hyperium/h2#636?

xandkar · 2024-03-01T14:44:02Z

Same two symptoms for me in examples/src/tls_rustls/server.rs:

http.serve_connection(conn, svc).await --> hyper::Error(Io, Kind(UnexpectedEof)) ;
info.peer_certificates() --> None.

My setup:

Void Linux, kernel 6.6.16_1;
rustc 1.75.0 (82e1608df 2023-12-21);
tonic 233711a.

Also happens outside of examples in the repo, in another project where dep versions mostly mirror those from the examples:

[dependencies]
clap = { version = "4.5.1", features = ["derive"] }
hyper = "0.14"
prost = "0.12.3"
rustls-pemfile = "1"
tokio = { version = "1.36.0", features = ["full"] }
tokio-rustls = "0.24.0"
hyper-rustls = {version = "0.24.0", features = ["http2"]}
tonic = "0.11.0"
tower = "0.4.13"
anyhow = { version = "1.0.80", features = ["backtrace"] }
tower-http = { version = "0.4", features = ["add-extension", "util"] }

[build-dependencies]
tonic-build = "0.11.0"

Consistently, every time, not just intermittent/random.

kc1212 · 2024-04-05T13:30:41Z

I'm also having this issue on macOS 14.3 with rustc 1.76.0. This repo https://github.com/kc1212/tonic-tls-poc will reproduce the issue by starting the server and then the client. The code is essentially what is in the rustls example. Below is the error message:

thread 'tokio-runtime-worker' panicked at src/server.rs:93:52:
called `Result::unwrap()` on an `Err` value: hyper::Error(Io, Kind(UnexpectedEof))
stack backtrace:
   0: rust_begin_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/result.rs:1649:5
   3: server::main::{{closure}}::{{closure}}
   4: std::panicking::try
   5: tokio::runtime::task::raw::poll
   6: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
   7: tokio::runtime::scheduler::multi_thread::worker::Context::run
   8: tokio::runtime::context::scoped::Scoped<T>::set
   9: tokio::runtime::context::runtime::enter_runtime
  10: tokio::runtime::scheduler::multi_thread::worker::run
  11: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  12: tokio::runtime::task::core::Core<T,S>::poll
  13: tokio::runtime::task::harness::Harness<T,S>::poll
  14: tokio::runtime::blocking::pool::Inner::run

LucioFranco added a commit to hyperium/h2 that referenced this issue Dec 14, 2022

server: Ignore NotConnected errors on shutdown

c11e186

Related to hyperium/tonic#1183

LucioFranco mentioned this issue Dec 14, 2022

server: Ignore NotConnected errors on shutdown hyperium/h2#654

Closed

LucioFranco mentioned this issue Dec 19, 2022

Internal: Unexpected EOF decoding stream #1199

Closed

repi mentioned this issue Nov 23, 2023

Embark issues EmbarkStudios/rust-ecosystem#36

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Running rustls example results in panic on the server side #1183

Running rustls example results in panic on the server side #1183

andreiltd commented Dec 9, 2022

LucioFranco commented Dec 12, 2022

LucioFranco commented Dec 12, 2022

andreiltd commented Dec 12, 2022

andreiltd commented Dec 12, 2022

LucioFranco commented Dec 13, 2022

andreiltd commented Dec 13, 2022

andreiltd commented Dec 13, 2022

LucioFranco commented Dec 13, 2022

andreiltd commented Dec 13, 2022

crisidev commented Dec 13, 2022

LucioFranco commented Dec 13, 2022

LucioFranco commented Dec 14, 2022

LucioFranco commented Dec 14, 2022

dinoling commented Feb 23, 2023

Nehliin commented Nov 23, 2023

xandkar commented Mar 1, 2024 •

edited

Loading

kc1212 commented Apr 5, 2024 •

edited

Loading

Running rustls example results in panic on the server side #1183

Running rustls example results in panic on the server side #1183

Comments

andreiltd commented Dec 9, 2022

Bug Report

Version

Platform

Description

LucioFranco commented Dec 12, 2022

LucioFranco commented Dec 12, 2022

andreiltd commented Dec 12, 2022

andreiltd commented Dec 12, 2022

LucioFranco commented Dec 13, 2022

andreiltd commented Dec 13, 2022

andreiltd commented Dec 13, 2022

LucioFranco commented Dec 13, 2022

andreiltd commented Dec 13, 2022

crisidev commented Dec 13, 2022

LucioFranco commented Dec 13, 2022

LucioFranco commented Dec 14, 2022

LucioFranco commented Dec 14, 2022

dinoling commented Feb 23, 2023

Nehliin commented Nov 23, 2023

xandkar commented Mar 1, 2024 • edited Loading

kc1212 commented Apr 5, 2024 • edited Loading

xandkar commented Mar 1, 2024 •

edited

Loading

kc1212 commented Apr 5, 2024 •

edited

Loading