Logs are filling with `Received empty message from server` (160 GB after a few hours)

[vasimi]

Basic information

zellij --version: 0.30.0
stty size: 24 155
uname -av: Linux hostname 5.15.0-40-generic #43-Ubuntu SMP Wed Jun 15 12:54:21 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

List of programs you interact with as, PROGRAM --version: output cropped meaningful, for example:
npm --version: 8.11.0
node --version: v16.15.1
git --version: git version 2.34.1

Further information
I'm not able to recreate the issue (after killing zellij and deleting the logfile), however, I'll continue running zellij with the --debug arg and tailing the logs so I can attach some useful logs when the issue happens again.

I noticed that my disk is filling up and I found out that the zellij logfile is 160 GB in size and ~20 MB are adding up every second.
This message from the zellij_client was repeatedly printing into the logfile, like 40 or 50 times a second.

ERROR  |zellij_client            | 2022-07-07 13:24:42.442 [router    ] [/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:291]: Received empty message from server

[imsnif]

Hey, thanks for reporting (and for your offer to continue trying to reproduce). We introduced log rotation in main, so from next version when this happens it shouldn't take up more than a couple hundred kilobytes in logs. Hopefully we'll also get to the bottom of why this issue is happening in the first place (my suspicion is a stale session left over from a previous version, but I'm not sure).

[vasimi]

Happened again during the night while I left an open session, put my computer to sleep and my SSH connection died. The disk was full, so I had to delete the logfile again. :/ But it looks like it's related to stale session.

[imsnif]

Sorry this happened to you @vasimi ! Just to stress: this is not released yet. This should be in the next version, so it makes sense that it's still happening.

[vasimi]

Yup, just wanted to let you know I suspect that stale session is causing this after SSH connection is improperly terminated.

[imsnif]

Yup, just wanted to let you know I suspect that stale session is causing this after SSH connection is improperly terminated.

Thanks! Might be of interest to @raphCode

[raphCode]

If the message was Received empty message from server again like in the first post, this should be fixed too in the upcoming release: #1535
Although I don't exactly see why it would happen on ssh connection drop, I would expect it the other way round: Received empty message from client

[vasimi]

Got the log! After the Panic occured in client, Received empty message from server starts to appear several times per second.

INFO   |zellij_client            | 2022-07-12 16:57:14.183 [main      ] [/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:124]: Starting Zellij client!
DEBUG  |status-bar               | 2022-07-12 16:57:14.185 [id: 1     ] quicknav_show_count: 21
ERROR  |zellij_client            | 2022-07-12 19:17:58.783 [stdin_handler] [/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:205]: Panic occured in client:
PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: Os { code: 5, kind: Uncategorized, message: "Input/output error" }), location: Location { file: "/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/zellij-client-0.30.0/src/os_input_output.rs", line: 121, col: 39 }, can_unwind: true }
INFO   |zellij_client            | 2022-07-12 19:17:58.788 [main      ] [/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:369]: Bye from Zellij!
ERROR  |zellij_client            | 2022-07-12 19:17:58.788 [main      ] [/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:205]: Panic occured in client:
PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: EIO), location: Location { file: "/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs", line: 370, col: 32 }, can_unwind: true }
ERROR  |zellij_client            | 2022-07-12 19:17:58.788 [input_handler] [/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:205]: Panic occured in client:
PanicInfo { payload: Any { .. }, message: Some(Encountered read error: RecvError), location: Location { file: "/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/zellij-client-0.30.0/src/input_handler.rs", line: 132, col: 29 }, can_unwind: true }
INFO   |zellij_client            | 2022-07-12 20:06:02.192 [main      ] [/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:124]: Starting Zellij client!
DEBUG  |status-bar               | 2022-07-12 20:06:02.195 [id: 1     ] quicknav_show_count: 21
INFO   |zellij_client            | 2022-07-12 20:07:11.340 [main      ] [/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:369]: Bye from Zellij!
INFO   |zellij_client            | 2022-07-13 10:44:02.995 [main      ] [/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:124]: Starting Zellij client!
DEBUG  |status-bar               | 2022-07-13 10:44:02.998 [id: 1     ] quicknav_show_count: 21
INFO   |zellij_client            | 2022-07-13 12:49:06.589 [main      ] [/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:124]: Starting Zellij client!
DEBUG  |status-bar               | 2022-07-13 12:49:06.593 [id: 1     ] quicknav_show_count: 21
ERROR  |zellij_client            | 2022-07-13 12:49:37.866 [stdin_handler] [/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:205]: Panic occured in client:
PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: Os { code: 5, kind: Uncategorized, message: "Input/output error" }), location: Location { file: "/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/zellij-client-0.30.0/src/os_input_output.rs", line: 121, col: 39 }, can_unwind: true }
ERROR  |zellij_client            | 2022-07-13 12:49:52.344 [router    ] [/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:291]: Received empty message from server

[raphCode]

I think that should be covered in the next release, please comment again if it is not!