Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ERL-891: Remote Shells can get stuck in compute extensive busy loop #4343

Closed
OTP-Maintainer opened this issue Mar 19, 2019 · 2 comments
Closed

ERL-891: Remote Shells can get stuck in compute extensive busy loop #4343

OTP-Maintainer opened this issue Mar 19, 2019 · 2 comments
Labels
bug Issue is reported as a bug help wanted Issue not worked on by OTP; help wanted from the community priority:medium team:VM Assigned to OTP team VM
Milestone
OTP-26.0

Comments

@OTP-Maintainer
Copy link

Original reporter: arnomi
Affected version: OTP-21.1
Component: erts
Migrated from: https://bugs.erlang.org/browse/ERL-891

I originally posted this on the Elixir Forum (https://elixirforum.com/t/remote-shells-going-rogue/20970). The problem seems to be related to ERL-472 but regrettably I am not able currently to easily reproduce it.

The setup is the following. We have a node running on a machine A. We connect from the same machine multiple remote shells to A and then close the connection to the remote shells (e.g. close TCP connection to machine A) such that the remote shells remain alive but in theory unused. 

After a while we see that the remote shells may become fully consumed in a busy loop. (The  "getting stuck" in `group.erl` `more_data/6` (https://github.com/erlang/otp/blob/master/lib/kernel/src/group.erl#L665). The `group.erl` process seems to get many empty messages from `user_drv` of the form `{#PID<0.73.0>, {:data, []}}` causing the load of the remote shell to go up to 100%. 

The annoying part is that we are not able to easily reproduce the issue, but have now run into it a few times.
@OTP-Maintainer
Copy link
Author

josevalim said:

Have you ran across a consistent way to reproduce this issue? I would love to take a look at it and maybe see if it is caused by IEx, but without a way to reproduce it, there is nothing I can do. :(

@OTP-Maintainer
Copy link
Author

john said:

Thanks for your report!

I've found a bug in {{ttsl_drv}} that seems to fit perfectly but I've been unable to provoke it. Can you test the following branch? I've rigged it to crash dump when the bug occurs.

https://github.com/jhogberg/otp/tree/john/erts/ttsl_drv_eof/ERL-891

@OTP-Maintainer OTP-Maintainer added bug Issue is reported as a bug help wanted Issue not worked on by OTP; help wanted from the community team:VM Assigned to OTP team VM priority:medium labels Feb 10, 2021
@garazdawi garazdawi added this to the OTP-26.0 milestone Apr 7, 2022
garazdawi added a commit to garazdawi/otp that referenced this issue Aug 28, 2022
@garazdawi
erts: Re-Implement shell using nif
7ad9f6b 
This commit re-implements the entire tty driver for both Unix
and Windows to use a common nif instead of two seperate drivers.

The Unix implementation works pretty much as it did before only that
a lot more of the terminal logic has been moved from Erlang to C.

The windows implementation now uses Windows Terminal Sequences, i.e.
the same sequences as most Unixes to control the terminal. This means
that werl.exe is no longer needed and erl.exe will have the "newshell"
with all the features normally only found on Unix.

The new implementation also uses dirty I/O threads for all I/O which
means that it can leave the FDs in blocking mode. This fixes problems
when the Erlang tty is interacting with other systems such as bash.

Closes erlang#3150
Closes erlang#3390
Closes erlang#4343
garazdawi added a commit to garazdawi/otp that referenced this issue Aug 29, 2022
@garazdawi
erts: Re-Implement shell using nif
ca790b1 
This commit re-implements the entire tty driver for both Unix
and Windows to use a common nif instead of two seperate drivers.

The Unix implementation works pretty much as it did before only that
a lot more of the terminal logic has been moved from Erlang to C.

The windows implementation now uses Windows Terminal Sequences, i.e.
the same sequences as most Unixes to control the terminal. This means
that werl.exe is no longer needed and erl.exe will have the "newshell"
with all the features normally only found on Unix.

The new implementation also uses dirty I/O threads for all I/O which
means that it can leave the FDs in blocking mode. This fixes problems
when the Erlang tty is interacting with other systems such as bash.

Closes erlang#3150
Closes erlang#3390
Closes erlang#4343
garazdawi added a commit to garazdawi/otp that referenced this issue Aug 29, 2022
@garazdawi
erts: Re-Implement shell using nif
d870319 
This commit re-implements the entire tty driver for both Unix
and Windows to use a common nif instead of two seperate drivers.

The Unix implementation works pretty much as it did before only that
a lot more of the terminal logic has been moved from Erlang to C.

The windows implementation now uses Windows Terminal Sequences, i.e.
the same sequences as most Unixes to control the terminal. This means
that werl.exe is no longer needed and erl.exe will have the "newshell"
with all the features normally only found on Unix.

The new implementation also uses dirty I/O threads for all I/O which
means that it can leave the FDs in blocking mode. This fixes problems
when the Erlang tty is interacting with other systems such as bash.

Closes erlang#3150
Closes erlang#3390
Closes erlang#4343
@chunshengit chunshengit mentioned this issue Sep 1, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Issue is reported as a bug help wanted Issue not worked on by OTP; help wanted from the community priority:medium team:VM Assigned to OTP team VM
Projects
None yet
Milestone
OTP-26.0
Development

No branches or pull requests
2 participants
@garazdawi @OTP-Maintainer