ERL-154: ERTS pollset can consume 100% CPU after manipulating an inet socket file descriptor from an Erlang port object

[OTP-Maintainer]

Original reporter: okeuday
Affected version: OTP-19.0
Component: erts
Migrated from: https://bugs.erlang.org/browse/ERL-154

---

I mentioned this problem on https://github.com/erlang/otp/pull/612 due to wanting to use the changes tracked there, to avoid the problem.  I have tracked the problem at https://github.com/CloudI/CloudI/issues/39 .  Making the problem manifest does involve a NIF, however, I have verified that the NIF is not what is consuming 100% CPU.  This problem was discovered due to using the undocumented functions prim_inet:getfd/1 and prim_inet:ignorefd/2 combined with dup2 to force the Erlang VM to accept a unix domain socket file descriptor.

I have created a more minimal example of the problem (based on the CloudI source code), to make this easier to test.  To use the attached "test.tar.gz" file, just uncompress and cd into the test directory.  Then type "make" (hopefully these basic options based on autoconf/automake/libtool works, but you might need to tweak a little based on your setup).  In one shell, do the following:

{noformat}
$ erl -pz ebin/
Erlang/OTP 19 [erts-8.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [kernel-poll:false]

Eshell V8.0  (abort with ^G)
1> socket_local:start_link().
{ok,<0.59.0>}
2> 
{noformat}

The pollset processing thread should now be consuming 100% of the CPU.
In another shell (to show the file descriptor still works), do:

{noformat}
$ ./test
input: I am sending this text to the unix file descriptor
write(54)
read(4) header
read(50)
output: I am sending this text to the unix file descriptor
input: 
{noformat}

Press return to exit.

[OTP-Maintainer]

raimo said:

I try to figure out what your code does...  Do I read it correctly when I think that you:
* Have a NIF create an AF_LOCAL listen socket and a thread that listens on that socket
* When the NIF thread returns from accept() it bangs the accepted file descriptor to an erlang process
* The erlang process sets receive and send buffer sizes on the received file descriptor through a NIF
* Then the erlang process:
** Creates a listen socket
** Connects to it from another socket
** Accepts the connection
** Gets the fd of that connection and opens it again with gen_tcp:fdopen/2
** Finally dups the accepted fd received from the NIF thread onto the fd opened with gen_tcp:fdopen/2

If so the gen_tcp:fdopen/2 has set the fd to nonblocking and entered into the VM pollset, and after that you dup another file descriptor onto it.  That sounds really weird!  I do not know if it is worth investigating, if I understood this correctly...

The first approach that I thought of was to have a NIF that creates an AF_LOCAL socket and return the fd.  Then to call gen_tcp:listen(0, \[\{fd,Fd\}\]) on it.

Or to just call gen_tcp:fdopen/2 on the fd produced by your NIF thread without all the listen, connect, accept stuff before that.  Not to dup to it after fdopen...

But I guess something with those simpler approaches did not work...

[OTP-Maintainer]

okeuday said:

I understand the approach is a nasty hack, but the context of the solution is important.  The options were:
* What is in the test, relying on dup2 to trick the Erlang VM with its own file descriptor after having passed internal inet validation
* Replicate all the inet source code with tweaks and then use that, as has been done at https://github.com/tonyrog/afunix
* Don't use the inet source code, and use something external like https://github.com/msantos/procket

When judging the 3 options, the test should be the least SLOC, so complexity-wise, it should be the best approach, despite it being a hack.  The Erlang module socket_local.erl has the following internal sequence:
# socket_open/4 -> cloudi_core_i_socket:local/1 which makes a listening unix domain socket and lets a background thread poll for the first accept (only caring about the first connection) and then sending the socket_local.erl process a {inet_async, undefined, undefined, {ok, FileDescriptor}} message, to make it look like prim_inet:async_accept/2 usage (since CloudI has multiple socket types being handled by the same module that socket_local.erl is based on)
# socket_accept/2 -> cloudi_socket_set/2 shows how the Erlang VM gets tricked by its own file descriptor, with a fake listener that allows the creation of a fake client connection from the listener, so that socket can be used as if it was created normally, despite being a normal inet socket, after doing the sequence prim_inet:getfd/1, prim_inet:ignorefd/2, gen_tcp:fdopen/2, cloudi_core_i_socket:set/2 where cloudi_core_i_socket:set/2 only does dup2 and closes the fake client connection from the listener
# after that, it is a normal socket use

I understand the approach is odd, but I assume that it may occur in the future with similar file descriptor handling.  It may be related to the non-blocking status of the socket not matching the file descriptor somehow.  However, it seems wrong that it makes the pollset thread consume 100% without any errors, because that means it isn't doing what it is suppose to, which is block on the poll.  I did look into it, but it wasn't clear why there was a problem.

[OTP-Maintainer]

raimo said:

Well, gen_tcp:fdopen/2 will set the Fd to nonblocking and then you dup2 another file descriptor onto it.  If that new file descriptor is not set to nonblocking the VM certainly might get confused by that.

Also, if you have set the {active,NotFalse} option on it the Fd should be in the poll call for the pollset thread when you dup2 a different file descriptor onto it, and this sound really weird.

[OTP-Maintainer]

okeuday said:

As you may see in the source code, {{cloudi_core_i_socket:setsockopts/3}} sets the Fd to nonblocking before the dup2.  The function {{cloudi_socket_set}} is executed after, and deals with all its sockets using Active==false, with the dup2 occurring at the end of that function.  After the function {{cloudi_socket_set}} is done, i.e., after dup2 has occurred with a nonblocking Fd, then Active==once is set.

[OTP-Maintainer]

raimo said:

Although it seems it could work I think it is too complex to dive into before knowing that this is the only possible solution.

If the new AF_LOCAL support in 19.0 should turn out to lack some feature I would prefer to add that if possible, and only if that path is not usable we should get back to this issue...

[OTP-Maintainer]

okeuday said:

I already changed the CloudI source code so that this approach is not used with Erlang/OTP >= 19.0, due to the new AF_LOCAL support.  My main concern about this bug, is if it manifests after http://bugs.erlang.org/browse/ERL-140 is fixed it should cause the whole machine to lock-up.  So, this bug might be something to investigate as part of the improvements necessary for http://bugs.erlang.org/browse/ERL-140 .

[OTP-Maintainer]

raimo said:

Ok. Great!

I'll close this one and have made a comment referencing here from ERL-140.

[OTP-Maintainer]

raimo said:

Not worth the time to reproduce for its own sake.  See also http://bugs.erlang.org/browse/ERL-140.