Passing sockets over pipes fails on windows #926

Closed
stakach opened this Issue Sep 12, 2013 · 41 comments

Comments

Projects
None yet
Contributor

stakach commented Sep 12, 2013

I'm working on implementing an FFI implementation of libuv for ruby compatible with the v0.11.13 release of libuv: https://github.com/cotag/libuv

All of my tests run on unix without issue however I cannot pass sockets over pipes on Windows. I continually receive: EAGAIN: resource temporarily unavailable no matter how much I delay or retry the accept. Once again this works flawlessly on all unix variants so I don't think I am doing anything wrong.

Also regular sockets and regular pipes work on windows, just not having any luck with write2 or read2...

ping @piscisaureus this may be relevant, I noticed that you've been working on windows compatibility.
Also, and unrelated, having an issue using unlink on windows after creating and writing a file with 777 permissions.

@stakach stakach referenced this issue in cotag/libuv Sep 12, 2013

Closed

Windows Segfault #1

Contributor

stakach commented Sep 12, 2013

Also @bnoordhuis you were working on the win write2 implementation recently. Any ideas?
I'm working through the libuv code atm

Contributor

saghul commented Sep 12, 2013

Does the libuv test suite pass for you? Last time I checked IPC worked fine on Windows.

Contributor

stakach commented Oct 18, 2013

I'm trying to pass sockets from one uv loop to another in the same process.

I looked at the test suite however was a little confused as to if it was passing sockets across different processes. Then if so, does this mean Windows doesn't support same process IPC only inter-process. As this would explain the issue.

Member

piscisaureus commented Oct 18, 2013

@stakach I don't think I actually ever tested passing sockets within the same process. However there is no theoretical reason it shouldn't work. However it could be that the socket isn't getting duplicated properly.

Just, guessing here, it's a long time ago: uv_spawn() sets some magic property on the uv_pipe_t handle so uv_write2 knows which process to duplicate the handle to. If you aren't spawning then this property might not set and WSADuplicateSocket could be failing.

Also note that passing uv_pipe_t handles over the IPC channel is currently not supported on windows.

Contributor

stakach commented Oct 18, 2013

Thanks for the pointer. I'll fiddle around with the libuv code and see if I can make it work then post any finding here / a pull request.

TCP connections are the only handles my tests are passing over IPC.

Contributor

stakach commented Oct 18, 2013

Ok, it seems: "Sockets can be shared among threads in a given process without using the WSADuplicateSocket function because a socket descriptor is valid in all threads of a process."

Source http://msdn.microsoft.com/en-us/library/windows/desktop/ms741565(v=vs.85).aspx
So a special case may need to be made when process ids match? Although not entirely sure how to go about this.. If it is possible.

Contributor

bnoordhuis commented Oct 18, 2013

@stakach What the MSDN documentation is saying is that you don't need to call WSADuplicateSocket(), not that you can't.

Can you test if test/benchmark-multi-accept.c works for you?

Member

piscisaureus commented Oct 18, 2013

@stakach

Ok, it seems: "Sockets can be shared among threads in a given process without using the WSADuplicateSocket function because a socket descriptor is valid in all threads of a process."

You need to duplicate in order not to mess up libuv's internal bookkeeping.

I'm having the same problems on Windows. Also, the tests named tcp_multi_accept* all fail in test/benchmark-multi-accept.c Is there any progress on a fix?

I am working with @grittygrease above on this and we are seeing the following.

We have a TCP server written using libuv with threads that works perfectly on Linux and Mac OS X but fails on Windows. The TCP server binds to a port in the main program, spawns a number of threads and passes the bound socket to them using pipes. Our code is similar in operation to test/benchmark-multi-accept.c.

In the threads we uv_listen() on the now duplicated socket and that call is successful. We then uv_run() but never receive a callback when a connection is made and, therefore, never get to call uv_accept() etc.

I have a suspicion that the duplication of the socket is failing when in the same process because pipe->ipc_pid is not set.

The code win/tcp.c does the following:

/* Use the IPC framing protocol. */
if (send_handle) {
  tcp_send_handle = (uv_tcp_t*)send_handle;

  err = uv_tcp_duplicate_socket(tcp_send_handle, handle->ipc_pid,
      &ipc_frame.socket_info);
  if (err) {
    return err;
  }

ipc_pid only gets set in uv_pipe_open or uv_spawn. Neither will happen if sending a uv_tcp_t via a pipe within the same process (but to a different thread). Thus handle->ipc_pid will be 0 and an error will occur.

Contributor

stakach commented Jan 29, 2014

I did some basic multiprocess socket passing on windows, to see if it would make a difference, and had no luck there either. The same code worked on OSX - however haven't had the time to dig into the internals of libuv

Contributor

indutny commented Jan 29, 2014

Will work on it

@ghost ghost assigned indutny Jan 29, 2014

Contributor

indutny commented Jan 29, 2014

Basically, the idea is to send pids in both ways at the pipe initialization and to use them as ipc_pid value

Contributor

indutny commented Jan 29, 2014

That was note for myself ^

Thanks. The other question I had about the source code is that when the uv_write2 happens the socket handle is duplicated (done in uv_pipe_write_impl via a call to uv_tcp_duplicate_socket). uv_duplicate_socket will do a listen() on the 'parent' socket.

Then uv_pipe_accept (in the subprocess/other thread) will call uv_tcp_import which does tcp->flags |= UV_HANDLE_SHARED_TCP_SOCKET; When I do a uv_listen on the socket in the subprocess/other thread this flag prevents a listen() from happening on the duplicated socket.

I'm unsure if that matters to the problem at hand but was something I spotted while reviewing the code.

Contributor

indutny commented Jan 31, 2014

After considering this, I found that sending pid over the IPC pipe is quite racey as users may start sending handles on that pipe before the PID message will arrive. I'll need to think about it for a bit...

Is there any news on this?

Contributor

indutny commented Feb 10, 2014

Not really, I don't think that this is going to work on windows (i.e. sending handles inside the same process), I'll consider working on another problem somewhere later, but I'm not really a windows guy here :) cc @piscisaureus

Contributor

indutny commented Feb 10, 2014

Though, I'll review and land a PR if you'll send one ;)

How far did you get? I'm interested in picking this up and seeing if we can patch it ourselves.

Contributor

indutny commented Feb 17, 2014

@jgrahamc I got to this https://github.com/indutny/libuv/compare/fix;gh-926 . I think it should be possible to wait for other side's ack before sending handles, just need to find time to implement it.

Thanks. I'll see what I can do.

John.

On Mon, Feb 17, 2014 at 1:18 PM, Fedor Indutny notifications@github.comwrote:

@jgrahamc https://github.com/jgrahamc I got to this
https://github.com/indutny/libuv/compare/fix;gh-926 . I think it should
be possible to wait for other side's ack before sending handles, just need
to find time to implement it.


Reply to this email directly or view it on GitHubhttps://github.com/joyent/libuv/issues/926#issuecomment-35255871
.

My book: The Geek Atlas, 128 Places Where Science and Technology Come Alive
http://geekatlas.com/ Signed copies: http://bit.ly/eSvmsV

Contributor

stakach commented Mar 14, 2014

Was just having a play around with the latest commits.
Looks like now when we have a pipe that supports write2 the pipe fails to accept: "accept error resource temporarily unavailable"

Regular pipes continue to work fine in Windows for me.

Contributor

saghul commented Mar 14, 2014

No IPC tests fail for me on Windows. Using pripes to send TCP handles across processes, that is. Unfortunately passing handles across threads continues to be an unsolved problem.

neoxic commented Mar 27, 2014

Fedor, I wonder if your pid sending patch is somewhere on the way.

Another (easier) option would be to leave it to the user, i.e. provide a windows-only function to set the ipc_pid field on Windows before they can send handles. Yeah, clumsy, but more robust and can be "officially" backed by platform "security requirements".:-)

CloudFlare is willing to pay a small bounty for proper support for this on
Windows. We would like to use libuv cross platform for a networked
application that we will be open sourcing. If there's someone who is wants
to work on this and has the skillset please get in contact with me directly.

John.

On Thu, Mar 27, 2014 at 6:20 PM, Arseny Vakhrushev <notifications@github.com

wrote:

Fedor, I wonder if your pid sending patch is somewhere on the way.

Another (easier) option would be to leave it to the user, i.e. provide a
windows-only function to set the ipc_pid field on Windows before they can
send handles. Yeah, clumsy, but more robust and can be "officially"
supported by platform "security requirements".:-)


Reply to this email directly or view it on GitHubhttps://github.com/joyent/libuv/issues/926#issuecomment-38841522
.

My book: The Geek Atlas, 128 Places Where Science and Technology Come Alive
http://geekatlas.com/ Signed copies: http://bit.ly/eSvmsV

@indutny indutny closed this in 6a657dc Apr 14, 2014

Contributor

retep998 commented Apr 16, 2014

@indutny Did you intend to close this issue with your commit?

@saghul saghul reopened this Apr 16, 2014

tojocky commented Aug 20, 2014

Any news on this topic?

Member

piscisaureus commented Aug 20, 2014

What are the use cases that people need this for?
Load balancing incoming connections, or something else?

@jgrahamc and @stakach

Specifically, we (CloudFlare) have a piece of software based on libuv that uses OpenSSL and we use libuv for threading and for handling a listen socket with multiple connections.

See my original comment above: #926 (comment)

Contributor

txdv commented Aug 20, 2014

If the bounty is still up, I can look at it at the weekend.

juanper commented Apr 30, 2015

Is possible setup ipc_pid in every call write2()?
I would like create my own pipe server side in my parent processes and control when pass tcp handles to my child processes.
Using uv_spawn() to pass pipe to child processes, libuv create 1 temporary pipe handle server side to each child process. Creating 1000 child processes, I'll have 1000 temporary pipes server handles and 1000 locally pipe handles passes to child processes using stdio struct in uv_spawn(), getting a total of 2000 handles.
If I use my own created server pipe handle this number is reduced to 1001 handles in parent process. Today I need set ipc_pid = GetCurrentProcessId() to use write2() and don't get error. (tested in libuv v1.4.2)

Contributor

stakach commented Apr 30, 2015

@juanper I'm not sure I understand the question and/or how it relates to the issue of passing TCP handles over pipes on Windows within the one process?

Or are you saying that if you create a proxy process and pass the handles to it you can then have the second process pass the sockets back to the original process?

Contributor

saghul commented Apr 30, 2015

You should be able to create the pipe yourself (pass 1 for IPC in
uv_pipe_init) and then in uv_spawn use the UV_INHERIT_STREAM flag.
On Apr 30, 2015 6:22 AM, "Juan José Pereira" notifications@github.com
wrote:

Is possible setup ipc_pid in every call write2()?
I would like create my own pipe server side in my parent processes and
control when pass tcp handles to my child processes.
Using uv_spawn() to pass pipe to child processes, libuv create 1 temporary
pipe handle server side to each child process. Creating 1000 child
processes, I'll have 1000 temporary pipes server handles and 1000 locally
pipe handles passes to child processes using stdio struct in uv_spawn(),
getting a total of 2000 handles.
If I use my own created server pipe handle this number is reduced to 1001
handles in parent process. Today I need set ipc_pid = GetCurrentProcessId()
to use write2() and don't get error. (tested in libuv v1.4.2)


Reply to this email directly or view it on GitHub
#926 (comment).

juanper commented Apr 30, 2015

@stakach in my case, I create a process who accept tcp connection, read first data block and according content of data block transfers tpc stream handle to a child process.

juanper commented Apr 30, 2015

@saghul I used UV_INHERIT_STREAM but all my attemps fails.
In uv__stdio_create() there is a test if "stream->flags & UV_HANDLE_CONNECTED" that allway fails in my tests. Have you a example code to follow? Thanks.

saghul added a commit to libuv/libuv that referenced this issue Nov 6, 2015

pipe: enable inprocess uv_write2 on Windows
When duplicating the socket handle being sent the target process
id is defaulted to the current process id. This enables uv_write2
to be used for thread-clustering in addition to process-clustering on
multi-threaded programming languages.

The ipc tests are updated to run in two modes. In the _inproc mode
the echo work is done on a second thread instead of in a second
process.

An internal function int uv_current_pid() is added to the windows
specific code which caches the value of GetCurrentProcessId(). This
means uv_write2 does not call GetCurrentProcessId() every inprocess
send.

Refs: joyent/libuv#926
PR-URL: #540
Reviewed-By: Saúl Ibarra Corretgé <saghul@gmail.com>
Contributor

saghul commented Nov 6, 2015

Fixed in libuv/libuv#540

Contributor

stakach commented Nov 6, 2015

Amazing work! Thanks!

juanper commented Nov 6, 2015

Great, thanks!

KeyserSoSay0 commented Jan 11, 2018

Sorry to revive a dead thread, but this still doesn't work by default and this is the first thread that comes up in searches.

So if you're wondering how to pass sockets between two different processes on windows: the trick is that WSADuplicateSocket(used inside uv_write2) actually requires the PID of the process that you want to pass the socket to, not the process that you're passing the socket from(which is what libuv currently assumes).

There's not really a way to do this internally to libuv, so you end up having to send a few handshake messages where you trade PIDs when you're setting up your connections. Then you can just manually set YourStream.pipe.conn.ipc_pid = WhateverThePIDOfTheOtherProcessIs sometime before you call uv_write2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment