New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intermittent error: ConnectNamedPipe failed: Windows error 232 #142

Open
octachrome opened this Issue Apr 17, 2018 · 8 comments

Comments

Projects
None yet
2 participants
@octachrome
Copy link

octachrome commented Apr 17, 2018

I use winpty in a Node application with https://github.com/daviwil/node-pty-prebuilt. A customer is reporting intermittent errors:

[56365.101 node.exe,p19464,t5808]: libwinpty error: code=5 msg='ConnectNamedPipe failed: Windows error 232'

Any ideas what could cause this or how I can find out why it is happening? I have not seen it in any of our test environments and no other customer has reported this. The customer says that the problem is intermittent, and retrying the operation usually works. He is using Windows 10 Enterprise.

I am using node-pty-prebuild v0.7.3, which uses winpty 0.4.3. Logs from winpty-debugserver.exe are attached: winpty2.log.

@rprichard

This comment has been minimized.

Copy link
Owner

rprichard commented Apr 17, 2018

I don't know why that would happen.

@octachrome

This comment has been minimized.

Copy link

octachrome commented Aug 31, 2018

I have had another report of this problem from a different customer. I believe that it is caused by McAfee Endpoint Security interfering with the startup of the winpty agent. Both clients have this antivirus software installed, and disabling McAfee resolves the issue.

I have reproduced the same error with Avast antivirus. When libwinpty launches the agent, Avast scans winpty-agent.exe for threats (see screenshot). It then appears to kill the agent and relauch it (with a different pid). This is all hidden from libwinpty - CreateProcessW returns the pid of the second agent once the scan has finished. The problem is that the first agent already connected to and disconnected from the control pipe, so when libwinpty calls ConnectNamedPipe it returns ERROR_NO_DATA, which is the documented behaviour when the client has already disconnected from the pipe.

It seems like Avast and McAfee have similar approaches to threat detection, and both involve running the agent twice, probably in some kind of sandbox for the first run. One way to handle this could be for libwinpty to accept multiple client connections and loop until one of them is proven to be OK, similar to this: https://docs.microsoft.com/en-us/windows/desktop/ipc/multithreaded-pipe-server. libwinpty would also have to call ConnectNamedPipe before spawning the agent, rather than afterwards as it does today.

Do you think this is a reasonable change to make? Or do you have any other ideas about how libwinpty could be modified to handle this situation? I am happy to implement any suggestions and submit a pull request.

There are several other reports of this error, so I think it is worth fixing. These antivirus tools are only going to get more common and more intrusive.

image

@rprichard

This comment has been minimized.

Copy link
Owner

rprichard commented Aug 31, 2018

Thanks for debugging this.

In winpty2.log, when the first pipe connection fails (to winpty-control-19464-4-...), I don't see a previous connection attempt to the pipe. Maybe when the agent runs inside the antivirus sandbox, it's not able to report trace() messages?

... One way to handle this could be for libwinpty to accept multiple client connections and loop until one of them is proven to be OK, similar to this: https://docs.microsoft.com/en-us/windows/desktop/ipc/multithreaded-pipe-server. libwinpty would also have to call ConnectNamedPipe before spawning the agent, rather than afterwards as it does today.

Do you think this is a reasonable change to make?

I suppose something like that has to be done?

Would libwinpty need a worker thread to service connection attempts? The pipe is using overlapped I/O already, so with one control-pipe instance, it's fairly easy to start the ConnectPipe operation before spawning the agent, then finish the attempt after CreateProcess returns, but if I understand correctly, something in libwinpty would need to start another ConnectPipe operation between the antivirus killing the first agent instance and starting the second instance.

Would it be sufficient if libwinpty created 2 or more instances of the control pipe up front, then used the first good instance after the CreateProcess call? As long as there's an instance ready, I think(?) the agent would be able to successfully connect.

I suspect the antivirus software would allow whitelisting the winpty-agent.exe process, but users either don't know they need to do that or lack admin privileges.

@octachrome

This comment has been minimized.

Copy link

octachrome commented Aug 31, 2018

I don't see a previous connection attempt to the pipe. Maybe when the agent runs inside the antivirus sandbox, it's not able to report trace() messages?

Yeah, I am assuming that McAfee blocks the trace messages somehow. I hope to verify this on the customer's machine soon. When running with Avast you can see both connections. See the attached winpty-avast.log.

Would it be sufficient if libwinpty created 2 or more instances of the control pipe up front, then used the first good instance after the CreateProcess call?

No idea. Presumably creating a multi-client pipe server is a normal thing to do so there must be a way of doing it such that the server doesn't miss any connections. I will look into it and see if I can get something working, using a worker thread if necessary.

rprichard added a commit that referenced this issue Sep 3, 2018

Work around antivirus programs
Some antivirus programs override CreateProcess() and run the child process
initially in a sandbox, then after deciding the process is OK, they run it
again for real.  The initial instance of winpty-agent.exe connects to
libwinpty's control pipe, then when the actual agent process starts later,
it can't connect to the pipe because the pipe is in a disconnected/broken
state.

Work around the problem by creating multiple instances of the control pipe
in libwinpty, then waiting on any of them to connect.  An error on one pipe
is logged to trace() but otherwise ignored as long as one of the pipes
eventually connects. The error isn't reported until the agent dies or the
connection has timed out.

In practice, the initial sandbox connection's ConnectNamedPipe operation
will probably succeed, but its child PID will be wrong, so this function
quietly ignores verifyPipeClientPid failures (as long as one pipe
succeeds).

Fixes #142

rprichard added a commit that referenced this issue Sep 3, 2018

Work around antivirus programs
Some antivirus programs override CreateProcess() and run the child process
initially in a sandbox, then after deciding the process is OK, they run it
again for real.  The initial instance of winpty-agent.exe connects to
libwinpty's control pipe, then when the actual agent process starts later,
it can't connect to the pipe because the pipe is in a disconnected/broken
state.

Work around the problem by creating multiple instances of the control pipe
in libwinpty, then waiting on any of them to connect.  An error on one pipe
is logged to trace() but otherwise ignored as long as one of the pipes
eventually connects. The error isn't reported until the agent dies or the
connection has timed out.

In practice, the initial sandbox connection's ConnectNamedPipe operation
will probably succeed, but its child PID will be wrong, so this function
quietly ignores verifyPipeClientPid failures (as long as one pipe
succeeds).

Fixes #142
@rprichard

This comment has been minimized.

Copy link
Owner

rprichard commented Sep 3, 2018

I can reproduce the issue with Avast, and I have a change that works around it: https://github.com/rprichard/winpty/tree/antivirus.

Are you able to reproduce the problem with McAfee, and can you check whether my antivirus branch fixes it?

@octachrome

This comment has been minimized.

Copy link

octachrome commented Sep 3, 2018

That's great, thanks so much for implementing this. I will ask our customer if they can test the patch with McAfee.

rprichard added a commit that referenced this issue Oct 15, 2018

Work around antivirus programs
Some antivirus programs override CreateProcess() and run the child process
initially in a sandbox, then after deciding the process is OK, they run it
again for real.  The initial instance of winpty-agent.exe connects to
libwinpty's control pipe, then when the actual agent process starts later,
it can't connect to the pipe because the pipe is in a disconnected/broken
state.

Work around the problem by creating multiple instances of the control pipe
in libwinpty, then waiting on any of them to connect.  An error on one pipe
is logged to trace() but otherwise ignored as long as one of the pipes
eventually connects. The error isn't reported until the agent dies or the
connection has timed out.

In practice, the initial sandbox connection's ConnectNamedPipe operation
will probably succeed, but its child PID will be wrong, so this function
quietly ignores verifyPipeClientPid failures (as long as one pipe
succeeds).

Fixes #142
@octachrome

This comment has been minimized.

Copy link

octachrome commented Nov 1, 2018

@rprichard Sorry for the delay, the customer took a while to respond, but they have confirmed that your change resolves the problem. Will you incorporate it into a winpty release at some point?

@rprichard

This comment has been minimized.

Copy link
Owner

rprichard commented Dec 8, 2018

Yeah, at some point I'll merge the fix and put out a new release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment