AllocConsole failing with ERROR_GEN_FAILURE as a service in rare cases

[tsmortenson]

We have a software package running on thousands of Windows machines from XP onward. In 2 cases we have received a bug report that the users application fails to call AllocConsole with error code ERROR_GEN_FAILURE. This has been seen once on a Windows Server 2008 system and the second on a Windows Server 2012 system. In both cases the error was encountered while running as a Service.
The error message is "A device attached to the system is not functioning. (0x1f)"

Unfortunately we have not been able to reproduce this on any other systems. It happens 100% of the time of the failing system however.

I was able to find any documentation that shows this as a possible error.
Any thoughts on this would be appreciated.

[HBelusca]

Does the problem also happen if the service is an "interactive service" (services.msc -> select your service in the list -> Properties -> "Log On" tab -> "Allow service to interact with desktop") ?

[tsmortenson]

Thank you for your reply. We will see if we can check that.
Unfortunately this is on a system that is a customer of a customer. So it is difficult to make a lot of requests.
I was hoping that someone familiar with the source might have an idea. My guess is that some other api call within the AllocConsole call is causing the error.

[tsmortenson]

I reread your question. As I said, I can't easily check exactly what you asked. But we do have the configuration of our program from that machine. And it was setup in a way that when the service was registered, it was to NOT be interactive. So quite sure that will be the case of looking on the properties page.
We have run this way on many other servers and they all work fine in this way.
We are allocating a console even though it is not visible to allow us to send a CTRL-BREAK signal to the the child process which shares the console. This is only possible if the console exists.

[HBelusca]

Note that my "suggestion" is in no way a fix. It's just something so that it may better indicate where the actual problem is.

[tsmortenson]

We were able to run a test program on the customer machine to try a few things out.

One test was to have our service create a child cmd.exe process, then call AttachConsole to get that child's console. This works on all of our test machines, but on this machine the CreateProcess call with console on cmd.exe fails.

One idea is that something in the customer's Group Policy might be preventing a Service from creating a console? I was not able to locate any such thing with Google. But does something like that exist?