run example.exe on Windows 10 output error msg: select, An operation was attempted on something that is not a socket

[AlexiaChen]

Hi Eeveryone:

System Enviroment:

• My Operating System is : windows 10 enterprise 2016 LTSB
• My build toolchain: MSVC (Visual Studio 2013), Cmake

Promblem Description

I just simply build libvncserver on windows 10 referred to: https://github.com/LibVNC/libvncserver/blob/master/.appveyor.yml
it build compeletely success with 0 errors.

I find example.exe in the folder of examples, and run example.exe from CMD. but it repeating output error message: select, An operation was attempted on something that is not a socket

it seems like the socket handle parameter did not reference a valid socket, or for select, a member of an fd_set was not valid.

So, I just download realVNC's VNC viewer to connect localhost:5900 but it is does not work and I tried no_vnc(html5 client) in webclients folder and does not work,too.

Then, I tried simple.exe, vncev.exe looks well when no client connected to them,
but they also ouptput error message as follows when vncviewer conected:
vncev.exe:

simple.exe:

Does anyone can solve this problem?

Finally, I tried use SDLviewer in client_examples folder, the server(vncev,exe, simple.exe, example.exe) also repeating show error messages like above.

I have really no idea to solve this from now.

[bk138]

Does this happen on other Windows versions as well? I have it here, too, Windows 10.

[AlexiaChen]

@bk138
Thank you for your report. sorry, I just tested on windows 10. I do not know the detail on other windows version. I will test on windows 7 tomorrow.

[AlexiaChen]

Hi @bk138 ：
I do the test on Windows 7 32bit without SP1(on VMware), I run vncev.exe, example.exe and simple.exe on windows 7, and use SDLviewer on win 10 to connected to win 7 on VMware, it also repeating error message as follows, but, the error message seems CHANGED!!!!
vncev in VMware:

simple in VMware:

example in VMware:

[AlexiaChen]

Maybe, microsoft changed the error tips from windows 7 to windows 10. Because, There are same error message on windows 10 like yesterday , :)

[AlexiaChen]

@bk138

I Debug the libvncserver and Comment the code sinppet in Sockets.c as follows:

then , vncev works well and running comtinue, it does not repeating oupt error message:
rfbProcessNewConnection: error in select
when the new connection comes.

but a new issue pops up (windows 10 and windows 7 comes same errror):

When I use RealVNC Viewer connect the server, RTB protocal report error: bad desktop size: 25600* 25600

[AlexiaChen]

@bk138
I fixed the bug is that winSocket is diferrent with Linux, win sock is actually usigned integer, and you assign -1 to it(wrong ! big integer, so , the error message is show not valid file descriptor) , want to represent invalid socket. but win socket's invalid socket officially offerred, its Macro INVALID_SOCKET.

[AlexiaChen]

But SDLViewer, ppmtest, TightVNCViewer RealVNCViewer does not works as well.

[aquawicket]

I can confirm the same issue on Win7 SP1 x64 using current libvncserver-master.
libvncserver builds with 0 errors with Cmake/VS2015. Running example code gives one of 2 repeated error message with no connectivity.

httpCheckFds: select: An operation was attempted on something that is not a socket.

or

rfbProcessNewConnection: error in select:

Is there a known fix for this?

.

[AlexiaChen]

@aquawicket no, there is no fix for this from now. I tried to fix this , but its a big challenge of grasp this whole project in limited time

[lvorobyov]

I am solved this problem under Windows XP and on MinGW GCC compiller. This problem lies in the types of data used. All of socket descriptors must be declared, as SOCKET instead int, so as in WinSock2 all of SOCKET are presented as void*, but under UNIX they are equivalent int. When you checking expression (socket(...) >= 0) under Windows, this condition is always TRUE, because void* is an unsigned integer type. Details: https://msdn.microsoft.com/ru-ru/library/windows/desktop/ms740516(v=vs.85).aspx .
Soon I will make a commit of my own solution of this problem.

[lvorobyov]

I present to your attention my pull request #185 and I'm waiting for your comments.
Best wishes, Lev Vorobjev.

[AlexiaChen]

@lvorobjev Thanks , I memtioned the bug is Socket data type above, I tried to fix it before, but the other issue pops up. so I stop research it. Let me check your pull request.

[gops4u]

Hi,

I found another link which provides insight into a similar issue and fix, https://sourceforge.net/p/libvncserver/bugs/39/

@Ivorobjev : Are you able to connect to your vnc server built using mingw using the novnc?
I was able to build the vnc server using mingw64_x86_64 but web-sockets couldn't be built.

[aquawicket]

That bug is 3 years old, Has libvncserver not worked on windows for that long? Or maybe it came incrementally with new win OS's. Anyway, What was the last version that worked on windows? I have no luck creating any server on windows. All errors above continue.

[bk138]

I think the reason is that it was almost exlusively built with MinGW for Windows, not MSVC.

[ChongDeng]

this issue still exist. cant run on Windows 10

[aquawicket]

whm200410 fork works for Windows, but has not been tested on all systems yet.
I use whm200410's version for windows. I use "this" fork for OSX and LINUX.
I think whm200410's fork breaks something on unix platforms. Still needs investigating.
I'll post errors when i get time. :)

Reference: #195

[aquawicket]

Update:
Only LibVNC vncserver is broken for Windows.
LibVNC vncclient works on Windows with this code. :)
whm200410 fork fixes vncserver on Windows, but may not work on unix


Important Files currently different between LibVNC and whm200410

libvncserver/httpd.c
libvncserver/main.c
libvncserver/rfbserver.c
ibvncserver/sockets.c
rfb/rfb.h
rfb/rfbclient.h
rfb/rfbproto.h

Currently testing lvorobjev patches
b938de3
bf9e45b
44c95bd

[sfhacker]

We are facing the same issues here on Windows 10. This is quite an old post and it (this issue) does not seem fixed. We should attempt to fix severe or critical and/or old issues first. Then move on to other issues.
Is there any patch available somewhere? We are new to this project and not much idea about how to test it (server & client) on Windows. Thanks in advance.

[bk138]

This relates to #185 which needs thorough review by someone familiar with the code and possible side effects. This will get closed once #185 or a similar PR is merged.

[Dian8]

Having "rfbProcessNewConnection: error in select: The operation completed successfully" on windows 7 too (libvncserver master)

[bk138]

Fixed by #257, thanks @tobydox