"Attempted to configure an invalid screen layout" and "Rect too big"

[nathanshearer]

Connections from tigervnc and turbovnc are impossible, always results in screen geometry errors and the connection is terminated.

VNC Server

• Windows 10 Pro
• UltraVNC 1.3.2 x64

VNC Client

• TurboVNC 2.2.6 vncviewer
• TigerVNC 1.110 vncviewer

Note that connections to UltraVNC 1.2.40 server work fine from TurboVNC 2.2.6 vncviewer.

TurboVNC/turbovnc#292
TigerVNC/tigervnc#1257

[nathanshearer]

Error log from TurboVNC vncviewer:

$ vncviewer 
libjawt.so path: /etc/java-config-2/current-system-vm/lib/amd64
CConn: connected to host 192.168.0.110 port 5900
CConnection: Server supports RFB protocol version 3.8
CConnection: Using RFB protocol version 3.8
CConn: Using pixel format depth 24 (32bpp) little-endian rgb888
CConn: Requesting Tight encoding
TurboVNC Helper: Disabling X11 full-screen mode for window 0x07400060
CConn: Enabling GII
No extended input devices.
CMsgHandler: Server sent us an invalid screen layout
CMsgReader: Rect too big: 64x64 at 960,512 exceeds 0x0
Rect too big

[nathanshearer]

This appears to be a problem with UltraVNC where the VNC protocol is violated:

See the comments from the TurboVNC development: TurboVNC/turbovnc#292

[nathanshearer]

Is this bug getting any attention?

I am now in a situation where I cannot connect to a remote system because it is using an UltraVNC server and all the open source clients are rejecting the connection because the UltraVNC server is using an invalid screen layout which violates the VNC protocol.

The exact error output by TigerVNC Viewer is: Attempted to configure an invalid screen layout

I would consider using an UltraVNC client for special cases like this, however it there is no UltraVNC client for linux.

I may need to consider switching all our systems away from UltraVNC and switch to something else.

[RudiDeVos]

Wasn't this fix ? In 1.3.4.2 . Report is for 1.3.2

[Neustradamus]

@nathanshearer: Can you test a more recent build and do a feedback?

[mbey-mw]

Since I'm struggling with ExtendedDesktopSize pseudo-encoding and compatibility issues, I would like to provide appropriate feedback on this here.
The fix for version 1.3.4.2 fixes the problem of the zero size rectangle and thus the screen geometry errors. Connections with TurboVNC and TigerVNC clients are possible again. However, there are other problems with multimonitor configurations that I have listed in a new issue #56.