xrdp protocol error with Windows 7, 10 client

[tfischer77]

I'm facing an issue when connecting with the Windows 7 client and the following settings:
32bpp, 1680x1050, Connection type: WAN
Login is possible, but if I open a picture with a lot of details on it, the client suddenly crashes with "protocol error". If I try to reconnect to the session with a e.g. 24bpp, everything is fine and the picture is shown. Also, if I'm connecting with a higher resolution, e.g. 1920x1200, no protocol error occurs.
Connecting with "LAN" (rfxcodec) is also ok. The behaviour is exacly the same as in #524, which I thought to be solved. Do we still have a buffer which is too small?
Running xrdp with the --nodaemon option gives the following:
ssl_tls_print_error: SSL_write: I/O error
xrdp_rdp_send_fastpath: xrdp_fastpath_send failed
got XRDP SIGPIPE(13)
ssl_tls_print_error: SSL_write: I/O error
xrdp_rdp_send_fastpath: xrdp_fastpath_send failed
got XRDP SIGPIPE(13)
ssl_tls_print_error: SSL_write: I/O error
xrdp_rdp_send_fastpath: xrdp_fastpath_send failed
got XRDP SIGPIPE(13)
ssl_tls_print_error: SSL_write: I/O error
xrdp_rdp_send_fastpath: xrdp_fastpath_send failed
got XRDP SIGPIPE(13)
ssl_tls_print_error: SSL_write: I/O error
xrdp_rdp_send_fastpath: xrdp_fastpath_send failed

However, disabling "fastpath" in xrdp.ini did not resolve the issue.
Is there some "overrun"? I expect if I connect with 24bpp, there is less data to be transferred.
Hint: The issue is not easy to reproduce. Steps that lead me to the crash: Open a xrdp session, then start the program ssvncviewer and connect to a different system with a "fancy" background image.
Tried to copy the background image and open it with an image viewer: no crash....
But if you have an idea where the issue could come from, I will be able to test the modified code.

[metalefty]

I also reproduced the issue.

[tfischer77]

I can also reproduce the issue with "remmina" if selecting 32bpp (not rfxcodec).
I change LLOGLN to a value > 10, which gives me the following output when crashing:
xrdp_crash_with_32bpp.txt
Sorry for the file size, but I added the whole log from login to crash.
Crashing xrdp works the following way: Start a xfce session, open e.g. a file dialog, move it a round for a while on the screen, and suddenly it stops working and remmina crashes completely (as well as windows 7 rdp client).

[jsorg71]

Since you use WAN, it's not RemoteFX session.
This does not happen with 24 or 16 bpp color, right?
This is issue with tile size 64x64 when it does not compress well, it's over 16K = 64 * 64 * 4.
The max RDP PDU size is a bit unclear. RDP uses 16 bits for the PDU length and top bit is used for something else so we have 15 bits. That should be 32768 max RDP PDU. MSTSC errors somewhere around 16K. Has anyone seen any documentation on this?

[tfischer77]

That's right, the issue occurs only with WAN and 32bpp. 24 and 16 are working. It seems to also work with higher resolutions (e.g. 1920x1200), as told before, but I could not fully test that.

[metalefty]

Not only Windows 7, Windows 10 shows protocol error as well.

[Suncatcher]

Yes, Windows 10 too. In my case it happens just after I connect to OpenVPN via Network Manager.
Though I cannot find reasonable explanation of it.
I can add also that I use Guacamole client too, and I it affects Guacamole as well. After establishing the OVPN connection within guest Guacamole screen get freezed and reconnect is unsuccessful. Only guest restart helps.

[Suncatcher]

Also I wanna add this happens also without OpenVPN. Did you find the cause?

[speidy]

@tfischer77 Can you tell what should be done in order to re-produce this issue?

[metalefty]

Here's the summary of reproduction procedure I and speidy discussed on gitter chat.

client=Win 7 mstsc 6.3.9600
xrdp=5def059
xorgxrdp=0.2.1
os=FreeBSD 11

32bit xorgxrdp, no RFX (connectionType!=LAN) caused protocol error for me.
24bit color doesn't cause protocol error.

[tfischer77]

I tested with almost the same setup. I took the latest devel branch for my tests. 24bit color is just working.

[Suncatcher]

I can say that I run 24bit, and this error is throwed in this mode too.
This is sesman output when listing sessions:

[20170621-10:14:42] [DEBUG] libscp initialized
[20170621-10:14:42] [INFO ] [v1c_mng:405] connection ok
Session ID: 1131
Session type: 3
Screen size: 1920x1080, color depth 24
Idle time: 0 day(s) 0 hour(s) 0 minute(s)
Connected: 2017/06/21 08:52
[20170621-10:14:42] [DEBUG] Closed socket 4 (AF_INET 127.0.0.1:48812)
[20170621-10:14:42] [CORE ] shutting down log subsystem...

[metalefty]

@Suncatcher In this case, your client's color depth setting does matter. The log shows always "color depth 24" even if you're connecting 16 bit color client.

@tfischer77 And 32bit color shows protocol error, right?

[metalefty]

@speidy I reproduced the protocol error before logging in to the session. At the login screen also cause protocol error. I think this is not xorgxrdp issue.

[tfischer77]

I switched to 24bit color in xrdp.ini. Since then everything seems to be ok. And we have hundreds of users connecting every days.... no complaints so far. 32bit still shows error...

[metalefty]

Maybe the difference between 24bit color and 32bit color is here.
https://github.com/neutrinolabs/xrdp/blob/devel/libxrdp/xrdp_orders.c#L2368

So 32bit compression have some problem. We're looking into that.

[metalefty]

Fixed by #803. Can anybody test?

[tfischer77]

@metalefty you mean by #804? Will test it today.

[metalefty]

Yes, #804.

[tfischer77]

Issue closed by #804.

[Suncatcher]

So we can safely connect via 32bit depth since today? which version should I update to, to have this fix?

[metalefty]

v0.9.3 will be released in a few days.