Pixel byte order regression with commit cc647db

[gearnau]

Hello,

I recently moved to TigerVNC1.10 and I am afraid that a regression was introduced, when compared to 1.7
It used to be OK when I was doing this: SSH from intel to SPARC system, X11 tunelling. Run vncviewer to localhost

Now with 1.10, I get colours messed up the same way as shown in #738

Originally posted by @gearnau in #738 (comment)

[gearnau]

How to reproduce:

1. login on x86 system
2. ssh -X <sparc>
3. vncserver
4. vncviewer 0:1

Colours are OK with 1.7 and not OK with 1.10
Both vnc server and viewer are running on SPARC
However, X11 is on x86
Both 1.7 and 1.10 show that they opt for little endian rgb888 but colours are garbled on 1.10

[johnmartin-oracle]

Backing out changeset cc647db, in particular the change to Surface_X11.cxx, does restore the correct behavior.

I need more time to digest this problem but I believe the fundamental issue is the changeset forces a compile
time assumption that the byte-order of the system on which vncviewer is run matches the system on which
the X window is rendered. In the failing case both vncviewer and vncserver are run on a big endian SPARC
but the rendered window is on a little endian x86:

(x86 little endian) ssh -X ---- X forwarding -----> (SPARC big endian) vncviewer ---> vncserver/Xvnc

[CendioOssman]

You guys are really trying to create the worst conditions possible for this code. :)

Unfortunately this seems to be a bit of a pain to figure out as the X Render specification is not very clear on how you determine byte ordering. I'm not sure I'll have time to look at it before a 1.12.0 release.

(any particular reason you are using X forwarding, which is slow, instead of running vncviewer on your client?)

[gearnau]

Simple reason: the thin clients from which we access have ssh and X forwarding but no "onboard" tigervnc.

[johnmartin-oracle]

Can we just use the X Visual mask to determine BGRA ordering at runtime?

johnmartin-oracle@adec29a

[CendioOssman]

XRender doesn't use visuals, so I'm afraid we cannot assume they are the same. I think someone needs to dig around in the X server code (or ask on the X mailing list). I'd guess that ImageByteOrder() is also relevant for XRender, but this needs to be confirmed.

[CendioOssman]

Fixed by #1084.