H264 for vncviewer

[mdevaev]

The Pi-KVM project has its own VNC server that is used to allow clients to access the hardware. At the moment we use JPEG compression, but now we have the encoder that implements hardware H264 compression with zero latency. Many of our users are asking to make support for H264 in VNC. However, no existing client supports H264, although this encoding is mentioned in rfbproto.

I suggest that we cooperate and do this work together. We can agree on an extension for the VNC protocol, I can make a server implementation, and your side would make an implementation for the client. As far as I understand, this is a feasible task, since the TigerVNC code is well structured and it all comes down to adding another decoder. You can take any library to decode H264, for example OpenH264 from Cisco (BSD license). After we have a reference implementation, I will negotiate with other mobile clients to support H264.

I understand that this is a low-priority job, so I think we can offer you a bounty for it. It would be great if someone from the core developers supported my initiative and could buy a beer after working on H264.

Discuss?

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[CendioOssman]

A bounty is always welcome, but unfortunately it won't likely attract us core developers as we are employed to work on TigerVNC. As such is more what's relevant for our use that drives us. And the licensing around H.264 is problematic enough that it's not terribly interesting from our point of view.

Not that this would block any PR from other people though. :)

Are there any support for open codecs in the hardware that could be used instead?

I also know that @dcommander has looked at H.264 for TurboVNC: TurboVNC/turbovnc#19

Many of our users are asking to make support for H264 in VNC. However, no existing client supports H264, although this encoding is mentioned in rfbproto.

Those encodings are registered by RealVNC, so I would guess it is supported by their proprietary client and server. Could be interesting to see how it works and see if we can interoperate before designing something from scratch.

[dcommander]

nVidia only supports H.264 and H.265 in their GPU video codec SDKs, although they also provide the nvJPEG library for encoding/decoding JPEG and JPEG2000 still images. My understanding is that MPEG-LA made H.264 royalty-free in perpetuity as long as the video stream is freely distributed, and H.265 is royalty-free for the first 100,000 devices. The intent of those patents is to make money on television and film, not on remote desktop software, so I seriously doubt that anyone would come after us for implementing those formats. There is also precedent for implementing H.264 in open source (libx264.) That being said, my research concluded that H.264 wasn't particularly useful without GPU acceleration. Basically, for the datasets that compressed significantly better with H.264 than with the TurboVNC codec (used by both TurboVNC and TigerVNC), the CPU overhead of libx264 was so high that the overall performance would have still been worse with H.264 except on extremely low-bandwidth networks. And using a proprietary GPU codec library in a GPL-licensed application would be tricky. The primary reasons why people want H.264 seem to be:

1. In some cases-- particularly for games and other video-like image workloads-- H.264 can use less bandwidth to achieve the same image quality (relative to the TurboVNC codec.) However, we haven't fully explored what is possible with progressive JPEG. The default progressive scan script in libjpeg-turbo (what you get if you call jpeg_simple_progression() or pass -progressive to cjpeg or tjbench) reduces compression performance by about 5x and decompression performance by about 3x in exchange for about 10% better compression, but a custom scan script can achieve nearly the same compression with about double the performance of the default scan script. There are ways to dial in the mix of JPEG vs. non-JPEG subrectangles in order to minimize bandwidth, and perhaps the resulting encoding method could be used with the highest compression level settings in TurboVNC or TigerVNC.
2. H.264 allows for GPU-based encoding, but this is problematic with Xvnc because the framebuffer is in main memory rather than on the GPU, and VNC doesn't encode the whole framebuffer in one throw. It's designed to encode and transmit only the parts of the framebuffer that have changed, and H.264 isn't particularly adept at this kind of encoding. (H.264 would expect you to feed it the entire framebuffer image and let it figure out what has changed, which is unnecessarily compute-intensive if you already know where the changes are, as Xvnc does.)

Where H.264 makes more sense is in VirtualGL, and in fact, I completed a proprietary contract last summer that involved developing an NVENC-based H.264 image transport for VirtualGL. But the people who are doing such things are dealing with singular applications that render their entire GUI using OpenGL and thus don't require a window manager or a full X server. The more generic case that Xvnc has to support isn't a good fit for H.264 at all, and as you can see from TurboVNC/turbovnc#19, it would take a lot of effort to implement it. Wayland potentially allows for remote display with a GPU-based framebuffer, and it doesn't have to deal with the non-image-based rendering primitives of X11, so it's generally a better fit for H.264 and similar codecs and a better fit for GPU encoding.

[mdevaev]

Implementing the server at the OS level and encoding the stream is a separate topic, as for me. In my case, the server is encoding from the video capture device, so I don't run into the problems that are described for xvnc. Of course, I don't deny the existence of this.

Now all I need is just a client implementation, since I have my own server. The implementation of the client is reduced to simple decoding of the frame, no matter what methods: software or using a hardware decoder. Given that modern machines are quite powerful, the first implementation can only have soft decoding. Also my experiments show a significant reduction in traffic, about 5-10 times depending on the bitrate, so this is definitely what I want for my users.

Now I propose to do two things:

• Agree on how we will transfer the h264 data. Perhaps it should be similar to Tight JPEG: message size and data after FramebufferUpdate message.
• Make a client implementation.

Then, if someone wants to implement this in their H264 servers in the future, they will have a ready-made protocol and client. I just want to get this process off the ground. My users use pikvm for emergency server management, and sometimes they have to do it from places where the Internet is very poor.

[dcommander]

5-10x improvement sounds bogus. Are you comparing apples to apples in terms of image quality?

Cendio has a sales-based business model and I have a services-based business model, but in both cases, we would have to make a business case for your proposed enhancements. In my case, I'm an OSS developer for hire, so I'll implement almost anything if someone pays me for my labor. However, adding H.264 support to my viewer wouldn't make good business sense unless I also added it to my server. Otherwise, I'd basically be encouraging my users to use someone else's server, and that will drive away potential business. But, as I already mentioned, adding H.264 support to my server is problematic.

I do agree that it would be nice to have a definitive RFB specification for H.264 encoding. I also agree with Pierre's suggestion that it needs to be compatible with RealVNC's existing implementation, if possible. Refer also to TurboVNC/turbovnc#18, which contains some H.264 discussion.

[mdevaev]

5-10x improvement sounds bogus. Are you comparing apples to apples in terms of image quality?

Ofc not. If I compare the same quality, the results will be identical to yours. In my case, I compared a video stream of minimal quality, which does not yet bleed from the eyes, that is, an acceptable picture for remote administration. When the quality is reduced, JPEG degrades quite significantly, while H264 is still acceptable. Hey, I'm not questioning your research :) I'm talking about my usecase.

Otherwise, I'd basically be encouraging my users to use someone else's server

I am in no way Cendio competitor. Their server is designed for the operating system. Mine is for the hardware without it. People don't need my product when their machine is fine, they use it when something is very badly broken. I offer a much lower degree of integration with the OS (by design), so I do not compete with the usual VNC.

Even if you can't integrate support into your servers right now, you can do it later, because at least we'll all get a working clients.

[dcommander]

To clarify, I am an independent OSS developer and am not affiliated with Cendio, although they do provide some money to one of my OSS projects (libjpeg-turbo.) I realize that you aren't my competitor or theirs, but RealVNC is. My concern is two-fold:

1. A viewer feature that isn't also present in our server would be hard to support, since our user community (which includes me) wouldn't be able to readily test the feature. I maintain libjpeg-turbo, VirtualGL, and TurboVNC as if they were fully integrated enterprise products, and that means I have to be mindful of scope creep and such. I can't introduce major new features unless I can fully support them, and fully supporting them requires "eating my own dog food" (using and testing the features myself on a regular basis.) Let's say I implemented H.264 encoding in the TurboVNC Viewer but later figured out that implementing it properly in the TurboVNC Server would require expanding the H.264 RFB encoding. That would be really hard to do if the encoding specification was already locked in by the viewer implementation.

2. At some point, I would have to document the viewer feature, and doing that would require explaining that it doesn't work with our server, which would also require explaining whose server it does work with. In our product space, that would mean giving free advertising to RealVNC, which isn't a good business move. (This is part and parcel of maintaining the open source projects as if they were fully integrated enterprise products.)

I am willing to participate in discussions regarding the best way to implement an H.264 RFB encoding, but I am unwilling to implement the feature in the TurboVNC Viewer unless it is also implemented in the TurboVNC Server.

[mdevaev]

I understand you. To be honest, I don't think compatibility with RealVNC is something necessary. Their client doesn't care about Tight support, anything other than their own server and the underlying VNC protocol. Of course, you can adapt to the proprietary protocol, but you can just support another type of compression, describe it in rfbproto and live happily ever after. I tell my users that RealVNC doesn't work, and it's true, then suggest that they use TigerVNC, TurboVNC, or any other open source. But if H264 appears in TigerVNC (let's say, okay, I'll add it myself) and it is accepted in the upstream, then I will have to recommend TigerVNC to my users for obvious reasons.

Okay, we'll agree on a standard. To avoid compatibility issues in the future, we can reserve some space for extensions.

@CendioOssman Do I understand you correctly: if I (or someone) bring you a high-quality patch for the client and a description of the new encoding for rfbproto, then you will accept it in the upstream and enable it in your builds?

[CendioOssman]

@CendioOssman Do I understand you correctly: if I (or someone) bring you a high-quality patch for the client and a description of the new encoding for rfbproto, then you will accept it in the upstream and enable it in your builds?

We'd accept the patch at least. Enabling it in the builds is a different matter as it depends on the availability of required libraries on the target platforms.

Note that the bar might be higher for a viewer-only patch since (as @dcommander points out) we won't be able to test it ourselves.

[mdevaev]

Sounds great. I have found a performer, and we will be able to provide a patch. I believe we will use ffmpeg as it is available on all platforms and portable. Anyway thank you. I'll be back with the prototype in a week or two and we'll discuss the protocol details. If you don't like it, we'll try to find an improved version together.

[mdevaev]

Here the prototype: #1194

The new encoding is registered under the number 50. The FramebufferUpdate consists of U32 data length, U32 with flags, and data.

The data is one or more H264 frames, which the client must parse with a regular H264 stream parser. This is necessary in a situation where the server works asynchronously and encodes frames even when the client has not sent a request. When the request is received, the server glues the I and P-frames together and sends them out. Usually each FramebufferUpdate will contain one I or P-frame, but if the client is slow, the server will glue the frames together so that the client doesn't miss anything. The client's task is to decode one or more frames inside itself and display the result on the screen.

Since H264 is a differential encoding protocol, it is necessary to store the rect context inside the decoder. The server can manage client contexts using two flags: 0x1 resets the specified context, 0x2 resets the contexts of all rects.

We did not use the existing H264 encoding number to avoid problems with RealVNC. The encoding is undocumented and completely under their control, so it's better to have an open implementation that won't be suddenly changed by a proprietary vendor.

The tests showed good image quality and stable operation of the client. Our server is already working with this. If you accept this PR, we will also implement H264 with this encoding in the TigerVNC server and send you a patch. And of course we'll document this in rfbproto

[mdevaev]

rfbproto: rfbproto/rfbproto#39