Skip to content

Building libvncserver

Sorecchione07435 edited this page Nov 10, 2026 · 8 revisions

Building libvncserver on Windows

This guide explains how to compile libvncserver from source on Windows using Microsoft Visual Studio 2022 and CMake.

libvncserver provides the VNC server functionality required by the VNC-enabled VirtualBox build.

1. Requirements

Before compiling libvncserver, make sure the following tools are installed:

  • Microsoft Visual Studio 2022 Build Tools.
  • MSVC v143 C++ build tools.
  • Windows SDK.
  • CMake.
  • Git.
  • Required third-party dependencies, depending on the selected build configuration.

Refer to Build Environment Setup for installation instructions.

2. Download the Source Code

Open a command prompt and clone the official libvncserver repository:

git clone https://github.com/LibVNC/libvncserver.git

Navigate to the source directory:

cd libvncserver

Note

Use the libvncserver version tested and supported by this project. If a specific release or commit is required, check it out before compiling.

For example:

git checkout <tested-version-or-commit>

3. Configure the Build

Create a separate build directory:

mkdir build

Configure the project using CMake and the Visual Studio 2022 generator:

cmake -S . -B build -G "Visual Studio 17 2022" -A x64 ^
    -DCMAKE_BUILD_TYPE=Release ^
    -DBUILD_SHARED_LIBS=OFF ^
    -DCMAKE_INSTALL_PREFIX=C:/Libraries/libvncserver ^
    -DWITH_LIBVNCSERVER=ON ^
    -DWITH_LIBVNCCLIENT=OFF ^
    -DWITH_EXAMPLES=OFF ^
    -DWITH_TESTS=OFF ^
    -DWITH_QT=OFF ^
    -DWITH_GTK=OFF ^
    -DWITH_SDL=OFF

Configuration options

Option Description
-G "Visual Studio 17 2022" Selects the Visual Studio 2022 generator.
-A x64 Generates a 64-bit build.
-DCMAKE_BUILD_TYPE=Release Requests a Release configuration. Visual Studio is a multi-configuration generator, so the actual configuration is selected during compilation.
-DBUILD_SHARED_LIBS=OFF Requests static libraries instead of shared libraries.
-DCMAKE_INSTALL_PREFIX Specifies the destination directory for the installed library.
-DWITH_LIBVNCSERVER=ON Enables the VNC server library.
-DWITH_LIBVNCCLIENT=OFF Disables the VNC client library.
-DWITH_EXAMPLES=OFF Disables example applications.
-DWITH_TESTS=OFF Disables test programs.
-DWITH_QT=OFF Disables the Qt example client.
-DWITH_GTK=OFF Disables the GTK example client.
-DWITH_SDL=OFF Disables the SDL example client.

Note

The options above are intended as a starting configuration for a server-only build. Additional features and > dependencies may be required depending on the VNC functionality enabled by VirtualBox.

4. Compile libvncserver

After CMake successfully generates the Visual Studio project files, compile the library:

cmake --build build --config Release

To use multiple CPU cores:

cmake --build build --config Release --parallel

CMake will invoke the selected Visual Studio compiler and generate the library binaries.

5. Install the Library

Once compilation has completed successfully, install the generated files:

cmake --install build --config Release

The files will be installed under the directory specified by CMAKE_INSTALL_PREFIX.

For the example configuration:

C:\Libraries\libvncserver

The installation directory should contain the generated library files, headers and CMake package configuration files.

6. Expected Directory Structure

The resulting installation directory should contain a structure similar to:

libvncserver/
│
├── bin/
│
├── include/
│   └── rfb/
│       ├── rfb.h
│       ├── rfbproto.h
│       └── ...
│
└── lib/
    └── cmake/
        └── LibVNCServer/

The exact contents and locations may vary depending on the libvncserver version and build configuration.

7. Integrating with VirtualBox

After installing libvncserver, configure the VirtualBox build environment to locate its headers and libraries.

The required include and library directories must be made available to the VirtualBox build system.

Important

The exact configuration variables depend on the project's KMK modifications and integration scripts. Use the variables defined by this repository rather than assuming that standard VirtualBox configurations automatically detect libvncserver.

Make sure that:

  • The library architecture matches the VirtualBox build architecture (x64).
  • The library was compiled using a compatible MSVC toolset.
  • The headers and compiled binaries come from the same libvncserver version.
  • All required transitive dependencies are available.
  • Static or dynamic linking is consistent with the project's integration configuration.

8. Troubleshooting

Problem Possible cause
CMake cannot find a dependency A required third-party library is missing or its location is not configured.
Visual Studio generator not found Visual Studio 2022 C++ build tools are not installed correctly.
Architecture mismatch The library and VirtualBox are built for different architectures.
Linker errors Missing libraries, incompatible configurations or unresolved dependencies.
Missing headers The installation step was not completed or the include path is incorrect.
DLL missing at runtime A required shared dependency is not available.

9. References

10. Next Steps

After successfully compiling and installing libvncserver, proceed to: