Error "read (104: Connection reset by peer)" after calling SendExtDesktopSize()

[ivanshchitov]

Hello!
I am developing the VNC-client for Sailfish OS using Qt and faced with the following problem.
After calling the SendExtDesktopSize() function after retrieving first frames the server connection fails and I have the messages "read (104: Connection reset by peer)" and "VNC connection closed" in logs.

Part of my code.

Code inside the client-class constructor:

• Initializes and sets up the rfbClient instance,
• Sets up signals of timer to run retrieving the rfb-messages every 10 millis.

// init rfbClient instance
m_rfbClient = rfbGetClient(8, 3, 4);
m_rfbClient->GetCredential = GetCredential;
m_rfbClient->GetPassword = GetPassword;
m_rfbClient->MallocFrameBuffer = MallocFrameBuffer;
m_rfbClient->GotFrameBufferUpdate = GotFrameBufferUpdate;
m_rfbClient->frameBuffer = nullptr;
m_rfbClient->canHandleNewFBSize = true;

// set up signals and slots of timer to retrieve the rfb-messages
m_updateTimer.setInterval(10);
connect(&m_updateTimer, &QTimer::timeout, this, &VNCClient::runWaitingRfbMessage);
connect(this, SIGNAL(connected(QSize)), &m_updateTimer, SLOT(start()));
connect(this, &VNCClient::disconnected, &m_updateTimer, &QTimer::stop);

Method to perform a connection to the VNC-server:

void VNCClient::connectToHost(const QString &hostName, const QString &port, const QString &username, const QString &password)
{
    m_userName = username;
    m_password = password;
    try {
        if (!ConnectToRFBServer(m_rfbClient, hostName.toUtf8(), port.toInt()))
            throw QStringLiteral("Unable to connect to VNC server %1:%2").arg(hostName, port);
 
        if (!InitialiseRFBConnection(m_rfbClient))
        throw QStringLiteral("Connection initialization failed");

        m_rfbClient->appData.encodingsString = "ultra";
        m_rfbClient->appData.qualityLevel = 5;
        m_rfbClient->appData.compressLevel = 9;
 
        if (!SetFormatAndEncodings(m_rfbClient))
            throw QStringLiteral("Set format and encoding failed");
 
        if (m_rfbClient->width == 0 && m_rfbClient->height == 0) {
            m_rfbClient->width = m_rfbClient->si.framebufferWidth;
            m_rfbClient->height = m_rfbClient->si.framebufferHeight;
            m_rfbClient->screen.width = m_rfbClient->si.framebufferWidth;
            m_rfbClient->screen.height = m_rfbClient->si.framebufferHeight;
       }
 
       MallocFrameBuffer(m_rfbClient);
       requestUpdate();
       emit connected(QSize(m_rfbClient->width, m_rfbClient->height));
    } catch (const QString &error) {
        emit errorOccured(error);
    }
}

Method to disconnect from the server:

void VNCClient::disconnectFromHost()
{
    if (m_rfbClient->sock) {
        close(m_rfbClient->sock);
        m_rfbClient->sock = 0;
    }
    if (m_rfbClient->frameBuffer) {
        free(m_rfbClient->frameBuffer);
        m_rfbClient->frameBuffer = nullptr;
    }
    if (m_lastErrorMessage.isEmpty()) {
        emit disconnected("", "");
    } else {
         qWarning() << "VNC Error" << m_lastErrorMessage;
         emit disconnected(m_lastErrorMessage, QStringLiteral("vnc"));
    }
}

Method to retrieve the rfb-messages. It calls requestUpdate() to send a message to update framebuffer in a separate thread via QTimer.

void VNCClient::runWaitingRfbMessage()
{
    if (WaitForMessage(m_rfbClient, 0)) {
        if (HandleRFBServerMessage(m_rfbClient)) {
            QTimer::singleShot(25, Qt::CoarseTimer, this, &VNCClient::requestUpdate);
        }
    }
}

Method to send a message to update the framebuffer.

void VNCClient::requestUpdate()
{
    m_rfbClient->updateRect.x = 0;
    m_rfbClient->updateRect.y = 0;
    m_rfbClient->updateRect.w = m_rfbClient->width;
    m_rfbClient->updateRect.h = m_rfbClient->height;
    SendIncrementalFramebufferUpdateRequest(m_rfbClient);
}

Method initializes the framebuffer.

rfbBool VNCClient::MallocFrameBuffer(rfbClient *rfbClient)
{
    if (rfbClient->frameBuffer)
        free(rfbClient->frameBuffer);
    rfbClient->frameBuffer = (uint8_t *)malloc(rfbClient->width * rfbClient->height * vncBytesPerPixel);
    return TRUE;
}

Also for testing I call SendExtDesktopSize(m_rfbClient, m_width, m_height); through clicking a button inside an app when the server gives me a picture and the picture is displayed. And when the call is finished I got an error that described above.

For running my client-app I use INOI R7.
The problem is reproduced when it's used Vine VNC Server for macOS and TightVNC for Windows as servers.
The problem is not reproduced when it's used UltraVNC for Windows as a server.

Can you please tell me what I'm doing wrong?

[bk138]

Hi @ivanshchitov, I didn't have the time to read all the posted code but might help with some pointers:

• Why do you call SendExtDesktopSize() in the first place? (I've rarely seen it and some severs do not support it at all)
• You might want to look at the server logs in question to see what's going on on this side.
• Also, you might want to use a tool like WireShark to see what's going on on the network, who closes the connection and why.
• You could use a client from the examples and add the call, this way you'd be sure to not do anything wrong in your implementation.

Also, please post general questions in https://gitter.im/LibVNC/libvncserver - we can leave the conversation here for this topic but in the future that's the right place :-)

[ivanshchitov]

@bk138,

• Actually I want to use SendExtDesktopSize() to set up a screen resolution on the VNC-server. It works for the UltraVNC server, but don't for TightVNC and Vine VNC Server.
• I found the server logs and there is a message "ERROR: Client Sent Message: unknown message type 251". So, this is not supported for the server as I understand.
• I tried to run the SDLvncviewer example. The example uses SendExtDesktopSize() and connection doesn't fail, the call just ignored.

So, if the SendExtDesktopSize() call is not supported for some servers, is there a way (some API) to determine that it is not supported?

Thank's for such quick answers.

[RokerHRO]

So, if the SendExtDesktopSize() call is not supported for some servers, is there a way (some API) to determine that it is not supported?

Sure. At least at the protocol level: The server must announce in its server capabilities in the initial connection handshake that it is able to generate "DesktopSize Pseudo-encoding" updates and understands the SetDesktopSize message type:

https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#id48

"[…] If a server supports the ExtendedDesktopSize encoding, it must also have basic support for the SetDesktopSize message although it may deny all requests to change the screen layout."

I don't know how these "capabilities" can be requested via API call to the VNC library, but I hope my answer might help you to look for the right keywords in the library documentation. :-)

[bk138]

libvncclient should keep track of server caps and not send stuff the server doesn't understand, I'll have a look.

[ivanshchitov]

@bk138,

I found such solution to determine that SendExtDesktopSize() is not supported:

bool res = SupportsClient2Server(m_rfbClient, rfbSetDesktopSize);

Is it a correct way?

[bk138]

Yes, that should be done in the lib. And set by the lib.

[ivanshchitov]

Thank you very much! I think this issue can be closed.

[bk138]

No, please leave it open as it might represent a valid defect.