libvncserver: close resources on error paths

[marcofortina]

Summary

Fixes the three resource cleanup paths reported by static analysis in #660.

Changes

• Close the directory/search handle when sending the initial file-transfer directory response fails.
• Return RFB_INVALID_SOCKET if restoring a successfully connected socket to blocking mode fails after the socket has been closed.
• Release the partially initialized client allocation, peer host string, and scaled-screen reference when rfbSetNonBlocking() fails while accepting a new TCP client.

Validation

Tested with a minimal local CMake configuration:

cmake -S . -B build-660-patch \
  -DWITH_EXAMPLES=OFF \
  -DWITH_TESTS=ON \
  -DWITH_OPENSSL=OFF \
  -DWITH_GNUTLS=OFF \
  -DWITH_GCRYPT=OFF \
  -DWITH_SDL=OFF \
  -DWITH_GTK=OFF \
  -DWITH_QT=OFF \
  -DWITH_FFMPEG=OFF \
  -DWITH_XCB=OFF \
  -DWITH_LIBSSHTUNNEL=OFF \
  -DWITH_SYSTEMD=OFF \
  -DCMAKE_BUILD_TYPE=Debug
cmake --build build-660-patch --parallel 1
ctest --test-dir build-660-patch --output-on-failure

Result:

100% tests passed, 0 tests failed out of 5

Notes

No new regression test is included because the three paths require forcing low-level OS/API failures (opendir/file-transfer send failure, sock_set_nonblocking() failure after connect, and accepted-client nonblocking setup failure). The patch keeps the changes surgical and directly closes the resources reported by static analysis.

Closes #660.