Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unset pixel buffer when x0vncserver client disconnects. #515

Closed
wants to merge 2 commits into from

Conversation

michalsrb
Copy link
Contributor

In XDesktop::start() we allocate pixel buffer and set it as the backend to the given VNCServer.
In XDesktop::stop() we deallocate the buffer, so we must unset it from the VNCServer as well.
Otherwise the VNCServer could try to access it and crash, for example in deferred update.

This prevents crashes after client disconnects from x0vncserver. Backtrace of such crash looks like this:

Program received signal SIGSEGV, Segmentation fault.
0x0000000100574c50 in ?? ()
(gdb) bt
#0  0x0000000100574c50 in ?? ()
#1  0x00000001000385de in rfb::VNCServerST::writeUpdate (this=this@entry=0x7fffffffdfc0) at /usr/src/debug/tigervnc-1.8.0/common/rfb/VNCServerST.cxx:595
#2  0x00000001000386c6 in rfb::VNCServerST::handleTimeout (this=0x7fffffffdfc0, t=<optimized out>) at /usr/src/debug/tigervnc-1.8.0/common/rfb/VNCServerST.cxx:514
#3  0x00000001000360cd in rfb::Timer::checkTimeouts () at /usr/src/debug/tigervnc-1.8.0/common/rfb/Timer.cxx:76
#4  0x0000000100036ae7 in rfb::VNCServerST::checkTimeouts (this=0x7fffffffdfc0) at /usr/src/debug/tigervnc-1.8.0/common/rfb/VNCServerST.cxx:203
#5  0x000000010002733f in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/tigervnc-1.8.0/unix/x0vncserver/x0vncserver.cxx:555

@CendioOssman
Copy link
Member

The changes in VNCServertST.cxx aren't described in the commit message. Are those an accident or what is the motivation there?

@michalsrb
Copy link
Contributor Author

I should have described it properly.
Originally the VNCServertST::setPixelBuffer(PixelBuffer* pb_) would not do anything when pb_ is 0. The changes make it actually set the pb to 0 and stop the deferred update timer.

@CendioOssman
Copy link
Member

Alright. Could you update the commit message? Or put that part in a separate commit?

Originally calling VNCServertST::setPixelBuffer(PixelBuffer* pb_) with pb_=0 would do nothing.
With this change pb will be set to 0 and deferred update timer will be stopped.
In XDesktop::start() we allocate pixel buffer and set it as the backend to the given VNCServer.
In XDesktop::stop() we deallocate the buffer, so we must unset it from the VNCServer as well.
Otherwise the VNCServer could try to access it and crash, for example in deferred update.
@michalsrb
Copy link
Contributor Author

Ok, split into two commits now.

@@ -313,6 +313,8 @@ void VNCServerST::setPixelBuffer(PixelBuffer* pb_, const ScreenSet& layout)
screenLayout = layout;

if (!pb) {
stopFrameClock();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps it is better to do this when the desktop stops? I.e. in VNCServerST::removeSocket().

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have placed it there because it is symetric with the startFrameClock() - it is called when pixel buffer is set, so stopFrameClock() is called when pixel buffer is unset.

Currently the call chain is VNCServerST::removeSocket() -> rfb::SDesktop::stop() (in this case x0vncserver's XDesktop::stop) -> VNCServerST::setPixelBuffer(0). But other implementation of rfb::SDesktop::stop may not want to unset the pixel buffer there, so the timer would stop and not start again for next connection? (Not sure.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, other server may not call VNCServerST::setPixelBuffer(0). However that won't wedge things as the clock will start again once new changes happen. The only downside is that the clock will continue to run for a bit after the desktop is "stopped", which might be a bit confusing. My idea was to make it more symmetrical with SDesktop::start()/stop().


// Check that the screen layout is still valid
if (!layout.validate(pb_->width(), pb_->height())) {
if (pb_ && !layout.validate(pb_->width(), pb_->height())) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably reset the screen layout if we are removing the pixel buffer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'll add that.

@CendioOssman
Copy link
Member

I noticed that WinVNC calls setPixelBuffer(0). I wonder why we haven't seen any complaints from that.

@CendioOssman
Copy link
Member

Ping @michalsrb

@CendioOssman
Copy link
Member

Updated and merged to master now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants