Fix resize crash #3107
Merged
Fix resize crash #3107
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The xrdp_enc_data contains a union for handling surface commands and gfx commands. Memory processing is different for these two options. The default destructor for the encoder FIFO only knows about surface commands. Consequently, if the encoder has queued GFX data when the encoder is closed, the destructor processes the queued data as if it contained surface commands rather than GFX commands. This typically causes a SEGV as the drects field of the overlaid surface command structure is not pointing at anything valid when it is freed.
On a resize, the encoder is deleted. At present this is done by asking the encoder to exit, and then waiting a second. - On slower systems, a second may not be enough, and so the encoder data structures are freed while they are still being used by the encoder. - On quicker systems, resizes are delayed by hundreds of milliseconds longer than they need to be. This commit adds a wait object which the encoder can use to signal it has actually finished.
Nexarian
reviewed
Jun 3, 2024
| @@ -1457,6 +1470,7 @@ proc_enc_msg(void *arg) | |||
| } | |||
|
|
|||
| } /* end while (cont) */ | |||
| g_set_wait_obj(self->xrdp_encoder_term_done); | |||
There was a problem hiding this comment.
I'm honestly surprised this wasn't already here. Good fix!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes a couple of problems with GFX resizing.
See #3105 for the background.
There's also detail on the commit comments about the changes.