fix: race condition in TIFF reader #3772
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The idea is to keep fruitlessly reading from a corrupted file.
TIFFInput::read_native_tiles had a very subtle race condition. There is a section where it has a task_set of decompression tasks that are sent to the thread queue. When the method ends, the task_set goes out of scope, and its destructor will wait for all the remaining tasks to finish. All fine, right? Except that other variables in the function include unique_ptr's with allocated memory that the tasks need. So what can happen is that if the compiler generates the code such that the destructor of the unique_ptr happens before the destructor of the task_set, the memory that the unique_ptr held may be deallocated before some of the tasks still running in threads have finished. In practice, we seem only to hit this race condition for particular cases involving corrupted files, because the early `return` seems quite likely to run those destructors before all the threads are done, but it was possible to encounter it for valid files as well, if the timing is just right. Shocking that we never had it reported. The solution is just to have an explitit tasks.wait(), which will block until all the tasks have finished before hitting the actual end of the method and letting the other destructors to run. And for the "early return" case, change it to a break so that we go through the wait call at the end. I believe that this fix addresses TALOS-2023-1709 / CVE-2023-24472.
lgritz
added a commit
to lgritz/OpenImageIO
that referenced
this pull request
Feb 13, 2023
* test: Add a corrupted test file * fix(IC): upon tile read error, mark the file as broken The idea is to keep fruitlessly reading from a corrupted file. * fix: race condition in TIFF reader TIFFInput::read_native_tiles had a very subtle race condition. There is a section where it has a task_set of decompression tasks that are sent to the thread queue. When the method ends, the task_set goes out of scope, and its destructor will wait for all the remaining tasks to finish. All fine, right? Except that other variables in the function include unique_ptr's with allocated memory that the tasks need. So what can happen is that if the compiler generates the code such that the destructor of the unique_ptr happens before the destructor of the task_set, the memory that the unique_ptr held may be deallocated before some of the tasks still running in threads have finished. In practice, we seem only to hit this race condition for particular cases involving corrupted files, because the early `return` seems quite likely to run those destructors before all the threads are done, but it was possible to encounter it for valid files as well, if the timing is just right. Shocking that we never had it reported. The solution is just to have an explitit tasks.wait(), which will block until all the tasks have finished before hitting the actual end of the method and letting the other destructors to run. And for the "early return" case, change it to a break so that we go through the wait call at the end. I believe that this fix addresses TALOS-2023-1709 / CVE-2023-24472.
lgritz
added a commit
to lgritz/OpenImageIO
that referenced
this pull request
Feb 13, 2023
* test: Add a corrupted test file * fix(IC): upon tile read error, mark the file as broken The idea is to keep fruitlessly reading from a corrupted file. * fix: race condition in TIFF reader TIFFInput::read_native_tiles had a very subtle race condition. There is a section where it has a task_set of decompression tasks that are sent to the thread queue. When the method ends, the task_set goes out of scope, and its destructor will wait for all the remaining tasks to finish. All fine, right? Except that other variables in the function include unique_ptr's with allocated memory that the tasks need. So what can happen is that if the compiler generates the code such that the destructor of the unique_ptr happens before the destructor of the task_set, the memory that the unique_ptr held may be deallocated before some of the tasks still running in threads have finished. In practice, we seem only to hit this race condition for particular cases involving corrupted files, because the early `return` seems quite likely to run those destructors before all the threads are done, but it was possible to encounter it for valid files as well, if the timing is just right. Shocking that we never had it reported. The solution is just to have an explitit tasks.wait(), which will block until all the tasks have finished before hitting the actual end of the method and letting the other destructors to run. And for the "early return" case, change it to a break so that we go through the wait call at the end. I believe that this fix addresses TALOS-2023-1709 / CVE-2023-24472.
lgritz
added a commit
to lgritz/OpenImageIO
that referenced
this pull request
Mar 8, 2023
* test: Add a corrupted test file * fix(IC): upon tile read error, mark the file as broken The idea is to keep fruitlessly reading from a corrupted file. * fix: race condition in TIFF reader TIFFInput::read_native_tiles had a very subtle race condition. There is a section where it has a task_set of decompression tasks that are sent to the thread queue. When the method ends, the task_set goes out of scope, and its destructor will wait for all the remaining tasks to finish. All fine, right? Except that other variables in the function include unique_ptr's with allocated memory that the tasks need. So what can happen is that if the compiler generates the code such that the destructor of the unique_ptr happens before the destructor of the task_set, the memory that the unique_ptr held may be deallocated before some of the tasks still running in threads have finished. In practice, we seem only to hit this race condition for particular cases involving corrupted files, because the early `return` seems quite likely to run those destructors before all the threads are done, but it was possible to encounter it for valid files as well, if the timing is just right. Shocking that we never had it reported. The solution is just to have an explitit tasks.wait(), which will block until all the tasks have finished before hitting the actual end of the method and letting the other destructors to run. And for the "early return" case, change it to a break so that we go through the wait call at the end. I believe that this fix addresses TALOS-2023-1709 / CVE-2023-24472.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TIFFInput::read_native_tiles had a very subtle race condition. There
is a section where it has a task_set of decompression tasks that are
sent to the thread queue. When the method ends, the task_set goes out
of scope, and its destructor will wait for all the remaining tasks to
finish. All fine, right? Except that other variables in the function
include unique_ptr's with allocated memory that the tasks need. So
what can happen is that if the compiler generates the code such that
the destructor of the unique_ptr happens before the destructor of the
task_set, the memory that the unique_ptr held may be deallocated
before some of the tasks still running in threads have finished.
In practice, we seem only to hit this race condition for particular
cases involving corrupted files, because the early
returnseemsquite likely to run those destructors before all the threads are done,
but it was possible to encounter it for valid files as well, if the
timing is just right. Shocking that we never had it reported.
The solution is just to have an explitit tasks.wait(), which will
block until all the tasks have finished before hitting the actual end
of the method and letting the other destructors to run. And for the
"early return" case, change it to a break so that we go through the
wait call at the end.
I believe that this fix addresses TALOS-2023-1709 / CVE-2023-24472.
Also, in ImageCache, a fix fix where upon tile read error, mark the
file as broken. The idea is to keep fruitlessly reading from a corrupted
file.