-
Notifications
You must be signed in to change notification settings - Fork 566
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix out-of-bounds reads when using OpenEXR decreasingY lineOrder. #4215
Fix out-of-bounds reads when using OpenEXR decreasingY lineOrder. #4215
Conversation
OpenEXR expects to process scanlines in decreasing order when the decreasingY lineOrder option is enabled. This change fixes the code that provides the chunks of scanlines to OpenEXR so that the proper scanlines are available when it accesses the FrameBuffer OIIO created. The old code was providing incorrect scanlines AND setting up the FrameBuffer with incorrect pointers so out-of-bounds reads were occurring. Signed-off-by: Aaron Colwell <300262+acolwell@users.noreply.github.com>
Signed-off-by: Aaron Colwell <300262+acolwell@users.noreply.github.com>
31e3689
to
3a07454
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM pending the few comments I had about how to make the for
loop iteration maybe a little less confusing. Other than that, looks good. Thanks for adding a careful test as well.
src/libOpenImageIO/imageoutput.cpp
Outdated
for (int y = yStart; ((isDecreasingY && y >= 0) | ||
|| (!isDecreasingY && y < m_spec.height)) | ||
&& ok; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same comment as above regarding ...; y != yEnd; ...
would be simpler, with yEnd = yStart + yDelta
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added similar fix here.
Signed-off-by: Aaron Colwell <300262+acolwell@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Addressed all comments. Thanks for the review.
src/libOpenImageIO/imageoutput.cpp
Outdated
for (int y = yStart; ((isDecreasingY && y >= 0) | ||
|| (!isDecreasingY && y < m_spec.height)) | ||
&& ok; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added similar fix here.
Signed-off-by: Aaron Colwell <300262+acolwell@users.noreply.github.com>
I'm not really sure what to do about the 1 failed builder. It doesn't appear to be related to my change. Unfortunately, I'm not familiar with fmt or the related cmake magic that appears to be including it so I don't have a suggested fix. |
That's not related to this PR. There is one build we do that uses the current top-of-tree of several major dependencies. It often fails because one of those dependencies has checked a bug into their own tree. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks for the patch!
7f41a6a
into
AcademySoftwareFoundation:master
…lineOrder. (AcademySoftwareFoundation#4215) This change fixes out-of-bounds reads and incorrect scanline ordering when OpenEXR's decreasingY lineOrder mode is used. OpenEXR expects to process scanlines in decreasing order when the decreasingY lineOrder option is enabled. This change fixes the code that provides the chunks of scanlines to OpenEXR so that the proper scanlines are available when it accesses the FrameBuffer OIIO created. The old code was providing incorrect scanlines AND setting up the FrameBuffer with incorrect pointers so out-of-bounds reads were occurring. The key things to keep in mind are: - Chunks of scanlines need to be sent to OpenEXROutput::write_scanlines() from the bottom of the image to the top when decreasingY is enabled. If the chunk's scanline range does not contain the scanlines OpenEXR is expecting to fetch, then it can cause out-of-bound reads because the wrong information is used to construct the pointers for the Slices created when constructing a FrameBuffer. - OpenEXROutput::write_scanlines() does its own chunking of scanlines and this logic must also properly start from the bottom of the chunk data passed into this function when decreasingY is enabled. - The to_native_rectangle() call in OpenEXROutput::write_scanlines() may return a pointer to m_scratch if a format conversion is needed. This means the pointers passed to OpenEXR, in this situation, _WILL NOT_ be a full frame buffer. This caused OpenEXR to do out-of-bounds reads because the computations used to create the Slices were using the wrong scanline number to adjust the pointer. - The progress callback computation needed to be modified to handle traversing scanlines in decreasing order. I added an openexr-decreasingY test to verify that the code no longer crashes because of the out-of-bound reads and it also verifies that the decreasingY images match the increasingY images. --------- Signed-off-by: Aaron Colwell <300262+acolwell@users.noreply.github.com>
…lineOrder. (AcademySoftwareFoundation#4215) This change fixes out-of-bounds reads and incorrect scanline ordering when OpenEXR's decreasingY lineOrder mode is used. OpenEXR expects to process scanlines in decreasing order when the decreasingY lineOrder option is enabled. This change fixes the code that provides the chunks of scanlines to OpenEXR so that the proper scanlines are available when it accesses the FrameBuffer OIIO created. The old code was providing incorrect scanlines AND setting up the FrameBuffer with incorrect pointers so out-of-bounds reads were occurring. The key things to keep in mind are: - Chunks of scanlines need to be sent to OpenEXROutput::write_scanlines() from the bottom of the image to the top when decreasingY is enabled. If the chunk's scanline range does not contain the scanlines OpenEXR is expecting to fetch, then it can cause out-of-bound reads because the wrong information is used to construct the pointers for the Slices created when constructing a FrameBuffer. - OpenEXROutput::write_scanlines() does its own chunking of scanlines and this logic must also properly start from the bottom of the chunk data passed into this function when decreasingY is enabled. - The to_native_rectangle() call in OpenEXROutput::write_scanlines() may return a pointer to m_scratch if a format conversion is needed. This means the pointers passed to OpenEXR, in this situation, _WILL NOT_ be a full frame buffer. This caused OpenEXR to do out-of-bounds reads because the computations used to create the Slices were using the wrong scanline number to adjust the pointer. - The progress callback computation needed to be modified to handle traversing scanlines in decreasing order. I added an openexr-decreasingY test to verify that the code no longer crashes because of the out-of-bound reads and it also verifies that the decreasingY images match the increasingY images. --------- Signed-off-by: Aaron Colwell <300262+acolwell@users.noreply.github.com>
…lineOrder. (AcademySoftwareFoundation#4215) This change fixes out-of-bounds reads and incorrect scanline ordering when OpenEXR's decreasingY lineOrder mode is used. OpenEXR expects to process scanlines in decreasing order when the decreasingY lineOrder option is enabled. This change fixes the code that provides the chunks of scanlines to OpenEXR so that the proper scanlines are available when it accesses the FrameBuffer OIIO created. The old code was providing incorrect scanlines AND setting up the FrameBuffer with incorrect pointers so out-of-bounds reads were occurring. The key things to keep in mind are: - Chunks of scanlines need to be sent to OpenEXROutput::write_scanlines() from the bottom of the image to the top when decreasingY is enabled. If the chunk's scanline range does not contain the scanlines OpenEXR is expecting to fetch, then it can cause out-of-bound reads because the wrong information is used to construct the pointers for the Slices created when constructing a FrameBuffer. - OpenEXROutput::write_scanlines() does its own chunking of scanlines and this logic must also properly start from the bottom of the chunk data passed into this function when decreasingY is enabled. - The to_native_rectangle() call in OpenEXROutput::write_scanlines() may return a pointer to m_scratch if a format conversion is needed. This means the pointers passed to OpenEXR, in this situation, _WILL NOT_ be a full frame buffer. This caused OpenEXR to do out-of-bounds reads because the computations used to create the Slices were using the wrong scanline number to adjust the pointer. - The progress callback computation needed to be modified to handle traversing scanlines in decreasing order. I added an openexr-decreasingY test to verify that the code no longer crashes because of the out-of-bound reads and it also verifies that the decreasingY images match the increasingY images. --------- Signed-off-by: Aaron Colwell <300262+acolwell@users.noreply.github.com>
Description
This change fixes out-of-bounds reads and incorrect scanline ordering when OpenEXR's decreasingY lineOrder mode is used.
OpenEXR expects to process scanlines in decreasing order when the decreasingY lineOrder option is enabled. This change fixes the code that provides the chunks of scanlines to OpenEXR so that the proper scanlines are available when it accesses the FrameBuffer OIIO created. The old code was providing incorrect scanlines AND setting up the FrameBuffer with incorrect pointers so out-of-bounds reads were occurring.
The key things to keep in mind are:
Tests
I added an openexr-decreasingY test to verify that the code no longer crashes because of the out-of-bound reads and it also verifies that the decreasingY images match the increasingY images.
Checklist:
(adding new test cases if necessary).
corresponding Python bindings (and if altering ImageBufAlgo functions, also
exposed the new functionality as oiiotool options).
already run clang-format before submitting, I definitely will look at the CI
test that runs clang-format and fix anything that it highlights as being
nonconforming.