Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix out-of-bounds reads when using OpenEXR decreasingY lineOrder. #4215

Merged

Conversation

acolwell
Copy link
Contributor

@acolwell acolwell commented Apr 3, 2024

Description

This change fixes out-of-bounds reads and incorrect scanline ordering when OpenEXR's decreasingY lineOrder mode is used.

OpenEXR expects to process scanlines in decreasing order when the decreasingY lineOrder option is enabled. This change fixes the code that provides the chunks of scanlines to OpenEXR so that the proper scanlines are available when it accesses the FrameBuffer OIIO created. The old code was providing incorrect scanlines AND setting up the FrameBuffer with incorrect pointers so out-of-bounds reads were occurring.

The key things to keep in mind are:

  • Chunks of scanlines need to be sent to OpenEXROutput::write_scanlines() from the bottom of the image to the top when decreasingY is enabled. If the chunk's scanline range does not contain the scanlines OpenEXR is expecting to fetch, then it can cause out-of-bound reads because the wrong information is used to construct the pointers for the Slices created when constructing a FrameBuffer.
  • OpenEXROutput::write_scanlines() does its own chunking of scanlines and this logic must also properly start from the bottom of the chunk data passed into this function when decreasingY is enabled.
  • The to_native_rectangle() call in OpenEXROutput::write_scanlines() may return a pointer to m_scratch if a format conversion is needed. This means the pointers passed to OpenEXR, in this situation, WILL NOT be a full frame buffer. This caused OpenEXR to do out-of-bounds reads because the computations used to create the Slices were using the wrong scanline number to adjust the pointer.
  • The progress callback computation needed to be modified to handle traversing scanlines in decreasing order.

Tests

I added an openexr-decreasingY test to verify that the code no longer crashes because of the out-of-bound reads and it also verifies that the decreasingY images match the increasingY images.

Checklist:

  • I have read the contribution guidelines.
  • I have updated the documentation, if applicable.
  • I have ensured that the change is tested somewhere in the testsuite
    (adding new test cases if necessary).
  • If I added or modified a C++ API call, I have also amended the
    corresponding Python bindings (and if altering ImageBufAlgo functions, also
    exposed the new functionality as oiiotool options).
  • My code follows the prevailing code style of this project. If I haven't
    already run clang-format before submitting, I definitely will look at the CI
    test that runs clang-format and fix anything that it highlights as being
    nonconforming.

OpenEXR expects to process scanlines in decreasing order when the
decreasingY lineOrder option is enabled. This change fixes the code
that provides the chunks of scanlines to OpenEXR so that the proper
scanlines are available when it accesses the FrameBuffer OIIO created.
The old code was providing incorrect scanlines AND setting up the
FrameBuffer with incorrect pointers so out-of-bounds reads were
occurring.

Signed-off-by: Aaron Colwell <300262+acolwell@users.noreply.github.com>
Copy link

linux-foundation-easycla bot commented Apr 3, 2024

CLA Signed

The committers listed above are authorized under a signed CLA.

Signed-off-by: Aaron Colwell <300262+acolwell@users.noreply.github.com>
Copy link
Collaborator

@lgritz lgritz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending the few comments I had about how to make the for loop iteration maybe a little less confusing. Other than that, looks good. Thanks for adding a careful test as well.

src/libOpenImageIO/imagebuf.cpp Outdated Show resolved Hide resolved
src/libOpenImageIO/imageoutput.cpp Show resolved Hide resolved
Comment on lines 531 to 533
for (int y = yStart; ((isDecreasingY && y >= 0)
|| (!isDecreasingY && y < m_spec.height))
&& ok;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment as above regarding ...; y != yEnd; ... would be simpler, with yEnd = yStart + yDelta

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added similar fix here.

src/openexr.imageio/exroutput.cpp Outdated Show resolved Hide resolved
Signed-off-by: Aaron Colwell <300262+acolwell@users.noreply.github.com>
Copy link
Contributor Author

@acolwell acolwell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed all comments. Thanks for the review.

src/libOpenImageIO/imagebuf.cpp Outdated Show resolved Hide resolved
src/libOpenImageIO/imageoutput.cpp Show resolved Hide resolved
Comment on lines 531 to 533
for (int y = yStart; ((isDecreasingY && y >= 0)
|| (!isDecreasingY && y < m_spec.height))
&& ok;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added similar fix here.

src/openexr.imageio/exroutput.cpp Outdated Show resolved Hide resolved
Signed-off-by: Aaron Colwell <300262+acolwell@users.noreply.github.com>
@acolwell
Copy link
Contributor Author

I'm not really sure what to do about the 1 failed builder. It doesn't appear to be related to my change. Unfortunately, I'm not familiar with fmt or the related cmake magic that appears to be including it so I don't have a suggested fix.

@lgritz
Copy link
Collaborator

lgritz commented Apr 12, 2024

That's not related to this PR. There is one build we do that uses the current top-of-tree of several major dependencies. It often fails because one of those dependencies has checked a bug into their own tree.

Copy link
Collaborator

@lgritz lgritz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the patch!

@lgritz lgritz merged commit 7f41a6a into AcademySoftwareFoundation:master Apr 12, 2024
23 of 24 checks passed
lgritz pushed a commit to lgritz/OpenImageIO that referenced this pull request Apr 13, 2024
…lineOrder. (AcademySoftwareFoundation#4215)

This change fixes out-of-bounds reads and incorrect scanline ordering
when OpenEXR's decreasingY lineOrder mode is used.

OpenEXR expects to process scanlines in decreasing order when the
decreasingY lineOrder option is enabled. This change fixes the code that
provides the chunks of scanlines to OpenEXR so that the proper scanlines
are available when it accesses the FrameBuffer OIIO created. The old
code was providing incorrect scanlines AND setting up the FrameBuffer
with incorrect pointers so out-of-bounds reads were occurring.

The key things to keep in mind are:
- Chunks of scanlines need to be sent to
OpenEXROutput::write_scanlines() from the bottom of the image to the top
when decreasingY is enabled. If the chunk's scanline range does not
contain the scanlines OpenEXR is expecting to fetch, then it can cause
out-of-bound reads because the wrong information is used to construct
the pointers for the Slices created when constructing a FrameBuffer.
- OpenEXROutput::write_scanlines() does its own chunking of scanlines
and this logic must also properly start from the bottom of the chunk
data passed into this function when decreasingY is enabled.
- The to_native_rectangle() call in OpenEXROutput::write_scanlines() may
return a pointer to m_scratch if a format conversion is needed. This
means the pointers passed to OpenEXR, in this situation, _WILL NOT_ be a
full frame buffer. This caused OpenEXR to do out-of-bounds reads because
the computations used to create the Slices were using the wrong scanline
number to adjust the pointer.
- The progress callback computation needed to be modified to handle
traversing scanlines in decreasing order.

I added an openexr-decreasingY test to verify that the code no longer
crashes because of the out-of-bound reads and it also verifies that the
decreasingY images match the increasingY images.

---------

Signed-off-by: Aaron Colwell <300262+acolwell@users.noreply.github.com>
lgritz pushed a commit to lgritz/OpenImageIO that referenced this pull request Apr 13, 2024
…lineOrder. (AcademySoftwareFoundation#4215)

This change fixes out-of-bounds reads and incorrect scanline ordering
when OpenEXR's decreasingY lineOrder mode is used.

OpenEXR expects to process scanlines in decreasing order when the
decreasingY lineOrder option is enabled. This change fixes the code that
provides the chunks of scanlines to OpenEXR so that the proper scanlines
are available when it accesses the FrameBuffer OIIO created. The old
code was providing incorrect scanlines AND setting up the FrameBuffer
with incorrect pointers so out-of-bounds reads were occurring.

The key things to keep in mind are:
- Chunks of scanlines need to be sent to
OpenEXROutput::write_scanlines() from the bottom of the image to the top
when decreasingY is enabled. If the chunk's scanline range does not
contain the scanlines OpenEXR is expecting to fetch, then it can cause
out-of-bound reads because the wrong information is used to construct
the pointers for the Slices created when constructing a FrameBuffer.
- OpenEXROutput::write_scanlines() does its own chunking of scanlines
and this logic must also properly start from the bottom of the chunk
data passed into this function when decreasingY is enabled.
- The to_native_rectangle() call in OpenEXROutput::write_scanlines() may
return a pointer to m_scratch if a format conversion is needed. This
means the pointers passed to OpenEXR, in this situation, _WILL NOT_ be a
full frame buffer. This caused OpenEXR to do out-of-bounds reads because
the computations used to create the Slices were using the wrong scanline
number to adjust the pointer.
- The progress callback computation needed to be modified to handle
traversing scanlines in decreasing order.

I added an openexr-decreasingY test to verify that the code no longer
crashes because of the out-of-bound reads and it also verifies that the
decreasingY images match the increasingY images.

---------

Signed-off-by: Aaron Colwell <300262+acolwell@users.noreply.github.com>
lgritz pushed a commit to lgritz/OpenImageIO that referenced this pull request Apr 15, 2024
…lineOrder. (AcademySoftwareFoundation#4215)

This change fixes out-of-bounds reads and incorrect scanline ordering
when OpenEXR's decreasingY lineOrder mode is used.

OpenEXR expects to process scanlines in decreasing order when the
decreasingY lineOrder option is enabled. This change fixes the code that
provides the chunks of scanlines to OpenEXR so that the proper scanlines
are available when it accesses the FrameBuffer OIIO created. The old
code was providing incorrect scanlines AND setting up the FrameBuffer
with incorrect pointers so out-of-bounds reads were occurring.

The key things to keep in mind are:
- Chunks of scanlines need to be sent to
OpenEXROutput::write_scanlines() from the bottom of the image to the top
when decreasingY is enabled. If the chunk's scanline range does not
contain the scanlines OpenEXR is expecting to fetch, then it can cause
out-of-bound reads because the wrong information is used to construct
the pointers for the Slices created when constructing a FrameBuffer.
- OpenEXROutput::write_scanlines() does its own chunking of scanlines
and this logic must also properly start from the bottom of the chunk
data passed into this function when decreasingY is enabled.
- The to_native_rectangle() call in OpenEXROutput::write_scanlines() may
return a pointer to m_scratch if a format conversion is needed. This
means the pointers passed to OpenEXR, in this situation, _WILL NOT_ be a
full frame buffer. This caused OpenEXR to do out-of-bounds reads because
the computations used to create the Slices were using the wrong scanline
number to adjust the pointer.
- The progress callback computation needed to be modified to handle
traversing scanlines in decreasing order.

I added an openexr-decreasingY test to verify that the code no longer
crashes because of the out-of-bound reads and it also verifies that the
decreasingY images match the increasingY images.

---------

Signed-off-by: Aaron Colwell <300262+acolwell@users.noreply.github.com>
@acolwell acolwell deleted the fix_decreasingY_crashes branch April 16, 2024 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants