Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

opj_stream_get_number_byte_left: Assertion `p_stream->m_byte_offset >= 0' failed. #1151

Closed
andrewharvey opened this issue Sep 26, 2018 · 4 comments

Comments

@andrewharvey
Copy link

@andrewharvey andrewharvey commented Sep 26, 2018

Using the jp2 file from https://s3-ap-southeast-2.amazonaws.com/opendata-aerial-imagery/mga55_gda94_10cm_2018_CoM_true_ortho.zip

GDAL 2.3.1, released 2018/06/22

gdalwarp -t_srs 'EPSG:3857' -co COMPRESS=LZW mga55_gda94_10cm_2018_CoM_true_ortho.jp2 CoM_10cm_May_2018_EPSG3857.tiff

Using band 4 of source image as alpha.
Creating output file that is 86077P x 87017L.
Processing mga55_gda94_10cm_2018_CoM_true_ortho.jp2 [1/1] : 0gdalwarp: /home/mathieu/debian/openjpeg2/trunk/openjpeg-2.3.0/src/lib/openjp2/cio.c:586: opj_stream_get_number_byte_left: Assertion `p_stream->m_byte_offset >= 0' failed.
Aborted
@KermMartian

This comment has been minimized.

Copy link

@KermMartian KermMartian commented Sep 25, 2019

I replicate this issue.

Edit: in fact, I replicate this issue with the exact same input file, trying to perform exactly the same conversion. Impressive. Makes me slightly doubt the integrity of the file.

@rouault

This comment has been minimized.

Copy link
Collaborator

@rouault rouault commented Sep 26, 2019

The image is fine. This works with GDAL 3.0 (presumably 2.4 as well) with openjpeg >= 2.3.0. But you need a lot of memory (9 GB)

@KermMartian

This comment has been minimized.

Copy link

@KermMartian KermMartian commented Oct 3, 2019

Using a machine with 256GB of RAM:

user@machine:~$ cat /proc/meminfo | head -n 3
MemTotal:       264128544 kB
MemFree:        259033148 kB
MemAvailable:   261993584 kB

And GDAL 2.4.0 compiled against apt-installed libopenjp2-7-dev (that's OpenJPEG 2.3.0):

user@machine:~$ gdalwarp -t_srs 'EPSG:3857' -co COMPRESS=LZW mga55_gda94_10cm_2018_CoM_true_ortho/mga55_gda94_10cm_2018_CoM_true_ortho.jp2 CoM_10cm_May_2018_EPSG3857.tiff
Using band 4 of source image as alpha.
Creating output file that is 86077P x 87017L.
Processing mga55_gda94_10cm_2018_CoM_true_ortho.jp2 [1/1] :
0
gdalwarp: /build/openjpeg2-NZDJBV/openjpeg2-2.3.0/src/lib/openjp2/cio.c:586: opj_stream_get_number_byte_left: Assertion `p_stream->m_byte_offset >= 0' failed.
Aborted

Similarly:

user@machine:~$ opj_decompress -i ~/mga55_gda94_10cm_2018_CoM_true_ortho.jp2 -OutFor TIF -o ~/mga55_gda94_10cm_2018_CoM_true_ortho.tif

[INFO] Start to read j2k main header (2147).
[INFO] Main header has been correctly decoded.
[INFO] No decoded area parameters, set the decoded area to the whole image
[INFO] Psot value of the current tile-part is equal to zero, we assuming it is the last tile-part of the codestream.
[INFO] Header of tile 1 / 1 has been read.
[ERROR] Tiles don't all have the same dimension. Skip the MCT step.
[ERROR] Failed to decode.
[ERROR] Failed to decode tile 1/1
[ERROR] Failed to decode the codestream in the JP2 file
ERROR -> opj_decompress: failed to decode image!

I grabbed a fresh copy of the image to perform this test, then unzipped it, but perhaps that's still tripping me up in some way (e.g., a problem unzipping a 3+GB zip)? Does your sha256sum match?

f617ab7ae5471ccafd7cedcb71392db17ca5981ca8f5ae555306aa8d6ecbd670  mga55_gda94_10cm_2018_CoM_true_ortho.jp2
rouault added a commit that referenced this issue Oct 3, 2019
…ecode_real(): proper deal with a number of samples larger than 4 billion (refs #1151)
@rouault

This comment has been minimized.

Copy link
Collaborator

@rouault rouault commented Oct 3, 2019

OK, I finally reproduced the issue. A debug build of openjpeg was needed (at least one without -DNDEBUG), so that assertions are compiled in. The bug was actually in the GDAL OpenJPEG I/O layer, that wasn't ready to deal with reading single-tile codestreams larger than 2GB.

There was also an error in the MCT stage of openjpeg triggered by opj_decompress, which caused the "Tiles don't all have the same dimension" error. couldn't test the fix since I don't have actually enough RAM to reach that point, but there were a few 'obvious' uint32 overflows I've fixed. Anyway that part wasn't triggered by GDAL since it reads by much smaller chunks, whereas opj_decompress is brute force and decompresses the whole image at once.

rouault added a commit to OSGeo/gdal that referenced this issue Oct 3, 2019
…more than 2 GB at once (fixes uclouvain/openjpeg#1151)
rouault added a commit to OSGeo/gdal that referenced this issue Oct 3, 2019
…more than 2 GB at once (fixes uclouvain/openjpeg#1151)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.