opj_stream_get_number_byte_left: Assertion `p_stream->m_byte_offset >= 0' failed.

[andrewharvey]

Using the jp2 file from https://s3-ap-southeast-2.amazonaws.com/opendata-aerial-imagery/mga55_gda94_10cm_2018_CoM_true_ortho.zip

GDAL 2.3.1, released 2018/06/22

gdalwarp -t_srs 'EPSG:3857' -co COMPRESS=LZW mga55_gda94_10cm_2018_CoM_true_ortho.jp2 CoM_10cm_May_2018_EPSG3857.tiff

Using band 4 of source image as alpha.
Creating output file that is 86077P x 87017L.
Processing mga55_gda94_10cm_2018_CoM_true_ortho.jp2 [1/1] : 0gdalwarp: /home/mathieu/debian/openjpeg2/trunk/openjpeg-2.3.0/src/lib/openjp2/cio.c:586: opj_stream_get_number_byte_left: Assertion `p_stream->m_byte_offset >= 0' failed.
Aborted

[KermMartian]

I replicate this issue.

Edit: in fact, I replicate this issue with the exact same input file, trying to perform exactly the same conversion. Impressive. Makes me slightly doubt the integrity of the file.

[rouault]

The image is fine. This works with GDAL 3.0 (presumably 2.4 as well) with openjpeg >= 2.3.0. But you need a lot of memory (9 GB)

[KermMartian]

Using a machine with 256GB of RAM:

user@machine:~$ cat /proc/meminfo | head -n 3
MemTotal:       264128544 kB
MemFree:        259033148 kB
MemAvailable:   261993584 kB

And GDAL 2.4.0 compiled against apt-installed libopenjp2-7-dev (that's OpenJPEG 2.3.0):

user@machine:~$ gdalwarp -t_srs 'EPSG:3857' -co COMPRESS=LZW mga55_gda94_10cm_2018_CoM_true_ortho/mga55_gda94_10cm_2018_CoM_true_ortho.jp2 CoM_10cm_May_2018_EPSG3857.tiff
Using band 4 of source image as alpha.
Creating output file that is 86077P x 87017L.
Processing mga55_gda94_10cm_2018_CoM_true_ortho.jp2 [1/1] :
0
gdalwarp: /build/openjpeg2-NZDJBV/openjpeg2-2.3.0/src/lib/openjp2/cio.c:586: opj_stream_get_number_byte_left: Assertion `p_stream->m_byte_offset >= 0' failed.
Aborted

Similarly:

user@machine:~$ opj_decompress -i ~/mga55_gda94_10cm_2018_CoM_true_ortho.jp2 -OutFor TIF -o ~/mga55_gda94_10cm_2018_CoM_true_ortho.tif

[INFO] Start to read j2k main header (2147).
[INFO] Main header has been correctly decoded.
[INFO] No decoded area parameters, set the decoded area to the whole image
[INFO] Psot value of the current tile-part is equal to zero, we assuming it is the last tile-part of the codestream.
[INFO] Header of tile 1 / 1 has been read.
[ERROR] Tiles don't all have the same dimension. Skip the MCT step.
[ERROR] Failed to decode.
[ERROR] Failed to decode tile 1/1
[ERROR] Failed to decode the codestream in the JP2 file
ERROR -> opj_decompress: failed to decode image!

I grabbed a fresh copy of the image to perform this test, then unzipped it, but perhaps that's still tripping me up in some way (e.g., a problem unzipping a 3+GB zip)? Does your sha256sum match?

f617ab7ae5471ccafd7cedcb71392db17ca5981ca8f5ae555306aa8d6ecbd670  mga55_gda94_10cm_2018_CoM_true_ortho.jp2

[rouault]

OK, I finally reproduced the issue. A debug build of openjpeg was needed (at least one without -DNDEBUG), so that assertions are compiled in. The bug was actually in the GDAL OpenJPEG I/O layer, that wasn't ready to deal with reading single-tile codestreams larger than 2GB.

There was also an error in the MCT stage of openjpeg triggered by opj_decompress, which caused the "Tiles don't all have the same dimension" error. couldn't test the fix since I don't have actually enough RAM to reach that point, but there were a few 'obvious' uint32 overflows I've fixed. Anyway that part wasn't triggered by GDAL since it reads by much smaller chunks, whereas opj_decompress is brute force and decompresses the whole image at once.