Implementation-defined behavior of malloc causes different behavior on Linux and AIX

[smuehlst]

This was reproduced with AIX and Linux builds with the current HEAD revision ee04854 of OpenJPEG.

JPEG 2000 image that demonstrates the problem (save as impldef.jp2, GitHub wouldn't let me attach a .jp2 file):

When the image is decompressed on Linux, opj_decompress converts the image successfully:

$ /Users/stm/git/openjpeg_c/bin/opj_decompress -i impldef.jp2 -o impldef.jp2.raw

[INFO] Start to read j2k main header (822).
[INFO] Main header has been correctly decoded.
[INFO] No decoded area parameters, set the decoded area to the whole image
[WARNING] Non conformant codestream TPsot==TNsot.
[WARNING] tgt_create tree->numnodes == 0, no tree created.
[WARNING] No incltree created.
[WARNING] tgt_create tree->numnodes == 0, no tree created.
[WARNING] No imsbtree created.
                                 precinct_cw = 1 x recinct_ch = 0
[WARNING] tgt_create tree->numnodes == 0, no tree created.
[WARNING] No incltree created.
[WARNING] tgt_create tree->numnodes == 0, no tree created.
[WARNING] No imsbtree created.
[INFO] Header of tile 1 / 1 has been read.
[INFO] Tile 1/1 has been decoded.
[INFO] Image data has been updated with tile 1.

[INFO] Stream reached its end !
Raw image characteristics: 3 components
Component 0 characteristics: 197x14x8 unsigned
Component 1 characteristics: 197x14x8 unsigned
Component 2 characteristics: 197x14x8 unsigned
[INFO] Generated Outfile impldef.jp2.raw
decode time: 1 ms

Attempting to decompress the image on AIX creates the following output (decompression fails):

$ bin/opj_decompress -i impldef.jp2 -o impldef.jp2.raw

[INFO] Start to read j2k main header (0).
[INFO] Main header has been correctly decoded.
[INFO] No decoded area parameters, set the decoded area to the whole image
[WARNING] Non conformant codestream TPsot==TNsot.
[ERROR] Cannot decode tile, memory error
[ERROR] Failed to decode the codestream in the JP2 file
ERROR -> opj_decompress: failed to decode image!

I tracked this down to the following code in tcd.c starting at line 932:

l_nb_code_blocks = l_current_precinct->cw * l_current_precinct->ch;
/*fprintf(stderr, "\t\t\t\t precinct_cw = %d x recinct_ch = %d\n",l_current_precinct->cw, l_current_precinct->ch);      */
l_nb_code_blocks_size = l_nb_code_blocks * (OPJ_UINT32)sizeof_block;

if (! l_current_precinct->cblks.blocks) {
        l_current_precinct->cblks.blocks = opj_malloc(l_nb_code_blocks_size);
        if (! l_current_precinct->cblks.blocks ) {
                return OPJ_FALSE;
        }

When I enable the debug fprintf, the following output is produced:

[INFO] Start to read j2k main header (822).
[INFO] Main header has been correctly decoded.
[INFO] No decoded area parameters, set the decoded area to the whole image
[WARNING] Non conformant codestream TPsot==TNsot.
                                 precinct_cw = 1 x recinct_ch = 1
                                 precinct_cw = 1 x recinct_ch = 1
                                 precinct_cw = 1 x recinct_ch = 0

recinct_ch = 0 causes l_nb_code_blocks and l_nb_code_blocks_sizeto be 0, and that causes an attempt to allocate 0 bytes in the line l_current_precinct->cblks.blocks = opj_malloc(l_nb_code_blocks_size);.

The problem is that attempts to allocate 0 bytes cause implementation-defined behavior. malloc may return a non-null or a null pointer. For example from the Open Group documentation on malloc:

If the size of the space requested is 0, the behavior is implementation-defined: the value returned shall be either a null pointer or a unique pointer.

malloc on AIX decides to return a null pointer, and therefore the function opj_tcd_init_tile fails. On Linux and other platforms that I tested on malloc returns a non-null pointer, and decompression succeeds.

Apparently the test image is broken according to OpenJPEG warnings, but nevertheless the behavior of OpenJPEG should be consistent across platforms and not be different because of implementation-defined behavior of malloc.

[mayeut]

Consistency across platforms could be achieved with #625
I don't think the image is actually broken, it's probably been compressed with more decomposition level that width/height allow which is, I think, conformant c.f. #215. This should decode well on AIX as well.

[malaterre]

I thought about that. But one would need to be somewhat smart with opj_malloc(0). What should we do: always return malloc(1); only return malloc(1) on AIX ?

[malaterre]

I think the C code instead should be fixes, malloc(0) feels so awkward.

[mayeut]

Agreed, I was more thinking that opj_malloc(0)should always return NULL.

[malaterre]

I was thinking of assert(0) but I prefer your solution. I guess this is time to merge #625