Byte overflow problem in SIXEL decoder

[saitoha]

Forwarded from mattn/go-sixel#5.

The color register #255 is treated as 257th color in mintty.

mintty/src/sixel.c

Line 599 in 324152b

st->color_index = 1 + st->params[0]; /* offset 1(background color) added */

It will cause overflow in

mintty/src/sixel.c

Line 395 in 324152b

image->data[pos] = st->color_index;

.

[saitoha]

test patch:
saitoha@7b6bf81

[mintty]

I wonder why and how SIXEL colour registers interfere with mintty's text colour handling.
What if the colours are redefined (with escape sequences); would that spoil image display?
Maybe the two mechanisms should be decoupled?

[saitoha]

I think if we need to emulate old hardware faithfully, text color registers have to be shared with that of SIXEL.
Actually, some old hardware users use -m(mapfile/fixed palette) option to integrate graphics on their 4bpp/8bpp framebuffer terminal.
But we can also implement separated color registers for each images. This strategy is convenient for users in modern GUI environment. Current mintty choose it.

[mintty]

So if there is currently no dependence of SIXEL colors on text 256 colors (if I understood you correctly, haven't had time to test it yet, sorry), why was there this surprising interference of the 257th color with mintty default color indications? Just trying to understand.

[mintty]

I was confused by "257th" color because a few internal attribute values above 256 are used for foreground etc by mintty. Unrelated, so forget my previous comments here.

About the patch: Do we really need all the type definitions, int8, int16, int32, uint8... I'm asking because that doesn't compile with an older gcc version.

[saitoha]

OK I use definitions of src/std.h and src/config.h in a664ed8.
Memory leak issue reported by saitoha/libsixel#50 (comment) is fixed with e92d9a5.

[mintty]

Thanks a lot.
See also saitoha/libsixel#50 (comment)

[saitoha]

Thanks for merging!

[mintty]

Released 2.6.2.

[mintty]

I'm afraid I have to revert the latest patch from this issue, in order to resolve #740.

[mintty]

Released 3.0.2.