Opacity 255 in Alpha Compositing to Behave as Simple Ink

[KasumiArai]

Open this file:

Eyedrop the black in the top left corner of the image in color+alpha mode. Index 15 is selected.
Click index 0 in the palette to select it as the foreground color.
Fill the black in the top left corner of the image with the foreground color with Alpha Compositing ink and alpha 255.
Eyedrop the black in the top left corner of the image. Index 15 is still selected. The expectation was for it to be filled with the same color index.
GIF of this case:

Aseprite 1.3-beta7-x64 Steam
Windows 10

It works as expected in Aseprite 1.2.30-x64 Steam on Windows 10.
It also works as expected with the other blacks in the palette that are not index 0 in Aseprite 1.3-beta7-x64 Steam
I understand the need to color match after compositing for semi transparent brush colors, but this current behavior confused me

[iamOgunyinka]

@dacap I want to ask you about this behavior. In this screenshot I attached, there is a palette that has seven (7) identical colors at different indexes [0, 4, 8, 12, 13, 14, 15]. When we add these colors to the octree map, using their indexes in the palette as the paletteIndex, new values overwrite the old values, as expected in any typical map.

// color 4278190080 is the color of interest here, so the octree map uses a function to calculate the index of the color.

int index = getHextet(4278190080, level);      // almost always yield `8` for levels in [0,8).

So we now have the following sequence:

a[8] = 0;
a[8] = 4;
a[8] = 8;
a[8] = 12;
a[8] = 13;
a[8] = 14;
a[8] = 15;

Of course, the last value is the only value that'd be retained in the map. So when we try to paint a region on the sprite with the first color in the palette, using the TransparentInkProcessing<IndexedTraits>, what we end up using is the color at index 15. So the Eyedrop's behavior is correct.

My question: is this a bug or the expected behavior? I can't tell which it is.

[dacap]

is this a bug or the expected behavior?

@iamOgunyinka I would say that this is a bug. In transparent layers we could expect to map RGBA=0,0,0,255 to the first match in the palette instead of the last one, e.g. in the original example it's 0 for background layers, and index 4 for transparent layers (but never entry 15 as there are other colors in the palette that matches first).

Anyway the "Alpha Compositing" is a little strange in indexed mode, I think that if there is no alpha composition involved (all opacity is 255), and the active color is index, it should just paint the selected index color (so if we have index 15 selected with "Alpha Compositing", it should paint with index 15, not with 0 or 4, this means that we should use just CopyInkProcesing, maybe something to fix here).

About the generation of the octree here, probably changing the order at which palette indexes are processed might fix part of the problem:

  for (int i=palette->size()-1; i>=0; i--) {
    if (i == maskIndex) {
      m_root.addColor(palette->entry(i), 0, &m_root, maskColorBestFitIndex, 8);
      continue;
    }
    m_root.addColor(palette->entry(i), 0, &m_root, i, 8);
  }

Maybe @Gasparoken can say something about it.

---

@KasumiArai about this case, you can workaround the issue using the old (v1.2) RGB color mapping instead of octree:

[Gasparoken]

Hi @iamOgunyinka and @dacap. Will investigate it.

[Gasparoken]

Anyway the "Alpha Compositing" is a little strange in indexed mode, I think that if there is no alpha composition involved (all opacity is 255), and the active color is index, it should just paint the selected index color (so if we have index 15 selected with "Alpha Compositing", it should paint with index 15, not with 0 or 4, this means that we should use just CopyInkProcesing, maybe something to fix here).

Alpha Compositing has the feature of compositing the color in the sprite with the current selected fg color, then the resulting color is searched through the palette to find the best fit (this search occurs only in Indexed Color Mode).

For example, assume a sprite in Indexed Color Mode with a Background layer, and it has a palette with white as 'transparent' index, green, dark green and semitransparent black:

On ink type = Alpha Compositing, If you pick semitransparent black and try painting over a green pixel, it'll result DARK GREEN (i.e. the better color index which RGB is near to the composition of green + semitransparent black).

On the other hand, on ink type = Simple, If you pick semitransparent black and try painting over any color, it'll result in the composition of TRANSPARENT INDEX (white in this case) + semitransparent black = gray.

I saw when we painting with the Transparent Index color in Alpha Compositing we turn false 'opaque' variable here and that's why Transparent Processing is selected here and finally, the 'pre-composition' is done here

I'm testing this change (but inks.lua test is failing so, I need to do a little more research on it):

I'll review octree scheme too (tomorrow).

[Gasparoken]

Thanks @iamOgunyinka! Yeah, you're right, the octree creation process is from the lower to the higher palette index, and each added color overwrites the previous correspondence INDEX <-> COLOR.

I'll change this behavior as @dacap said.

Finally, thanks @KasumiArai for this report!