Erase all pixels on a layer based on their RGB value / draw BEHIND and COLOR ERASE brush modes

[blurymind]

note:
I moved this out of #180 so it is easier to discover and discuss individually.

The request:

Add these layer commands (to the layers menu or as a tool) that can be assigned to shortcuts:

• Clean up lines with current color (on layer) - deletes all the black, cyan or red pixels on the layer - depending on what user has selected for current color from the "sketch" palette
• Colorize lines to current color (on layer) - colorizes all the pixels on a layer to black, cyan or red - depending on what user has selected for current color from the "sketch" palette
• Add a "sketch" palette preset with 3 sketch colors. (cyan,black and red).

Benefits:
It is a feature that would be beneficial if you are doing animation on single layers #180 and need to clean it up (as you do in most cases).
It would also be beneficial to sketching in general - simplifying the process by eliminating the need to create and use multiple layers in order to clean up your sketch before coloring.

Example in other software:
One way to be able to sketch, clean up and ink a drawing without the need to use multiple layers is to do it like tvpaint -
https://vimeo.com/57043308 (see at 5:28)

It has 4 compositing inks that can be erased individually - red, cyan, green and black.
The user sketches in one color (cyan), then for the second pass sketches with another color(black) over the old color. Then with a single click of a button, he erases the first color that is underneath (cyan) and gets a clean drawing with only the second pass (black).
If he wants to further refine the drawing, the artist turns the cleaned sketch he has (black ink) into a another color (cyan) again - with a single click. Then repeats the process.

This is really a fantastic way to work on a single layer and mimics the traditional animator workflow as well! In traditional animation animators sketch with the cyan pencil, then sketch with the black pencil on top of the cyan. Then erase the cyan pencil with an eraser that only affects it.

I am not sure how hard it would be for you to get a color by it's rgb value on a layer and erase it, or colorize it to another rgb value. Tvpaint has 4 colors to do this workflow, but you can really get away with two - cyan and black.

[blurymind]

Another application that utilizes this blue pencil and black pencil technique is PAP:
http://animationpaper.com/tutorials/
(see lesson 4 and lesson 8)

When you are happy with the rough sketching you can switch to drawing black [F6]. Now refine your sketchy blue drawings by drawing on top of everything in black. Details you left out before, like facial features, arms, hair etc - can be put in now.

Once you have finished refining, you can get rid of the old blue lines with the Clear Blue function - leaving only the new black lines. (The Clear Blue icon is normally placed in userwindow 2 [F2]).

Also Here is they have the colorize line command I am suggesting:

After removing all blue lines, you click and hold the Black to Blue icon - select "All" in the option menu. Now all your new black lines are converted to blue, and you have your detailed animation ready for cleanup.

I hope this explains it better than my first post.
Its basically two commands:

• erase all lines on the layer that are blue (rough sketching color)
• colorize all lines on the layer to blue

But in my suggestion I am wondering if it can be done in any color. So Instead of blue, the command could use the current color's RGB values and use the palette to have predefined defaults for a rough sketch color and an inking color.

[blurymind]

I made a quick mockup with inkscape of how this command could be accessed:

There may be more elegant ways to access it (best of all would be just two keyboard shortcuts).
I really cant live without this feature in tvpaint. It is SO useful for cleaning up single layer drawings.

[blurymind]

Here you can download and try applications that have this feature:

• PAP (free- windows):
  http://animationpaper.com/old-pap-free-download/
• Tvpaint (Demo is free - windows,mac,linux)
  http://www.tvpaint.com/demo

To further see how it is designed there

[blurymind]

after further research, I found that this feature is even more useful if you have the ability to draw with a color Below any other color that is not the color that you are drawing with.

How it works :

• It sets brush to alpha mode that protects already existing pixels on the layer from being affected by new strokes. So if you have a black line and you draw over it with a blue pencil, it's pixels do not get affected by the blue pencil.
  • This should of course be optional, because some times you want to overwrite pixels with another color when you draw on top of them. For example if you draw with a black pencil on top of blue lines, you want the black pencil to draw over the blue pencil lines.

User case example :
To illustrate this, lets say we have a hand and need to draw a new variation of it. We will start with the previous drawing, but erase the parts that need to be changed:

Then we use the blue pencil to sketch a drawing of the new hand variation:

The blue pencil is set to NOT affect existing pixels by drawing on top of them.

We finish the blue line sketch, then we do the opposite, we set a black pencil and set it to overwrite already existing pixels with its color and ink with it:

Finally, we execute the "erase all color on a layer" command, with color set to blue! What we are left with is the clean line art:

All done on a single layer!

You can also apply this to selective eraser mode, where the eraser tool can be set to work only on lines that have the same color as the one that is currently set to the eraser tool.

Why else is this useful?

• When you draw for animation, and in general - it is good practice to draw an object complete, when the object is behind another object and then erase the part of the line that is behind the other object. In this case we have the fingers of the hand covering the handle. We must draw the entire handle, then clean up the part that is behind the fingers.

Without this brush mode+ command, it is very difficult to clean up the handle without messing up the lines of the hand. If you however draw the handle in blue or any different color from the hand, then you can selectively erase only it's pixels without affecting the hand. Finally when you are done, you can colorize the entire layer to black - turning whats left of the blue handle lines to black.

[blurymind]

Sketching with multiple pencils (black and blue for example) will work great with the tool management system that is already implemented in drawpile.

This workflow would also greatly compliment live paintchat sessions and motivate people to use less layers to do complete drawings and make it extremely easy to cleanup drawings.

It is not useful only for the animation workflow, which is currently limited to one layer. It adds great value to all user cases in drawpile.

[blurymind]

updated the name of the proposal and some of the last posts so as to make it more clear what it does

[callaa]

I implemented an "underpaint" blend mode, since it is basically the same as the normal mode with just the alpha channels reversed.

I tried implementing a color erase mode too, and it kinda works, but not reliably enough for me. The problem is that when you have semi-transparency, mixing arbitrary colors is a destructive process and can't really be reversed. It's only possible with a limited palette of distinct color channels (like black, red, green and blue)

Given these limitations, I'm not sure how to work this feature into Drawpile's UI.

[blurymind]

The limited palette would do fine! :)
If it works with two or three colors that is more than enough.

Actually this feature is implemented as a limited palette in all other software that has it.

This could be worked out into the ui in two different ways i think.

Approach 1
As a special new tool that is like the brush tool, but is limited to 3-4 colors (red,green,blue and black - one of them must be black). Maybe call it the "sketch" tool?
Here drawpile's interface design fits great, because it's slot icons already communicate to the user what the tool color is. The users can set three slots with the "sketch" tool with different colors and settings.
In it's right side panel you can switch between the 3-4 sketch colors and also execute the commands to

• Clear all pixels with sketch color,
• Set all pixels to sketch color.

Set sketch brush operation mode to:

• Draw normally,
• Draw with locked alpha (when you want to color a part of a black line to blue),
• Draw behind (when you sketch with blue lines and you dont want to affect black lines),
• Eraser all ( when you want to erase a part of a line that is in any color),
• Eraser by color (when you want to erase only lines with the set color)

The draw behind brush mode for the normal brush is still useful when you want to correct issues with incomplete bucket fills btw.

Approach 2
As a "sketch" mode for the brush tool. But then you need to have it change the UI of the right panel to the accommodate the changes.

Thank you for the work so far btw!
Drawpile is getting really cool updates that are innovative as compared to other open source software out there.

[blurymind]

approach 1
I made a mockup of a possible Sketch tool:

To explain approach 1. The icon is not great, just took the pen tool icon and added a little something. The commands drop down menu button could take the currently active sketch color as a visual hint. But that is just an idea. Not sure if it would work well in the ui.

For approach 2, it would be more complicated to come up with a design I think :)

"erase by color" is an extra feature btw, it is not a must have. Just nice to have in some cases, but not that important.

[blurymind]

approach 2
you could also just add the sketch colors to the brush/pen tool. You get the commands menu only while you are on one of the sketch colors. If the tool is set to another color, the menu gets grayed out. Here is a quick mockup:

Maybe the sketch colors could show up when you have set the tool to be in sketch mode?

In this case the most important feature is the ability of clearing all the blue line pixels on the layer after we have sketched on top of them with the black pen. The clear pixels command and the colorize pixels commands. The pixels that need to get cleared are always bellow the ones that you intend to keep. Have that in mind.
The rest is just nice and complimentary.

I guess I need to test out this feature in drawpile - once there is an implementation in order to see if stroke alpha matters so much.

[blurymind]

Here is a line test to investigate if alpha affects it in tvpaint:
https://youtu.be/BQh1T_smh3g
Black lines dont seem to get affected at all - even when semi-transparent. I am not sure how they did it. it's all bitmap and its on a single layer. But it is locked to those predefined colors that you see in the interface.

Please note that drawing semi-transparent lines is not a common user case scenario. If you look at the hand holding a handle example - lines there are crisp - with not much alpha going on in black strokes.

We want black strokes to not be affected by blue/green/red strokes bellow them.

Honestly having black and blue is enough to get this workflow.

[blurymind]

updated one of the previous posts in order to add a gui mockup for approach 2.

Approach 2 would be more flexible I think. Hope this helps :)

No other open source has this workflow/feature at the moment. Mypaint could have benefited from it as well.

[blurymind]

The simplest possible implementation is in PAP (plastic animation paper)

There you have only a black and a blue pencil. The blue pencil is always set to "under" painting mode.
And what you are left with are these commands:

• turn blue pencil strokes to black (tursn all strokes to black)
• turn black pencil strokes to blue (turns all strokes to blue)
• erase all blue pencil strokes without affecting any of the black ones.

Having more than one sketching color is very advantageous for cleaning up animation however. But pap's implementation is better than nothing.

Approach 3 - least obtrusive to the current design - "Sketching" menu containing the commands
The least obtrussive approach to designing this is simply by adding the commands to wipe a color channel somewhere in the menu (wipe by RGB channel).

• wipe all blue pixels
• wipe all red pixels
• wipe all green pixels
• wipe all black pixels

then in the color palette add a presset for sketching that has pure red,green,blue and black

if you want to make this even more complete, you can later add to the eraser tool modes to selectively erase color channels (red,green,blue,black)

If its too complicated to have so many colors, you can go with only black and blue.

Doing it in this approach has the advantage of being able to use any of the other drawing tools in drawpile to sketch and ink, then clean up the sketch pass that is bellow.

[blurymind]

Here is a mockup for Approach 3

In this example all of the functionality is inside a simple menu, where each entry has a submenu with the 4 colors.

I think its pretty clear what it does that way and it is done without changing the UI too much. Without touching any of the tool settings. Just a clean, simple menu with the commands.

The Sketching menu can also be placed inside "tools" if you think its cluttering too much.
These are all commands that affect pixels on the active layer. The Sketching menu could be placed under a "Layer" category as well.

That is the best i can come up with. :) I hope you like it

[callaa]

Thanks, lots of good ideas and information! I'm not sure if I can fit this in before the 1.0 release, which I'm hoping to do in a few weeks. I'll probably have to postpone the rest of the implementation until after version 2.0.

[blurymind]

I see. Thank you for the work so far :)
Let me know if you need some testing on a fork of drawpile with features which are not completely implemented. I made a buildscript for arch linux, which allows me to easily have it built and running
https://aur.archlinux.org/packages/drawpile-git/

Looking forward to the 1.0 release! And even more for 2.0

I've been doing clean up, inbetweens and coloring on a production with tvpaint, and this feature is at the core of the clean up workflow. It's essential

[callaa]

Actually, now that I think about it, a simple sketch tool (like in your approach 1 mockup) might be possible to implement without any major protocol changes, just by adding two more blending modes:

Behind (already done!) and normal paint

For sketching. Nothing special here.

Color erase

This is the tricky bit. A general purpose color erase mode only works well when colors are not mixed (i.e. opacity is always 100%) However, when we limit the palette to red, green and blue, we can just zero out the channel to erase a specific color. I think the alpha channel is calculated by taking the multiplied color value and subtracting it from the current alpha ( a=a-(c*a) )

The colorize pixels action can be implemented as a Recolor mode FillRect.
Likewise, clear pixels can be done with a color erase FillRect.

So, if these blending modes work as I think they will, I'll be adding this feature for 1.0 after all!

[blurymind]

This sounds about right. :)
You described them correct. This is awesome news!

To me personally the biggest feature is the clear pixels and colorize pixels commands. Those should work nicely with all the drawing tools in drawpile, for as long as one uses the right colors.

As mentioned before, opacity is not that important, as the final ink line is usually solid. But if you manage to get even that to work perfectly, then i guess it will be technically better.
I guess that with some anti-aliasing on the black line, the color mixing with the alpha could be a problem. So hopefully zeroing out the channel will resolve it. Will the clear pixels command use the same approach?

Can't wait to try out an implementation and play with it. This is going to be so great

[callaa]

In the above image, the horizontal lines were drawn first and the vertical lines were drawn after, but with the "Behind" blending mode.
Red, green and blue color erase was applied to the three groups on the right.

It looks like the color eraser isn't quite as simple as I thought. Even thought blending primary colors does not mix the channels, it can still cause the other color channels to darken. I would guess this happens in tvpaint too if you paint semi-transparent blue over black?

I noticed that Gimp has a pretty good color erase mode that can work with any color (although in this use case, the palette should still be limited to R/G/B for clean separation). I'll see if I can adapt their blending code.

[blurymind]

In this line test I am replicating your experiment in tvpaint:
https://youtu.be/leDwFkGLjgs

For the experiment I draw a black line normally- on top and also draw it with "behind" in order to illustrate ontop vs behind.

As you can see tvpaint is successful at two things:

• when a semi transparent black line is on top of a blue/red/green line and the blue/red/green line is wiped - the black line's opacity is not affected at all. If however blue/red/green is on top of the black line and is then wiped, the black line's opacity gets affected
• When you tell tvpaint to wipe a colored line, it really wipes it. If it overlaps with another colored line in a different color, it does not affect the color of the other line when its wiped where they overlap - just its alpha. In your example for some reason the color changes to black where one color overlaps another color and a wipe occurs.

It looks like you are really close to getting this working. If the line darkening is solved.
Do you get line darkening if you set all the colored lines to "behind"?
I dont get line darkening in tvpaint. Just partial or full loss of alpha of line bellow when wiping the pixels of a line above it.

[blurymind]

After looking at gimp's color erase mode - it seems to do the erasure compositing right - just like tvpaint in the experiment!
Would it be possible to take the algorithm from gimp then apply it to fillrect with erase (wipe all red/green/blue/black pixels command) and to the brush "erase color" mode? I hope you manage to do it. That blending mode is fantastic!

This is what I found:
https://git.gnome.org/browse/gimp/commit/?id=5d7bf3ca4b56f4f8b89f13a39460c8dc6c4a0755
and
https://git.gnome.org/browse/gimp/log/?id=925ee2175086c5f80127729e9901ee5a6ebd7cf1&qt=grep&q=color+erase

Hope it helps. :)
Thank you for the progress so far

[callaa]

I think it works! Thanks for testing with tvpaint. The new color erase mode seems to be equivalent to what it has.

I added a button to Eraser settings to toggle between normal mode and color eraser.
Now I think the only features missing are the Colorize and Clear Color actions, which should be easy. I'll add them to the Edit menu next to Fill and Clear.

[blurymind]

Awesome!! :DD
I think you nailed it. I will compile a new build and play with it tonight.
I want to make sure that if the black line is ontop of a blue line and the blue line is erased, the black line doesnt get affected. In the example image all black lines get affected, but that is brobably because they are drawn bellow the colored lines.

Drawpile is becoming more and more amazing as an animation tool. Thank you for having the patience to read through this thread and for implementing it so quickly. I am happy to have helped.

[blurymind]

Ok, I just tried it and it works like a charm!!
The eraser tool enhancements are fantastic too.

I noticed you called the command "color erase selection" and "recolor selection" . This made me try to lasso select a few pixels and see if the commands actually works on a selection.

They dont work on a lasso selection yet.
They do however work on the entire canvas and on a square selection.

[callaa]

Great.

Yeah, lasso selection doesn't work yet. Rectangular selections are filled with the FillRect command, but non-rectangular ones are sent as bitmaps. The problem is that unlike FillRect, PutImage only supports a few basic blending modes. I'll have to see if I can make it use the same compositing code.

I think this feature is now done for version 1.0, unless I've forgotten something?

[blurymind]

I think that you pretty much nailed it. It is functioning beautifully!
Thank you for this. It makes drawpile so much more useful for animation!! :)

if you want to add some polish to it, you could add a new sketching palette:

You could add blending modes to the bucket fill tool some time in the feature, with the color erase and recolor options available there, then the command will be even easier to discover and the tool would communicate what color is set to be erased. In gimp one can just set the bucket tool to "recolor" and to "color erase" blending modes.

But i think its better as a bind-able command - the way you have made it now.
It takes a keyboard shortcut to execute it, which is fast workflow! In gimp you need to click and click and click

[callaa]

PutImage blend modes (and thus, recolor/erase with lasso selections) now work! The extra modes are not yet exposed in the UI, but the protocol now supports all the same modes for images as it does for brushes and layers.

[blurymind]

Awesome work! :) Thank you for these improvements.