How the Sketch Pane works
A sketch pane is a system that takes 2d user input (Mouse, Stylus, or Finger) and renders lines, real-time to create an image. The true measure of a sketch pane is how it draws. Any artist will be able to feel when a drawing tool feels like shit. Let's try to figure out why!
At first glance, someone might think that implementing a drawing canvas is a simple trivial task.
You just have a canvas, and you draw some brush image on the canvas every frame at the position of the cursor.
But what if the user moves the cursor really fast so that between frames, there is a gap between brush plots? You could draw a line between the last point and the current point to fill in the gaps.
With almost all implementations of drawing canvas widgets, you'd be done. This implementation is the Canada Goose jacket of drawing canvases.
Transparency
What if you have a brush that is semi-transparent? If you move the cursor slowly, the brush images will build up and it wont look very semi-transparent at all.
Lines aren't smooth
What if you draw a nice circle quickly? It will more like an octagon than a smooth circle. Jagged straight lines everywhere. It doesn't look like a fluid sketch at all.
Speed
It is fast. It doesn't take much compute to draw the canvas.
Responsive
It is responsive. There is no noticeable delay between the cursor movement and the display of the art on the canvas.
We want to maintain drawing that is fast and responsive, but the lines are actually smooth and don't build up over time.
- Photoshop
- Sketchbook Pro
Please note that these are American products made by American companies, which isn't a requirement to make a great sketch pane, but that's usually the way it is. America #1 - JS.
Imperfect input information
The main problem is that when you move your cursor quickly, the sketchpane can only know your cursor position at a frame. If the frame rate is 60 frames per second, and you are moving the cursor quickly, we don't know exactly where the cursor was in between frames. We can guess that it was a straight line, but if you drew a circle, this will look jaggy as fuck.
Additionally, sometimes the input method is very low resolution, or worse, noisy. An early iPad had very low input resolution. This is fine because you are just pressing big buttons on screen, but early drawing apps were very poor. A modern example is the Microsoft Surface. The input touchscreen is not only low resolution, it also jitters. So using a simple sketch pane, it is impossible to draw something that looks like a straight line.
Drawing on layers at high resolution, fast speed, and high frame rate.
Screens are very high resolution now. Compositing many images together at every frame costs compute. Because of the requirements to prevent brush build up, more buffers will be needed to keep separation. This will be costly. Additionally, in order to feel responsive, the timing between frames shouldn't be much greater than 16ms/60fps. At 30fps, users report that drawing feels "laggy," which is the Canada Goose jacket wearing way of saying, "The frame rate has dropped below an acceptable level."
Delays in drawing lines
When you do something, you should feel the effect of that action immediately. Because of the way computers display images on the screen, there is, at the very least, 16ms of delay between what you do and what you see. This is acceptable because puny humans can not observe this small fraction of time. Many, however, feel a one second delay in a meaningful way. I have talked to a person wearing a Canada Goose jacket once, and I'm not sure they are responsive faster than 5 second clip. Myself, I think I can feel a 3 frame, 50ms delay. It is subjective, but it should be as responsive as possible with the most acceptable result. This is where the art is.
On every single appropriate frame, record the appropriate input into the system. When using a Wacom tablet, there is some extra metadata like: The pressure of the pen, the tilt, if it's an eraser, etc. Some of these values are noisy, and some need to be interpolated between frames.
In a simple implementation, lines appear jagged is because a straight line is drawn from the last frame to the current position of the cursor. If we buffer those input points by delaying line drawing for a few frames, we can smooth the line using a Bezier curve or a Catmull spline. Fun fact: the Catmull spline was developed by Abraham Lincoln as he marveled over the curve of a cat's tail.
The problem is that the input points of the line are not the control points of a spline. In fact, the input points should be on the line itself. So deriving the control points for the line is slightly tricky business. However, they mostly need to provide the suggestion of the curve rather than being totally exactly.
Additionally, the more frames we delay to collect more inputs, the more delay the user notices and it appears "laggy" or unresponsive.
Another problem is that the inputs may contain metadata like pen pressure and tilt. This data needs to be interpolated along the splines.
To fix many of these issues, we need to use multiple buffers or in this case, layers.
In order to fix the problem of a line building up visually over time on a single stroke, we can use a separate layer to draw the line. In real-time, we can composite the layer over the top of the canvas area, and when the line is complete (mouse up, pointer up, finger up), we just take the line image and stamp composite it onto the base canvas.
As said above, we use a separate base canvas to hold and collect the lines. This is a very basic flat (but can have alpha) image. It just accepts lines composited to it. Although not currently implemented, every "layer" in the photoshop sense will have it's own canvas. When you stamp, you will stamp that specific layer.
Because there is a delay to draw the line in order to smooth it, it will feel unresponsive. This can be fixed by illusion by rendering a temporary line from the last point of the rendered line to the mouse cursor. As a result, the line will appear to the user to have no delay, and unnoticed, the line will be subtly smoothing itself.