Make piet capable of being a back-end for resvg

[raphlinus]

The resvg project has a very helpful list of requirements for back-ends. I think piet is now at a point where it would be possible to start coding such a back-end, and it makes sense to carefully review where we stand.

Imaging model and blending/compositing

Right now, piet effectively implements a painter's imaging model, where each object is alpha-composited on top of pixels on the canvas surface. There is an entire class of operations - group opacity, nonstandard blend modes, using an image as a mask, and of course filters, that do not fit into this imaging model. I'm hesitating slightly at this point, because I want to make sure to expose this functionality in a way that piet back-ends can efficiently implement them, and also doesn't add too much complexity and burden for implementers.

The imaging model of resvg is fundamentally a tree, with nodes representing various blend and mask operations. The way resvg works with, say, the Cairo back-end, is to treat the graphics library as a source for pixel data, and do the blending and filter operations in software. We can do this approach, and it is probably the shortest path to getting a working integration with resvg. Basically it requires exposing an interface for creating an image surface, and for getting RGBA pixels out of the image surface. This is is already done to some extent in the demo apps for writing image data, but (significantly) not in the web back-end, as the demo app has a one-way flow of data, drawing on the screen.

When rendering is done on GPU, getting pixel data back is inefficient. Doing blending operations in software is doubly so, especially for pointwise operations (including alpha groups, mask, and blend modes), all of which can be implemented efficiently in GPU shaders. The Direct2D library has a rich set of effects, all of which seem to be available on Windows 7 with platform update. Not using these is leaving performance on the table.

That said, requiring the complete set of effects for all piet back-ends may be onerous. Further, many of these effects are of fairly limited use in UI contexts. If I were to prioritize, I'd pick group opacity, as that is something that's quite useful for building UI, and is also reasonably easy to implement. I think it can serve as a bit of a proxy for potentially more sophisticated blend modes.

For blend modes, it's a bit confusing what functionality to support. resvg seems to implement a limited set, consisting of Normal, Multiply, Screen, Darken, and Lighten. The SVG Compositing spec has a much larger set. It also specifies complex options such as "enable-background" and "knock-out". The latter is effectively a second alpha channel, and seems to be imported from PDF 1.4 transparency. The complexity of implementing that for the web back-end would be considerable, and doing so performantly is challenging. But I think expanding the set of "comp-op" values is fairly straightforward, as all 3 current back-ends seem to support the same basic set.

What I'm leaning towards is exposing a "get RGBA pixels" interface so resvg can use software techniques for blending, and punting on the complex blends, with the likely exception of group opacity.

Sophisticated text

There's no question piet needs fancier text (partially captured in #8 and #10), and there are number of text things in the resvg requirements list. I would like to punt on this for now. Part of the issue is that the web canvas back-end has only primitive text, as does the current Cario "toy text" implementation. For web canvas, we'll have to return "not implemented" error results for some queries, or fake it somehow. For the others, I want to get a solid text solution, but feel this is orthogonal to most of the rest of the resvg requirements

Gradients

To some extent, gradients are the easiest problem, though of course the devil is always in the details. I propose to subsume gradients under the existing Brush associated type, and simply add more builders. It is likely we'll have to do some workarounds for rendering quirks.

In any case, I'm posting this issue to get a read on whether it makes sense to start implementation work on integrating resvg and piet, and what needs to be done on the piet side. @RazrFalcon is of course invited to provide feedback as well.

[RazrFalcon]

As for the text, the next resvg release will use its own text layout, so there is no need for a backend to support text. Mainly because the SVG text layout is extremely complex and you need full control over each glyph. pango has some support of this, but Qt has no access to glyphs. So I'm dropping them.

Also, the default cairo text support is laughable. You must use pango.

Here is a list of problems that I encounter during resvg development.

Problems with Qt:

• QPainterPath::addText ignores BIDI reordering. QTBUG-72890
• No access to raw glyphs.
• No access to a raw layout. The lowest you can get is QTextLayout.
• Have to init QGuiApplication (font cache basically) to render text, which is very slow (20-100ms).

Problems with pango:

• word-spacing isn't supported.
• font-stretch isn't supported.
• Small caps isn't supported. #123
• Incorrect handling of letter-spacing on Arabic/cursive scripts.
• BIDI reordering is kinda broken. At least in librsvg/Inkscape it works a bit incorrectly.
• Unintuitive API.
• Depends on glib, which is a pain to distribute outside the Linux.

So I have no idea what exactly piet should support. As for SVG 1.1, it can render only lines. It doesn't support multiline text (unlike SVG 2.0). And pango, Qt are designed for multiline text. So in terms of GUI rendering, you shouldn't care about SVG.

[RazrFalcon]

resvg seems to implement a limited set, consisting of Normal, Multiply, Screen, Darken, and Lighten.

No, I use all methods that are defined in the backend_requirements.md. The one you listed are for feBlend filter only.

enable-background

Which isn't supported by browsers (at least by Chrome and Firefox), because it's too expensive (I presume). Also, it's deprecated in SVG 2.0. resvg doesn't support it either yet.

whether it makes sense to start implementation work on integrating resvg and piet

I will try to do this today.

[raphlinus]

Thanks for the responses, @RazrFalcon. It's not easy for me to figure out the current state of SVG, as the specs seem to be pretty divergent from implementations still. And obviously I was looking in the wrong place regarding blend modes.

Stay tuned for more details about how I expect to handle text in piet. This is a topic I have some knowledge about, the real question is what scope to take on. I am leaning away from pango, as I see its limitations clearly. Also, it's not portably packaged as a Rust crate, although I'm sure this could be done. I hope to have more details on text soon.

Please let me know (either in this issue or file new ones) what features you find missing in piet and want added. I have some bandwidth for implementing things.

[RazrFalcon]

as the specs seem to be pretty divergent from implementations still

I think that no one really fully supports the SVG. At least according to my tests. And yes, the specs are pretty vague about a lot of things.

Also, it's not portably packaged as a Rust crate

It is. I'm using it.

[raphlinus]

Also, it's not portably packaged as a Rust crate

It is. I'm using it.

We have different definitions of "portably packaged as a Rust crate." What I mean is that "cargo build" succeeds out of the gate. I agree that it is possible to get pango running on non-Linux systems.

[RazrFalcon]

Oh, you mean like a crate with bundled sources? I don't think that you can do this with pango, since it depends on lot of things.

[raphlinus]

That is what I meant. I do believe some of those dependencies can be broken - we did this already with making the glib dependency of cairo optional. (And by "we" I mean @rtsuk and I).

[RazrFalcon]

Done.

Only paths rendering is implemented and layout handling is still uses cairo.

1. Do you plan to put backends behind cargo features or do you plan to keep them as separate crates. I thought that piet will be like QPainter, completely hiding the internals. But for now, you have to write a lot of backend-specific code too.
2. Too much Result. I understand that you are using this for Unimplemented, but I think that panic/warning would be enough.
3. Should we use default values for StrokeStyle? Option makes api a bit verbose.
4. Naming is a bit inconsistent. You have curveto and not curve_to, but line_cap and not linecap. I prefer the one with underline.
5. I need get_transform/set_transform methods. You have the transform method, but you can't reset the transform using it.

[raphlinus]

Wow, that was fast.

1. Do you plan to put backends behind cargo features or do you plan to keep them as separate crates. I thought that piet will be like QPainter, completely hiding the internals. But for now, you have to write a lot of backend-specific code too.

I think I will do a piet-common crate that will by default select the best back-end, and also have methods for creating image surfaces. If you're just making a png or an RGBA pixel buffer, the back-end details can be hidden, but when wiring up in a UI, you do need specific interfaces. There's a bit of discussion about this in linebender/druid#8 and more on the zulip chat.

2. Too much Result. I understand that you are using this for Unimplemented, but I think that panic/warning would be enough.

This is a style question. I think panic is basically unacceptable in a GUI app, though much more acceptable in a CLI. Basically, every time I've tried to cut corners, I've regretted it. I think ? makes this tolerable.

3. Should we use default values for StrokeStyle? Option makes api a bit verbose.

I'm open to this as a change. The existing API was adapted after Direct2D, and I think it makes sense when StrokeStyle is an associated type that likely represents an allocation. But that's changed now.

4. Naming is a bit inconsistent. You have curveto and not curve_to, but line_cap and not linecap. I prefer the one with underline.

In this case, I'm trying to honor the PostScript legacy, but I agree, curve_to would be more consistent with Rust style.

5. I need get_transform/set_transform methods. You have the transform method, but you can't reset the transform using it.

The theory here is that you use save/restore to reset the transform back. In general I prefer a one-way flow of information from the client into the drawing API. I can reconsider if you feel this is absolutely required.

[RazrFalcon]

Wow, that was fast.

Well, we can only fill and stroke paths =)

I see. Then I will wait for gradients and piet-common.

[cbrewster]

I've been working on a resvg piet backend for fun (probably should combine efforts with @RazrFalcon). My current tree is at https://github.com/cbrewster/resvg/tree/piet-backend. I am using the new piet-common so hopefully this will allow us to switch the piet backend easily.

I think following through with this will really help expand Piet's feature set in a practical way.

Currently I have paths being filled; however, I am running into some areas where Piet is lacking in features. I will make a more comprehensive list as time goes on, but here are the ones I have run into so far.

Missing

• Affine transforms on gradients
• Gradient spread methods
• Layer support (Potentially could be done outside of piet)
• Antialias settings

[RazrFalcon]

@cbrewster Does it based on my piet branch?

I think following through with this will really help expand Piet's feature set in a practical way.

The main problem is text. We can't do anything without it. I'm implementing a text layout on the resvg side, without using any backends, but it's moving slowly. See font-kit, skribo issues.

Also there are filters (direct access to pixels), composition modes, raw pointers (#44), etc. A lot of work.

[cbrewster]

@RazrFalcon I didn't even think to check if you had a branch already, so I started my own and then later noticed your branch. Although it looks like your branch assumes the cairo backend since it looked like you worked on it before piet-common. Hopefully with piet-common, we won't need to write any piet backend-specific code in resvg.

I haven't gotten into text stuff yet, I was just planning to implement everything that piet allows first and then fill in the missing pieces as more support comes.

[RazrFalcon]

I see. Yes, my attempt was before piet-common and it's pretty outdated already. On the other hand, piet didn't progress much since than. Do you plan writing the piet itself?

[cbrewster]

My hope is to at least open up some discussion about what piet should support (I am happy to help implement new features as well).

I do want to be careful about how much we add to Piet though, as it could make it harder to implement new backends in the future. I think we are at a point where we should figure out all of the features piet needs to have before other backends are implemented which makes it harder to change piet in the future.

[raphlinus]

What @cbrewster said. I am being deliberately very careful in adding new features, because it's important that they are supported by all back-ends. A large part of the reason I did the GPU investigation is so I could get a better handle on what could be reasonably easily and efficiently implemented on GPU. Stuff like blur remains a big challenge.

Text is going to go slowly, simply because it's a very hard problem. I'm willing to mentor work in the space, and hope it's compatible with the piet umbrella, but also realistically don't have a lot of time for it in the next few weeks.