Implement text() for webgl mode

[kjhollen]

Nature of issue?

• Found a bug
• Existing feature enhancement
• New feature request

Most appropriate sub-area of p5.js?

• Color
• Core
• Data
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Other (specify if possible)

Feature enhancement details:

text() only works in 2D mode currently. there is a lot we can leverage from the OpenType library already included here to generate opengl textures for use in rendering. I'll sign up to take this on soon. :)

[kjhollen]

been a while; just an update to say I've got some spare cycles in the next couple weeks and this is at the top of my todo list!

[rdsilver]

Hey KJ, any update on this? :)

[kjhollen]

progress is slow! but, from what I've researched, I think our options are

1. pull an image from HTML (using something like dom-to-image to use as a texture)

2. have webgl 3D mode lean on an instance of p5.Graphics in 2D mode, which it uses to make a text rendering

3. use the outlines from opentype (which we already depend on) to render real geometry, which doesn't perform or scale well, but it is very accurate.

and there's another common gl + text technique that I've ruled out, which is to render a map of all the glyphs to a texture and use the uv's for a single glyph, but that requires writing a layout engine, too, which I think is not a good solution for us (there are lots of other internationalized libraries out there that already do this way better than we could hope to with our focus/resources).

right now I'm leaning toward 2) because it feels like we'd have the most control over that approach, but it feels a little funny to have 3D mode with a dependency on 2D mode. open to other thoughts or suggestions!

[dhowe]

so how does option 2 work without option 3? one nice thing about 3 is the possibility of adding z-coordinates to get 3d text objects if desired, though you're correct that performance is a real issue here

[kjhollen]

ah oops to clarify the difference here is that option 2 would just pull a texture/image from the graphics instance with the text rendered (rendering 2 triangles / 6 points per call to text()), while in option 3 we'd push the open type points directly to open gl arrays and no textures would be allocated.

[Spongman]

I'd like to grab this if possible.

i have made some good progress on an anti-aliased glyph renderer that renders opentype fonts directly using bezier-crossing winding rules:

(two triangles)

all that's left to do is to put it into the p5 library. i'm hoping to reuse as much of the existing p5.js canvas opentype font-rendering code as possible.

[kjhollen]

Thank you for offering to take this one, @Spongman. This issue is currently assigned to @AdilRabbani, who I believe is planning to work on text rendering as the next phase of his Google Summer of Code project. I would like to let Adil continue his work before reassigning the issue.

[Spongman]

ok. well, if anyone's interested, this is how far i got: https://codepen.io/Spongman/pen/djyxvo

FWIW: i went with a variant of #3, above. maybe one of the other methods will yield better results?

this one's really mesmerising: https://codepen.io/Spongman/full/pZvJxv/

[AdilRabbani]

Hey @Spongman is the technique you used the same as this paper? http://jcgt.org/published/0006/02/02/paper.pdf

[Spongman]

yeah, it looks like that's quite similar to what i ended up with. i originally had a single curve array for each glyph, but i found that didn't scale well for complex fonts like nardis since the fragment shader had to always test every curve against each pixel. i then tried separating the curves into an MxN grid, but while the fragment shader performance is good in that case, it turns out having the curve spans being O(n^2) doesn't scale in terms of space requirements. splitting the curves into rows & columns is a good tradeoff between the two.

in that paper it looks like they're encoding all the glyph information in a single 4x16 integer texture (which unfortunately isn't supported in webgl). my solution uses 5 regular 4x8 float textures which does require a certain amount of manipulation in order to get the precision for control point placement, but does result in a more efficient use of space. i'm really not quite sure what all the 0x2E74 stuff is about, though.

i found that most of the complexity is related to packing the data into textures correctly, subdividing cubics (here i reused my old c++ code from swfmill) and managing boundary cases.

here's my branch with all of this in: https://github.com/Spongman/p5.js/tree/webgl-text

examples are in /test/manual-test-examples/webgl/text/

also in there is the factoring out of the previously 2d-specific text code into a form that's shared between the 2d & 3d renderers.

[AdilRabbani]

Hey @mlarghydracept and @kjhollen probably the easiest way to render text in webGL in p5.js right now is :

var _text;

function setup() {
  createCanvas(400, 300, WEBGL);
  _text = createGraphics(400,300);

  _text.fill(0);
  _text.textAlign(CENTER);
  _text.textSize(50);
  _text.text('p5.js is cool!', 200, 150);
}

function draw() {
  orbitControl();
  background(230);

  texture(_text);
  plane(400,300);
}

There are some drawbacks though. We are using a 2D text as a texture here and this would give blurry text when the camera is moved. This basically means we need a new texture to be rendered for every 'changed view'.

SDF text to the Rescue

A Signed Distance Field rendering would mean that we create a font texture that would not contain the information for the glyph pixel itself but information about the distance to the border of the glyph. The texture looks like this :

A nice interactive example would be this : https://mapbox.github.io/tiny-sdf/

Some observations :

1. We can get good looking text now for small and large text size. But making the text too large would still make it look 'kind of ugly'.

2. Increasing the size of the texture, increases the quality of the text rendered. But with a bigger size texture we would also be increasing the time required to make the texture itself (when loading a font) (The example above chops down the number of characters so it is not a proper example of this point).

3. The biggest disadvantage of this technique is that we would also need a proper layout engine for non-monospaced fonts.

Notice the irregular spacing in the resultant text.

Another method?

Another method we have is the one @Spongman has already worked on. This is by using the points we get using opentype and the knowledge of bezier curves. An interactive example of the approach is this : http://wdobbie.com/pdf/

This technique is harder on the GPU instead of the CPU because of the increased complexity of the shader code. But the text is crispier at any font size. To me, this seems better than the previous technique considering the drawbacks and the advantages. Also, looking at the examples already posted by @Spongman shows that we really don't have a performance issue as well.

I think that it is best to assign this issue to @Spongman as he has already done most of the work required for this task and go with the GPU based rendering approach. I'll be happy to look over other issues in p5.js. 😃

SDF fonts : http://hack.chrons.me/opengl-text-rendering/
Fonts on GPU : http://wdobbie.com/post/gpu-text-rendering-with-vector-textures/

[stalgiag]

Thank you for the thorough and well-researched post @AdilRabbani! I agree that @Spongman's approach is already working quite nicely and that your energy might be better spent elsewhere. There are still plenty of issues and features to tackle :-)

[Spongman]

indeed, that's great stuff. i hadn't seen the zooming demo before, it's pretty amazing that you can get that kind of performance from webgl!

if nobody has any objections, i'll clean up by branch a bit and submit a PR...

[stalgiag]

@Spongman that sounds good to me! Also, I suggest that @AdilRabbani works with you on reviewing the code for that PR since he has spent a significant amount of time researching this topic.