Subpixel rendering of texts

[kisvegabor]

In continuation of the related topic on the Forum.

cc @puzrin

[kisvegabor]

@puzrin I've noticed that box_w and adv_w are 3 times larger. Now I'd like to deal only with rendering and not with subpixel kerning. Do you think it's ok to simply round up the widths?
E.g. box_w = 11 ➡️ box_w = 4?

However, IMO the ofs_x should be rounded down.
E.g. ofs_x = 4 ➡️ ofs_x = 1.
What do you think?

[puzrin]

I'd suggest to keep things separate. Offsets in in glyphs are for "floating" bitmaps. Instead of inventing fractional offsets, it's enouth to shift bitmap's content (and increase bounding box if needed). Bounding box base corner position can be always integer.

When you receive font with subpixel data, the only difference is that bitmap has 3x density/pixels in selected direction. So, nothing to do with existing offset values.

Explanations above are in context of data structures, not depending on implementation. Is this what you asked about?

[kisvegabor]

I'd suggest to keep things separate. Offsets in in glyphs are for "floating" bitmaps. Instead of inventing fractional offsets, it's enouth to shift bitmap's content (and increase bounding box if needed).

Still not fully clear. So do you suggest if ofs_x = 4, skip one pixel and skip the first color channel?

[puzrin]

Probably i did not understand your question. I replied about font data.

If you asking about algorythm - i have no experience with those. If you wish to render subpixel font as ordinary one - that's not good idea.

Proper image downscale needs filtering to keep result sharp. There are tons of reading about such topic in imagemagick docs. I say so as author of https://github.com/nodeca/pica. In short - any simple approach will give bad result, just don't do that.

[kisvegabor]

Okay, thanks.
I can deal with it next week.

[kisvegabor]

Just quick hacky experiment:

[embeddedt]

I note that 2, 3, 5, and 6 appear out of place on the keyboard (i.e. too far to the right).

[kisvegabor]

I note that 2, 3, 5, and 6 appear out of place on the keyboard (i.e. too far to the right).

Thank you! Probably I make 1-pixel error in some cases. I'll investigate.

[pete-pjb]

Also the spacing of abc button looks a little off....
If that is helpful...

[puzrin]

@kisvegabor

I'm not sure it's correct to check subpixel rendering via screenshot. AFAIK a real photo with high resolution needed, to show how things look in your eyes.

Did result on your LCD improved significantly or not?

[embeddedt]

@puzrin As far as I know, screenshots captured on Linux simply retrieve the raw values of the logical RGB pixels. Therefore, it shouldn't affect the subpixel rendering.

Here's part of your comment scaled to 400% with no interpolation:

You'll have to click it to see it at proper size.

[kisvegabor]

@embeddedt By zooming into the image the result really won't be better. At first, I just wanted to see the red-ish pixels on the left and blue-ish pixels on the right.

[kisvegabor]

@puzrin
Here is something I don't understand with ofs_x in the generated font.
Both snippets contain only the "ABC" letters.

Normal:

    {.bitmap_index = 0, .adv_w = 0, .box_w = 0, .box_h = 0, .ofs_x = 0, .ofs_y = 0} /* id = 0 reserved */,
    {.bitmap_index = 0, .adv_w = 160, .box_w = 11, .box_h = 12, .ofs_x = 0, .ofs_y = 0},   //A
    {.bitmap_index = 43, .adv_w = 160, .box_w = 9, .box_h = 12, .ofs_x = 1, .ofs_y = 0},   //B 
    {.bitmap_index = 77, .adv_w = 160, .box_w = 10, .box_h = 12, .ofs_x = 0, .ofs_y = 0}   //C

with LCD flag:

    {.bitmap_index = 0, .adv_w = 0, .box_w = 0, .box_h = 0, .ofs_x = 0, .ofs_y = 0} /* id = 0 reserved */,
    {.bitmap_index = 0, .adv_w = 160, .box_w = 33, .box_h = 12, .ofs_x = 1, .ofs_y = 0},
    {.bitmap_index = 106, .adv_w = 160, .box_w = 26, .box_h = 12, .ofs_x = 1, .ofs_y = 0},
    {.bitmap_index = 188, .adv_w = 160, .box_w = 29, .box_h = 12, .ofs_x = 1, .ofs_y = 0}

With LCD flag all letters have ofs_x = 1. I don't know whether ofs_x has 1 px (I guess it is) or 1/3 px resolution but the difference seems odd.

If I manually change the ofs_x values the letter spaces are correct.
Original:

Manually fixed:

Do you have any idea?

[puzrin]

Here is main draw code.

We just extract bitmap & coords without change, those are integer. May be in subpixel mode we should care about lsb_delta & rsb_delta? I have no idea and would like to avoid diving into C api of FT. If you say exactly what should be done - that will be updated.

[puzrin]

May be ofs_NN of subpixel dimention is in 1/3 of pixels too?

[kisvegabor]

I have no idea. It might have 1/3 or 1 resolution but IMO it shouldn't be the same for the 3 gylphs. I've asked Jon_Smirl from the Forum to comment on this.

[puzrin]

@kisvegabor please use version from master:

• There was a bug with missed X sign.
• AdvanceWidth missed fractional part (not fatal, but kerning could be affected).
• We added a lot of FreeType glyph properties into dump. That should help to check if something goes wrong or incorrect property used.

I hope, that should fix all issues you asked about. X seems to be in pixels.

[kisvegabor]

@puzrin
Thank you for the update! It seems it has solved the issue!

[kisvegabor]

I've pushed the code to feat-subpixel. Now I set the type of font manually in the draw function.

@puzrin Can you add a flag to lv_font_t? E.g. subpixel = 1.

For backward compatibility, it might be useful to add this option only if it's 1. (Fields are initialized to 0 by default)

[puzrin]

Can you add a flag to lv_font_t? E.g. subpixel = 1.

What about vertical orientation? There are 2 options, --lcd & --lcd-v.

[kisvegabor]

Do you insists on --lcd-v? I'd be happy with only --lcd too.

[embeddedt]

@kisvegabor I think --lcd-v is necessary, because, depending on the orientation of the display, there might be a need to have vertical subpixel rendering.

[kisvegabor]

Correct me if I'm wrong but as I see there are two options:

1. "RGB" in lines: we can have subpixel rendering horizontally
2. "RGB" in columns: we can have subpixel rendering vertically (IMO not really important and more complicated because lines should be buffed)

So we cannot do horizontal sub-pixel rendering on vertical RGB display with --lcd-v.

[puzrin]

Do you insists on --lcd-v? I'd be happy with only --lcd too.

I do not insist on implementation. But i would insist on reserving value in data design. Since it's technically possible to have at least 2 types of subpixels layout (+ none), this should be expressed in lvgl writer somehow.

For bin writer i planned store this info in "compression" byte, but now tend to add independent header field (this change will be breaking).

[embeddedt]

So we cannot do horizontal sub-pixel rendering on vertical RGB display with --lcd-v.

Right. It would have to be vertical sub-pixel rendering.

[puzrin]

Landed non-breaking changes to 0.3.0 release of lv_convert, and pushed the rest to dev branch https://github.com/littlevgl/lv_font_conv/commits/dev

@kisvegabor, i need your decision about subpixel field size, to update lvgl writer. I'd suggest make it 2 bits wide at least.

[jonsmirl]

Don't confuse portrait and landscape mode LCDs, both of those want horizontal subpixel rendering. The only time you would use vertical subpixels is if you mounted the LCD wrong or if your display rotates. Vertical subpixel rendering has little visible impact. Check your display with a microscope and make sure the color bars are running horizontally, if they are you have it mounted correctly.

[jonsmirl]

The current code is looking great. It should be sufficient for most users. As far as I know this is the first implementation of subpixel rendering for microcontrollers.

If you want to improve it more -- play around with higher order filtering functions. These help but the improvements are not huge.

Implement subpixel spacing and kerning. This doesn't improve legibility but it lets you pack more characters onto a tiny display without loss of legibility.

[jonsmirl]

Now it is working, but something is inverted.

That is the opposite of what it should be doing. That's why you are seeing color halos
Text is "List" on black background. I made fonts like this...

jonsmirl@ares:/aosp/lv_font_conv$ ./lv_font_conv.js --font Roboto-Regular.ttf -r 0x20-0x7F --size 22 --format lvgl --bpp 3 --lcd -o lv_font_roboto_22.c
jonsmirl@ares:/aosp/lv_font_conv$ ./lv_font_conv.js --font Roboto-Regular.ttf -r 0x20-0x7F --size 12 --format lvgl --bpp 3 --lcd -o lv_font_roboto_12.c
jonsmirl@ares:/aosp/lv_font_conv$ ./lv_font_conv.js --font Roboto-Regular.ttf -r 0x20-0x7F --size 16 --format lvgl --bpp 3 --lcd -o lv_font_roboto_16.c
jonsmirl@ares:/aosp/lv_font_conv$

Is this code BGR and I have an RGB display?

[puzrin]

Convertor calls looks correct. If you have issues with speed regressions - try --bpp 4 --no-compress --force-fast-kern-format

Photo looks like everything except 0 & 255 is inverted.

[jonsmirl]

I figured out that I needed to remove #define LV_SUBPX_BGR 1
Here is subpixel 'W'

Here is non-subpixel 'W'

List

So it looks like it is working correctly! There are still some visible artifacts which can probably addressed with a higher order filter.

[puzrin]

@jonsmirl with what i see on screenshots, visual difference should be huge. Do you see it in normal scale?

[jonsmirl]

Here is photo from phone. This is cheapo 320x240 LCD. The word 'Chart' not in complete focus. It is hard to get everything in perfect focus.

[jonsmirl]

There is an obvious improvement vs non-subpixel. Here is not subpixel. The difference is much more visible to human eye than camera. My cell phone camera is processing the picture and it looks better in photo than it does in real life. Zoom in on "Write" and you can see how much clearer.

[jonsmirl]

Don't draw an opinion based photos. The camera is making the non-subpixel version look better than it really is. The camera processing is removing the brightness variations in various parts of the letters. I can easily see on display, but it is missing in photo.

To my human eye, the subpixel display is far better than the old version. I do see some slight color fringing on 'i' which can be removed with a filter.The biggest change is that the brightness across the glyph is now uniform. With old code some parts of glyph were brighter than others and that was very noticeable.

[puzrin]

@jonsmirl yes, difference is notable. Could you add --autohint-strong (only in subpixel mode)? It controls FT_LOAD_TARGET_LIGHT & FT_LOAD_TARGET_NORMAL flags.

https://github.com/littlevgl/lv_font_conv/blob/master/lib/freetype/index.js#L194-L195

I'd like to understand, will difference be notable and will it break kerning significantly (without subpixel mode FT_LOAD_TARGET_NORMAL is not very useful)

[jonsmirl]

This is with strong autohint on 'W'. Not sure what is going on, I have to enable LV_SUBPX_BGR to get this one to display right.

I don't see in difference in 'W'. Typically auto-hint does not reveal itself until font point size much lower.

[jonsmirl]

So in demo.c it makes a tabview with "Write List Chart" text.
How do I change the font size? Is default 12 or 16pt?

If I push the point down below 10 autohint will start to show up.

[jonsmirl]

What autohint does is start replacing the font glyphs with human designed specific pixels when the point size gets tiny. For example it ensures that the arm on an 'r' does not disappear. And it makes sure a 'k' does not turn into an 'h'. Things like that.

[embeddedt]

How do I change the font size? Is default 12 or 16pt?

You have to reconvert the font. You'll need to use the offline font converter as the online one hasn't been updated to support this feature.

[jonsmirl]

This is subpixel plus strong autohint and tiny points. Even tiny 7pt is still readable.
9pt

8pt

7pt

[kisvegabor]

On my TFT there wasn't a huge difference. But good to know that it's not the case on other TFTs.

In v6.1 I'll add a built-in font with subpixel rendering to show this feature.

Do you have any idea about updating filtering to avoid color bleeding?

[puzrin]

@kisvegabor #1210 (comment) - it's strange why @jonsmirl had to enable LV_SUBPX_BGR (and in previous post he mentioned disabling that). Display on photos has RGB subpixels (as majority of displays).

[jonsmirl]

The lower the DPI of the display, the more subpixel rendering helps. The LCD I am using is 96DPI and it makes an obvious difference.

I only had to turn BGR on when I turned on strong autohint. Maybe the fonts not getting generated right? Autohint is only needed on fonts below 10pt. Plus on below 10pt kerning doesn't really work so there is no conflict.

[puzrin]

I don't know how freetype output should be processed. I just send options and extract alpha bitmap. IMO if difference is not significant, we can forget about stronger autohinting and use light + subpixels.

In this scope it could be useful to optimize decompressor (fixing it will add notable value). Here i explained details.

[jonsmirl]

Font's below 10pt are probably of limited usefulness on an embedded LCD. Main use would likely be displaying license info in tiny text so that it all fits. A font point is 1/72th in. So a 10pt font should be 10/72in high. If you don't know the DPI of the display you can't calculate this.

Also note that the way we are doing things, a 10pt font on a 200DPI screen is half the size of a 10pt font on a 100DPI screen. Your desktop gets the screen DPI from the EDID. Since it knows the DPI it can make the 10pt font size consistent between the two screens. All of this is too complex for an embedded LCD that can't be swapped. What we have now is fine.

When someone complains that 10pt is not the right size, now you have the answer - we are working in pixels instead of inches.

[kisvegabor]

I only had to turn BGR on when I turned on strong autohint. Maybe the fonts not getting generated right? Autohint is only needed on fonts below 10pt. Plus on below 10pt kerning doesn't really work so there is no conflict.

I also don't know what might happen. It might be really worth a question on FreeType's mailing list.

In this scope it could be useful to optimize decompressor (fixing it will add notable value). Here i explained details.

True, subpixel rendering makes compression effective for normal-sized fonts too. The current algorithm is working but probably can be optimized to make it faster. However, I'd like to immerse in other things now. I leave it open for contribution 🙂

[jonsmirl]

If there are plans to make lvgl do auto layout, then it needs to learn about display DPI. For example if you have two 6in square LCDs and one is 200DPI and the other is 100DPI, auto layout can not make them look the same without knowing the DPI.

[kisvegabor]

@jonsmirl
There is an LV_DPI config in lv_conf.h. You can use it to set the paddings of layouts (e.g. style.body.padding.hor = LV_DPI / 2). You can also use it to set the sizes (e.g. lv_obj_set_size(btn, LV_DPI * 2, LV_DPI)). LittlevGL already uses LV_DPI to set the default sizes/values.

However, the fonts will be really smaller.

[jonsmirl]

LV_DPI needs to scale the point size too and then select the closest matching font.

How about swapping all of the default fonts for the subpixel ones in release 6 and see if anyone notices? This is trivial to undo, include the old fonts in the build and add a release note saying to copy them back if there are problems.

[kisvegabor]

using subpixel fonts has cost in two regards:

• they are 3 times larger in size
• they are ~50 % slower to render

My idea is to add a subpixel version too for the 12px built-in font to make it easier for people to test.

[jonsmirl]

They are not three times larger if you are replacing an 8b gray scale with a 3b (x3=9b) subpixel. And are they really 3X larger with compression?

A 2b subpixel (6b total) will likely look better than a 8b gray scale. And it should be smaller.

[embeddedt]

LV_DPI needs to scale the point size too and then select the closest matching font.

@jonsmirl Fonts are not chosen by point size in LittlevGL; you manually have to choose the font you want. So we don't really operate in terms of point size (like FreeType and other libraries do).

[kisvegabor]

They are not three times larger if you are replacing an 8b gray scale with a 3b (x3=9b) subpixel. And are they really 3X larger with compression?

@jonsmirl If you set bpp=3 in the font converter then it will generate "3b (x3=9b) subpixel" as you mentioned. However, it's really 3 times larger than 3b/px.

And are they really 3X larger with compression?

With fonts with more pixels, the compressor becomes more effective. However, decompressing increases rendering time further.

A 2b subpixel (6b total) will likely look better than a 8b gray scale. And it should be smaller.

You can try it out by setting bpp=2

[jonsmirl]

You should compare 3bpp subpixel to 8bpp normal. And then compare 1bpp subpixel to 3bpp normal. Those pairs have similar accuracy. It is not a good comparison between 8bpp normal to 8bpp subpixel, doing that is gives much higher fidelity to the subpixel case -- it is not an equal comparison.

I suspect the equivalent sub-pixel fonts may actually be a tiny bit smaller.

[puzrin]

They are not three times larger if you are replacing an 8b gray scale with a 3b (x3=9b) subpixel.

Nobody uses 8b here. 4b max. > 4b does not give visual difference. 3b is enougth IMO.

And are they really 3X larger with compression?

With compression those are ~ 2x larger. Compression ratio increases on size grow. To see almost exact size for your particular case - run convertor with binary output and try with and without --no-compress.

A 2b subpixel (6b total) will likely look better than a 8b gray scale. And it should be smaller.

I would not recommend 2b. Too bad precision. IMO 3b is optimal.

[jonsmirl]

It is not really 2b of precision, it is 2b per subpixel or 6b for the whole pixel.

[embeddedt]

6b for the whole pixel.

That's still more than 4b for the whole pixel.