(WIP) Dynamic glyph rasterization and unlimited number of font textures

[georgegerdin]

This PR is my attempt at bringing dynamic glyph rasterization and unlimited number of font textures to Dear ImGui. This is my proposal to fix #3365 #3406 #797 and all issues related to chinese glyphs, japanese glyphs etc...

Only the glfw_opengl2 and glfw_opengl3 backends are currently supported. The directx backends where changed to let me compile the example but do not work. I intend to fix them but wanted to show you my results before I get time for that. Tested on windows. Freetype, baked anti-aliased lines, software rendered mouse cursor and custom rects are not yet supported. The white pixel is located at (0, 0) on all generated font textures.

ImFontAtlas and ImFont has been completely revamped to generate glyphs upon use and packed in the same manner as fontstash. The rendering of the final text is the same as normal Dear ImGui. Instead of each font atlas having a personal FontSize the ImFontAtlas has a FontSize common to all loaded fonts.

Future plans is adding the missing features from normal Dear ImGui font code and unloading glyphs that have not been used for a certain amount of time.

[ocornut]

Great work, thanks for posting this!
It's great to see a working proof of concept for that.

I haven't dug in this, but some early suggestions:

• Try to use standardized greppable markers for known issues (e.g. FIXME-DYNAMICFONT). It seem to have lots of commented code/features feature and we'd need some work of tracking to figure out all the whys and what-to-do. This generally makes a PR less overwhelming to review and parse through.
• I think we can separate and abstract the work on back-ends. For shadows we started working on (unpushed) api to have a standardized way of requesting back-end to reload the texture or part of it. It should be fairly simple at a first glance but in practice when (rarely) need to rewrite already written pixels (e.g. for compacting/garbage collecting) we have an issue with in-flight frames, which separately impact the interface with back-ends. That's a whole separate topic we could work on separately, so I suggest for now not worrying too much about the back-ends aspect. Hopefully once we have that protocol the font system can only say "here are some dirty rectangles, let the back-end figure it out". Having this PR along with the work on shadows is a great combination to test that feature.
• We would need to allow resizing textures. Since this would be a very rare event, my plan to was to "simply" scale the UV of all submitted vertices (for which the TextureID is matching) so it become possible to do it mid-frame without impacting the frame.
• GIF has visible glitches with missing/flickering characters during rescale (haven't investigated).
• GIving you access to imgui_dev repo if you want to run some perf tests/comparaison. I need to stress that in all of this the performance requirement are quite paramount, since CalcTextSize() can be bit of a bottleneck in extremely large (clipped) ui.

[georgegerdin]

Thank you for the feedback Omar. I have added FIXME-DYNAMICFONT comments in the places where I soon intend to fix features. The remaining commented blocks are for CustomRect and baked antialiased lines which is not yet supported. I think the flickering is because I am currently generating glyphs in one frame and uploading to the graphics card in the next frame but this should be easy to fix by uploading at the end of the current frame instead.

[ocornut]

I noticed in a gif you posted that it fallbacks to many textures, but then how are drawcalls handled? Wouldn’t this lead to drawcalls explosion very fast?

[georgegerdin]

There is only one draw call per font texture used. I have not implemented resizing font textures yet since I thought multiple font textures would still be a prerequisite to make it scalable since there is a limit how large a texture can be but no limit on how many textures there can be.

ImFont::RenderText works like this: (lines correspond to latest commit 986738c)

1. Loop through font characters and FindGlyph for each character like in master ImFont::RenderText
2. The first glyph decides the primary font texture (line 3030-3035)

PrimReserve data for vertices and indices(line 3038 in latest push) just like in master ImFont::RenderText (line 3037-3044). This needs wait until here because we need to know which is the primary font texture.

2. If the glyph is located on another font texture than the primary one:

1. If this is the first time this font texture is used create a FontRenderList job for this font texture (line 3061 - 3068)
2. Store the glyph data for later rendering (line 3125-3157) (same logic as master but this data will be merged to the draw list at the end of the function

3. If the glyph is on the primary font texture render it to the draw list like in master (line 3219-3228)

4. When all characters have been iterated clean up the vertex and index buffers (line 3238-3248)

5. memcpy the characters on other font textures(collected in step 2) from the FontRenderList jobs to the end of the draw list (line 3255-3276).

In summary:

1. In the worst case scenario the number of drawcalls equal the number of font textures.
2. The number of draw calls equal the number of font textures the RenderText call needs to use depending on which font texture the glyph lives.
3. If all glyphs are located on the first font texture there should be no difference from before.

My latest push fixes the flicker issues.

[ocornut]

My latest push fixes the flicker issues.

Great!

multiple font textures would still be a prerequisite

I think the the draw call explosion would be so problematic it is saner to focus on a system when we either

• 1. never ever expect multiple font textures.
• 2. very rarely expect multiple font textures and recover from it.
     So focusing on e.g. eviction/gc.

In the worst case scenario the number of drawcalls equal the number of font textures

But to be z-order preserving that's per call to RenderText()? And we want to support thousands of calls to RenderText() every frame, interleaved with shapes. To minimize it guess we could have a scheme were:

• Texture data to support shape rendering could be present in every texture.
• Garbage collection would reassign/move glyphs to reassemble fonts together in same texture.

I've added you the imgui_dev/ repository btw, when you run test_app under "Perfs" you can run perf tests (and probably add new ones) to compare branches, a critical part of this feature is and has always been performance of CalcTextSize() and RenderText() in the normal idle rendering case, for both Debug and Optimized builds, so it would be good to have early metrics of how the current version fare and see if we are within an acceptable ballpark for perfs.

Thanks!

[ClaasJG]

Suggested change

	*backup_glyph = NULL; // This glyph has already been rendered and uploeaded to the graphics card. No backup glyph needed.
	*backup_glyph = NULL; // This glyph has already been rendered and uploaded to the graphics card. No backup glyph needed.

[georgegerdin]

I think the the draw call explosion would be so problematic it is saner to focus on a system when we either

* 1. never ever expect multiple font textures.

* 1. very rarely expect multiple font textures and recover from it.
     So focusing on e.g. eviction/gc.

In the worst case scenario the number of drawcalls equal the number of font textures

But to be z-order preserving that's per call to RenderText()? And we want to support thousands of calls to RenderText() every frame, interleaved with shapes. To minimize it guess we could have a scheme were:

* Texture data to support shape rendering could be present in every texture.

* Garbage collection would reassign/move glyphs to reassemble fonts together in same texture.

I've added you the imgui_dev/ repository btw, when you run test_app under "Perfs" you can run perf tests (and probably add new ones) to compare branches, a critical part of this feature is and has always been performance of CalcTextSize() and RenderText() in the normal idle rendering case, for both Debug and Optimized builds, so it would be good to have early metrics of how the current version fare and see if we are within an acceptable ballpark for perfs.

Thanks!

I will focus on garbage collection and rearranging the glyphs to allow them to remain on the first font texture.

These are the results I get when comparing to vanilla imgui. Am I reading it right that my version in -O3 is about a millisecond slower per frame? Since the tests results are similar when displaying proggy-clean on texture 1 and roboto-medium on texture 2 in release mode I think it is an indication that most of the disparity compared to master are optimizations I have not yet implemented.

With master imgui -O0 (Debug):

perf,perf_draw_text_too_long,1.674,x5,master,Release,X64,Linux,GCC,2020-09-28
perf,perf_draw_text_too_long_clipped,0.764,x5,master,Release,X64,Linux,GCC,2020-09-28
perf,perf_draw_text_too_long_wrapped,2.382,x5,master,Release,X64,Linux,GCC,2020-09-28
perf,perf_draw_text_too_long_wrapped_clipped,1.123,x5,master,Release,X64,Linux,GCC,2020-09-28

With master imgui -O3 (Release):

perf,perf_draw_text_too_long,0.894,x5,master,Release,X64,Linux,GCC,2020-09-30
perf,perf_draw_text_too_long_clipped,0.372,x5,master,Release,X64,Linux,GCC,2020-09-30
perf,perf_draw_text_too_long_wrapped,1.348,x5,master,Release,X64,Linux,GCC,2020-09-30
perf,perf_draw_text_too_long_wrapped_clipped,0.554,x5,master,Release,X64,Linux,GCC,2020-09-30

With dynamic fonts -O0 (Debug): (Proggy-Clean on font texture 1)

perf,perf_draw_text_too_long,4.230,x5,master,Release,X64,Linux,GCC,2020-09-30
perf,perf_draw_text_too_long_clipped,2.148,x5,master,Release,X64,Linux,GCC,2020-09-30
perf,perf_draw_text_too_long_wrapped,6.402,x5,master,Release,X64,Linux,GCC,2020-09-30
perf,perf_draw_text_too_long_wrapped_clipped,3.508,x5,master,Release,X64,Linux,GCC,2020-09-30

With dynamic fonts -O3 (Release): (Proggy-Clean on font texture 1)

perf,perf_draw_text_too_long,2.035,x5,master,Release,X64,Linux,GCC,2020-09-30
perf,perf_draw_text_too_long_clipped,0.980,x5,master,Release,X64,Linux,GCC,2020-09-30
perf,perf_draw_text_too_long_wrapped,2.871,x5,master,Release,X64,Linux,GCC,2020-09-30
perf,perf_draw_text_too_long_wrapped_clipped,1.407,x5,master,Release,X64,Linux,GCC,2020-09-30

With dynamic fonts -O0 (Debug): (Roboto-Medium on font texture 2)

perf,perf_draw_text_too_long,4.228,x5,master,Release,X64,Linux,GCC,2020-09-30
perf,perf_draw_text_too_long_clipped,2.123,x5,master,Release,X64,Linux,GCC,2020-09-30
perf,perf_draw_text_too_long_wrapped,6.337,x5,master,Release,X64,Linux,GCC,2020-09-30
perf,perf_draw_text_too_long_wrapped_clipped,3.357,x5,master,Release,X64,Linux,GCC,2020-09-30

With dynamic fonts -O3 (Release): (Roboto-Medium on font texture 2)

perf,perf_draw_text_too_long,2.099,x5,master,Release,X64,Linux,GCC,2020-09-30
perf,perf_draw_text_too_long_clipped,1.058,x5,master,Release,X64,Linux,GCC,2020-09-30
perf,perf_draw_text_too_long_wrapped,2.991,x5,master,Release,X64,Linux,GCC,2020-09-30
perf,perf_draw_text_too_long_wrapped_clipped,1.420,x5,master,Release,X64,Linux,GCC,2020-09-30

[ocornut]

Am I reading it right that my version in -O3 is about a millisecond slower per frame?

Thanks for the update! Yes that's correct. Another way to read it is to state it is currently about 2-2.5 slower in both -O0 and -O3.

For info, the Perf function measure the base dt before starting the test, so the time display is the added difference from that baseline (actual app may be e.g. an extra 1 ms to do the swap etc.).

Those specific four tests are stressing one specific aspect - hammering text functions only. Since this is not really what a real app does, maybe we'd get more real-world representative number by running the perf_demo_all test which opens everything in the demo (and fix their pos/size to ensure comparable clipping across sessions). For large amount of clipped content there also will be more calls to CalcTextSize without the DrawText part (since rendering tends to be limited to visible stuff). There the ratio should comparatively be smaller as more time will be spent in other ImGui code, layout, clipping etc.

Later on, another test which will be worth pursuing will be with lots of glyphs (e.g. rendering a typical of Chinese text) as with the new algorithm the hashing will be stressed a little more. Depending on how it far we can adjust the hashing, or perhaps even have specialization for ascii ranges of if find it is worth it.

(hope we can later work on systems to turn that CSV data into easy to visualize stuff)
(do you happen know which ifdef we could tap into to automatically set build_info.Type to "Debug" in ImBuildGetCompilationInfo() with GCC? i'm not using GCC much here)

[frink]

FWIW: I just found this header-only font library tonight that seems amazingly close to the overall architectural methodology of Dear Imgui. It's quite different from what you're doing in this PR but might still be an interesting read...

https://github.com/kv01/ttf-parser - Seems like a mature abandon'ed project...

I really don't know much about how the font stuff works. Might be a complete waste of time. But anyway. Just trying to be friendly and pass something interesting along. It's about a thousand lines, comments and all, so should be a quick read. License is identical too so you are free to "borrow code" if it does end up being useful.

If it helps ,great. If it's a distraction, sorry about that...

In any case, keep up the good work. Looking forward to dynamically re-sizable fonts!!

[georgegerdin]

I left a PR on how to change the makefile to detect debug mode on Linux. I don't think you can do it only using a #define.

frink: Thank you, but that looks like a parser of .ttf-files and I am still using STB or optionally Freetype for rendering those like vanilla imgui. This branch is about how imgui could handle the generated glyphs on the font texture to display them in different sizes while remaining fast. So it is a step after the library you are linking to. Freetype is a heavier dependency but more visually pleasing and the industry standard.

[frink]

Yes it is a parse that parses straight to GL triangles if I understand correctly. Not sure if it's better. Just different. First time I've seen GPU engaged to rasterize fonts. My thought is that if we're using GL for everything already why rasterize fonts as an intermediary step? You've probably got a better but it seemed like an interesting idea...

[Khodyn]

Any news on when this will be merged? This is exactly what I need.

[frink]

Any news on when this will be merged? This is exactly what I need.

Could be a while. Good code takes time and this one is no picnic.

Right now the best thing you can do is oversample your font atlas in the init stage and then use font scaling.
This PR will be better than that method but should not require a large refactoring of code once this is release.
I've found that oversampling is tolerable in most situations...

[georgegerdin]

In the latest update only one font texture is used 99% of the time. When the primary font texture is full glyphs spill over to overflow texture(s). On the next frame all glyphs are moved to a cache and the font atlas re-assembled with the custom rects and the currently visible glyphs.

This is ocornut:master on my machine:
1624793551148158,perf,perf_demo_all,0.858,x5,master,Release,X64,Linux,GCC,2021-06-27

This is the dynamic font branch on my machine (no glyph overflows triggered):
perf,perf_demo_all,0.860,x5,master,Release,X64,Linux,GCC,2021-06-27
perf,perf_demo_all,0.798,x5,master,Release,X64,Linux,GCC,2021-06-27