[Proposal] Optimal custom fonts using machine learning

[cdluminate]

The fonts used by us day to day are not specifically designed for printing images. In order to improve the result of character art, custom font is a doable solution, as proved by BE256 and BE512 fonts.

The highest resolution that chafa supports is 8x8 according to the bitmap. There are 2^64 possible combinations, but the whole Unicode table is not enough for that. However, my bold assumption is that most solutions in the space formed of 2^64 combinations are useless.

To find out the most useful N patterns in this space, we can take advantage from machine learning.

The proposed procedure for creating such custom font looks like this:

1. sample M random crops (in ratio w=1:h=2) from an image dataset

2. turn the M crops into binarized bitmaps using histogram and downsample to 8x8

3. find N cluster centers in the space by leveraging Kmeans algorithm, with the M (64x1) binaryzed vectors as the dataset.

4. convert the N vectors into C bitmap header and SVG plots.

Highlight:

1. Best resolution.

2. Easy to code. Automatic font and C code generation.

3. I don't know what glyph is good for printing character art, but the algorithm can figure it out.

Assignee: myself

[hpjansson]

Sounds like an interesting project. We could definitely ship such a font with Chafa and support it with --symbols.

[cdluminate]

Now the goal of this issue turned into

1. find the optimal set of glyphs
2. documentation
3. autotools: add option to enable this feature

[cdluminate]

Let's move forward a bit: update the font generator and really ship a (basically working) font file?

[hpjansson]

Let's move forward a bit: update the font generator and really ship a (basically working) font file?

Sounds good to me.

I don't like to keep generated blobs in git, but it's important to keep the build and installation process simple, so here's what I think we should do:

• Keep the JSON from the last good training run in git, and also the TTF. Both gzipped.
• Have Makefile rule for generating $(JSON).gz -> $(TTF).gz, disabled by default. Configure switch kind of like --enable-gtk-doc. This depends on Python + modules (e.g. fontforge).
• Install the TTF file and chafa8x8.py always (maybe call chafa8x8.py something else to avoid confusion).

We may also have to keep the generated C source (also gzipped), but I'd prefer to load it dynamically, either from the JSON or the TTF. I'll have to see what's the best solution here, maybe we should be using a more compact format instead of JSON.

How does that sound?

[cdluminate]

Sounds good to me.

maybe we should be using a more compact format instead of JSON.

Any suggestion? I think JSON is just the format of best compatibility, even if not the most compact one.

Besides, we can add a configure flag called, e.g. --enable-kmeans-font which triggers the TTF file generation, and defines a C macro like CHAFA_HAS_KMEANS_FONT:

#ifdef CHAFA_HAS_KMEANS_FONT
#include "auto-generated.c"
#endif

[hpjansson]

Yeah, I was just thinking about the overhead of parsing about a megabyte of JSON on startup. However, I think we could go for a better solution where Chafa just loads the glyphs right out of the font file. That would be optimal, since we could adapt to any font and get better results in e.g. ascii mode too. It should be enough to link with FreeType or Harfbuzz, which are very common and low in the stack.

[hpjansson]

Chafa would then have a switch, e.g. --font-glyphs fontfile.ttf which would import glyphs from a font and map them to their respective Unicode code points. You could even specify it multiple times. Then there would be --symbols range where you could specify that you want to allow custom code point ranges, e.g. the one used in the k-means font. Using those two switches together you would get the desired result.

[cdluminate]

Cool! I like this idea.

[hpjansson]

It's in master now. Here's how to use it:

chafa --glyph-file chafa8x8.ttf --symbols 0x100000..0x101000

The font loading is a little bit slow, and I need to fine tune the bitmap generator, but I'm already getting improved output with e.g. chafa --glyph-file ter-x12n.pcf --symbols all where the font file corresponds to the Terminus font I'm using in the terminal.

[cdluminate]

Nice. Now I think the C code generation part can be safely removed from fontgen. Will submit a PR to overhaul fontgen when I got enough time to work on it.

[clort81]

This looks fun.
Grabbed a bunch of images and put them in ~/coco

$ ./chafa8x8.py CreateDataset --glob "coco/*.jpg"
Traceback (most recent call last):
  File "/media/sd/Projects/TermFun/chafedit/tools/fontgen/./chafa8x8.py", line 15, in <module>
    from sklearn.cluster import KMeans, MiniBatchKMeans
ModuleNotFoundError: No module named 'sklearn'

alright then

$ pip3 install sklearn
Requirement already satisfied: sklearn in /usr/local/lib/python3.10/dist-packages (0.0.post1)
$ pip3 install KMeans
Requirement already satisfied: KMeans in /usr/local/lib/python3.10/dist-packages (1.0.2)
$ pip3 install MiniBatchKMeans
ERROR: Could not find a version that satisfies the requirement MiniBatchKMeans (from versions: none)
ERROR: No matching distribution found for MiniBatchKMeans

Ah yes, python is satan. nevermind.

FWIW for my textart, braille already gives an 2x4 matrix that resolves everything quite well. Particularly if your font uses 'full block' braille glyphs.

Beyond that resolution there are few gains due to two-color limitation.