fonts: provide opentype files for X11 bitmap fonts

[rnhmjoj]

Since 23edc9c bitmap fonts in PCF and BDF format stopped working in applications using pango as a backend (gtk apps for example).

Bitmap fonts packaged in nixpkgs should be updated to provide OTB (OpenType Bitmap) fonts. If unsupported upstream the xorg.fonttofsnt tool may be used to generate them.

Possibly affected packages:

• clearlyU (clearlyU: split outputs in out and otb #79638)
• creep (creep: split outputs in out and otb #79641)
• dina (dina: generate otb files with fontforge #79664)
• dina-pcf (dina: generate otb files with fontforge #79664)
• envypn-font (envypn-font: split outputs in out and otb #79666)
• gohufont (gohufont: generate opentype files #75163)
• powerline-fonts (already provides OTF and TTF)
• profont (profont: install otb variant #79756)
• proggyfonts (no upstream support or good results with ff or fonttosfnt)
• siji (siji: split output into out, bdf, otb #80220)
• spleen (spleen: install otf font #79671)
• tamsyn (tamsyn: generate otb and split output #80228)
• terminus_font (terminus_font: generate otb and split output #80261)
• tewi (tewi-font: generate otb and split output #80270)
• ucs-fonts (ucs-fonts: generate pcf, otb and split output #80366)
• unifont (unifont: generate otb and split output #80322)
• uni-vga (uni-vga: generate otb, psf and split output #80326)
• unscii (already provides OTF and TTF)
• uw-ttyp0 (uw-ttyp0: split outputs in out and otb #79633)

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
  - 'clearlyU'
  - 'creep'
  - 'dina'
  - 'dina-pcf'
  - 'envypn-font'
  - 'gohufont'
  - 'powerline-fonts'
  - 'profont'
  - 'proggyfonts'
  - 'siji'
  - 'spleen'
  - 'tamsyn'
  - 'terminus_font'
  - 'tewi'
  - 'ucs-fonts'
  - 'unifont'
  - 'uni-vga'
  - 'unscii'
  - 'uw-ttyp0'

# a list of nixos modules affected by the problem
module:

[lheckemann]

There's also the fonts in the xorg attrset such as fontmiscmisc.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-are-your-goals-for-20-03/4773/13

[multun]

Should we keep the former PCF / BDF fonts in?

[rnhmjoj]

Should we keep the former PCF / BDF fonts in?

Definitely: most software still use PCF/BDF fonts, especially older one.

[multun]

@rnhmjoj given the following powerline-fonts package and this i3 font config: font pango:Terminess Powerline, 10, i3 fails to load properly the font when the pcf / bdf fonts are present. It succeeds, however, when the two rm -rf commands are uncommented. Any idea why?

[rnhmjoj]

@multun: I'm not sure. It could be pango is loading the pcf first and failing. Does that happen with other fonts too?

[mtreca]

@multun I think that the fonttofsnt package messes up foundry names when converting bdf/pcf fonts. You could try exactly matching the font name using fc-match but in my experience it is better to keep a single version of your font on your system.

[mtreca]

@rnhmjoj I tried the font myself with awesomewm and I have the same issue. I indeed think that Pango loads the pcf/bdf version first.

I tried a disabling type1 bitmap fonts and/or bitmaps as a whole in fontconfig using

  fonts.fontconfig = {
    allowBitmaps = false;
    allowType1 = false;
  };

but it seems that otb fonts are still considered as bitmap fonts by fontconfig, so this doesn't work.

[rnhmjoj]

Void decided to downgrade pango to 1.43. In facts the 1.44 doesn't really bring any new feature (the package is in "maintenance mode") but only breakages: besides bitmap also lots of vector fonts now have spacing issues. We may consider this.

[jtojnar]

Many programs (e.g. Nautilus) already depend on Pango 1.44.

[mtreca]

Another solution that is found on the AUR is to provide two different packages, for instance gohufont and gohufont-otb. It duplicates a fair amount of code but at least there is no clashing between bitmapped and non-bitmapped fonts.

[rnhmjoj]

@Vxid: maybe we could use multiple outputs to avoid duplicating the packages.

[mtreca]

@rnhmjoj I did not know this was possible. That would be much cleaner indeed. I am not sure on how to do it though. I could change all the problematics don't packages if I had an blueprint on how to do so. I did the same thing with bitmap fonts on the AUR, by copying existing PKGBUILDs.

[rnhmjoj]

Before doing this on a scale, I'd like to solve the problem that fonttosfnt is not deterministic, or rather, it produces slightly different outputs on diferent architectures. Currently this breaks all fonts being fixed-output derivations.

[jtojnar]

Things generated by fonttosfnt probably should not be FODs anyway.

[rnhmjoj]

@jtojnar Why not? Fonts are just data: they shouldn't depend on the platform generating them and should be fixed outputs.

[rnhmjoj]

Good news: I sorted out the trouble with fonttosfnt in #75163.
I also reproduced the issue @multun and @Vxid described and I can say splitting the outputs seems to work:

• Installing the otb output works (apparently but I haven't tested extensively) for both X11 (using Xft) and GTK applications.

• Installing the out output works for X11 only (and console, optionally).

• Installing both still confuses pango resulting into broken glyphs. I opened an issue at pango about this.

The bad news is that fixed output and multiple outputs are incompatible so we'll have to choose one. I don't know if there is any workaround.

Another option I tried is renaming the font family to something like nameOT to avoid the confusion. It works but can't be done with fonttosfnt (yet, I could try to patch it); if the fonts are provided as BDFs it's pretty trivial to change the name with, say, sed.

[mtreca]

@rnhmjoj That's very good news. If the otb output works as intended for both GTK and X11 applications, I think your last solution involving renaming old bdf files would be the best. I am ready to test-drive some fonts to make sure they all display correctly after renaming them.

[rnhmjoj]

By the way, @jtojnar made me realized fixed output for a package that isn't strictly unpacking some source is a cheat: because an update to any dependency could theoretically change the result but nix won't rebuild the package. So it's actually better to remove the fixed output constraint and split the outputs.

I think your last solution involving renaming old bdf files would be the best.

I now think the best approach is to split the outputs. Renaming can't be done in general, because many fonts are provided as pcf and they can't be modified so easily (it requires writing a fontforge script). Also renaming the font will require people to change their configuration files.

[rnhmjoj]

You can check out my PR for gohufont.

[rnhmjoj]

You can install the unstable fonts easily:

$ nix-channel --add https://nixos.org/channels/nixos-unstable nixos-unstable

and in configuration.nix

{
    fonts.fonts = 
      let unstable = import <nixos-unstable> { };
      in [ unstable.uw-ttyp0.otb ];
}

[wucke13]

You can install the unstable fonts easily:

That is a good point.

I don't think it's a good idea to bring these changes to 20.03

I kindly disagree. As is well known, the whole pango issue already caused a lot of headache for many people, having it fixed in NixOS once and for all ASAP would be my personal preference.

[rnhmjoj]

I kindly disagree. As is well known, the whole pango issue already caused a lot of headache for many people, having it fixed in NixOS once and for all ASAP would be my personal preference.

Backporting these changes as is going to to very little: since the fonts are split in different outputs and OpenType fonts are not in the default one, GTK applications will still be broken by default.
Users must be made aware of the otb output and choose to use it: this is only possibile if the outputs are document in the manual or mentioned in the release changes.

Moverover, I was using the same bitmap font in GTK and X11 applications so I had to change my .Xresources to use the OpenType variant because the two can't be installed together.

This situation could change if Pango were to fix the font loading issue: I would just merge the outputs and backport the changes along with a pango patch.

[rnhmjoj]

I'm taking a look at the xorg fonts, now. The packages containing type 1 bitmap fonts apparently are:

  "fontadobe100dpi"
  "fontadobe75dpi"
  "fontadobeutopia100dpi"
  "fontadobeutopia75dpi"
  "fontadobeutopiatype1"
  "fontarabicmisc"
  "fontbh100dpi"
  "fontbh75dpi"
  "fontbhlucidatypewriter100dpi"
  "fontbhlucidatypewriter75dpi"
  "fontbhtype1"
  "fontbitstream100dpi"
  "fontbitstream75dpi"
  "fontbitstreamtype1"
  "fontcronyxcyrillic"
  "fontcursormisc"
  "fontdaewoomisc"
  "fontdecmisc"
  "fontibmtype1"
  "fontisasmisc"
  "fontjismisc"
  "fontmicromisc"
  "fontmisccyrillic"
  "fontmiscethiopic"
  "fontmiscmeltho"
  "fontmiscmisc"
  "fontmuttmisc"
  "fontschumachermisc"
  "fontscreencyrillic"
  "fontsonymisc"
  "fontsunmisc"
  "fontwinitzkicyrillic"
  "fontxfree86type1"

These are all collections of fonts and there are pcf, pfa and afm, which I'm not sure they are supported by fonttosfnt. Reading their fonts.dir they contain complexively 3439 fonts, ugh.

[rnhmjoj]

Having removed the vector fonts and run fonttosfnt we are left with:

    "fontadobeutopiatype1"
    "fontbhtype1"
    "fontbitstreamtype1"
    "fontibmtype1"
    "fontxfree86type1"

    "fontmicromisc"
    "fontsunmisc"
    "fontcursormisc"
    "fontdecmisc"

The first "type1" half simply can't be converted with this tool, the second half should but it's failing.
I'll have to try with fontforge again. Of course, all of these should be checked to see if the output is usable.

[rnhmjoj]

So, with #91777 and #95907 applications using pango should be able to load bitmap fonts without more user intervations.

The only problem left is the convertion of the xorg fonts but I give up on it: the current tools (fontforge, fonttosfnt) can't reliably handle most fonts and proportional bitmap fonts at all. I'll soon open a PR to convert the type1 vector fonts (xorg.font*type1) and post the expression I used to test, in case anyone else wants to try it.

[rnhmjoj]

Here's the expression: https://gist.github.com/rnhmjoj/bfa1af90c7f0f99b730eb8d4a7f6bd32

[mtreca]

@rnhmjoj Thanks a lot for your on bitmap fonts, I have been using the otb output you provided for the past six months without any issues.

[rnhmjoj]

Note that on unstable the otb output will disappear in the next channel update. So, if you're tracking unstable you're ok because pango will be updated at the same time with the fix needed; on the other hard, if you're on NixOS 20.03 you may have to pin the unstable channel, because the pango update won't be backported.

[schmittlauch]

So the only thing preventing this from getting closed seem to be proggyfonts because of its bad conversion result.

How do we want to proceed with this?

• keep this open until proggyfonts provides some OTFs upstream
• close the issue regardless of proggyfonts and maybe add a comment in its Nix expression
• close this but drop proggyfonts from nixpkgs

[rnhmjoj]

Well, yes but there are also hundreds of fonts in pkgs.xorg that I couldn't convert.
Removing the packages doesn't make sense because the fonts work perfectly fine in Freetype-based programs.
If the fonts are still actively supported I would create an upstream issue, in the meantime I think this can be kept open for visibility.

[stale]

I marked this as stale due to inactivity. → More info

[rnhmjoj]

It seems fonttosfnt has improved a bit since last time I tried it. It's worth trying another round of convertions.

[rnhmjoj]

Unfortunately no big changes: fonttosfnt is still producing fonts with broken metrics, mostly; fontforge seems to have gotten better, but still far from the original.

[stale]

I marked this as stale due to inactivity. → More info

[rnhmjoj]

I guess we can close this issue now: it doesn't look like there's much interest in converting the remaining fonts.