Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some Iosevka variants broken #694

Closed
4 tasks done
Finii opened this issue Nov 29, 2021 · 10 comments
Closed
4 tasks done

Some Iosevka variants broken #694

Finii opened this issue Nov 29, 2021 · 10 comments
Labels

Comments

@Finii
Copy link
Collaborator

Finii commented Nov 29, 2021

🗹 Requirements

  • A brief but descriptive title of your issue
  • I have searched the issues for my issue and found nothing related and/or helpful
  • I have read or scanned the FAQ
  • I have read or scanned the Wiki

🎯 Subject of the issue

The patched Iosevka Mono fonts are more or less broken

Some Iosevka Mono fonts can not be opened at all (or it takes more than some minutes), while most other open with error messages of this kind:

patched-fonts/Iosevka/Thin-Italic/complete/Iosevka Thin Italic Nerd Font Complete Mono.ttf
PythonUI_Init()
copyUIMethodsToBaseTable()
Internal Error:
Subtable status not filled in for 0th subtable of 'calt' Contextual Alternates lookup 26
Subtable status not filled in for 1th subtable of 'calt' Contextual Alternates lookup 26
Subtable status not filled in for 2th subtable of 'calt' Contextual Alternates lookup 26
Subtable status not filled in for 0th subtable of 'dlig' Discretionary Ligatures lookup 32
Subtable status not filled in for 1th subtable of 'dlig' Discretionary Ligatures lookup 32
Subtable status not filled in for 2th subtable of 'dlig' Discretionary Ligatures lookup 32
Subtable status not filled in for 0th subtable of Reverse Contextual Chaining Substitution lookup 35
Subtable status not filled in for 1th subtable of Reverse Contextual Chaining Substitution lookup 35
Subtable status not filled in for 2th subtable of Reverse Contextual Chaining Substitution lookup 35
Subtable status not filled in for 0th subtable of Reverse Contextual Chaining Substitution lookup 38
Subtable status not filled in for 1th subtable of Reverse Contex

I believe fontforge tries to fix the fonts that do not open for extended times like this:

...
Attempt to reference lookup 11293 (within a contextual lookup), but there are only 429 lookups in 'GSUB'
Attempt to reference lookup 11805 (within a contextual lookup), but there are only 429 lookups in 'GSUB'
Attempt to reference lookup 12317 (within a contextual lookup), but there are only 429 lookups in 'GSUB'
Attempt to reference lookup 14109 (within a contextual lookup), but there are only 429 lookups in 'GSUB'
Attempt to reference lookup 14877 (within a contextual lookup), but there are only 429 lookups in 'GSUB'
Attempt to reference lookup 15901 (within a contextual lookup), but there are only 429 lookups in 'GSUB'
Attempt to reference lookup 23837 (within a contextual lookup), but there are only 429 lookups in 'GSUB'
Attempt to reference lookup 24605 (within a contextual lookup), but there are only 429 lookups in 'GSUB'
Attempt to reference lookup 25373 (within a contextual lookup), but there are only 429 lookups in 'GSUB'
Attempt to reference lookup 25885 (within a contextual lookup), but there are only 429 lookups in 'GSUB'
Attempt to reference lookup 27933 (within a contextual lookup), but there are only 429 lookups in 'GSUB'
Attempt to reference lookup 28701 (within a contextual lookup), but there are only 429 lookups in 'GSUB'
Attempt to reference lookup 47901 (within a contextual lookup), but there are only 429 lookups in 'GSUB'
Attempt to reference lookup 49181 (within a contextual lookup), but there are only 429 lookups in 'GSUB'
...
[on and on is scrolls]

I guess the problem is the ligature removal with the src/unpatched-fonts/Iosevka/config.json.

🔧 Your Setup

  • Which font are you using (e.g. Anonymice Powerline Nerd Font Complete.ttf)? Iosevka Mono *
  • Which terminal emulator are you using (e.g. iterm2, urxvt, gnome, konsole)?
    none, fontforge (20201107) to just open the font
    Used both:
 Based on sources from 2021-01-15 15:55 UTC-ML-D-GDK3.
 Based on sources from 2021-11-29 12:07 UTC-ML-D-GDK3. (HEAD)
  • Are you using OS X, Linux or Windows? And which specific version or distribution? Ubuntu 21.10

★ Optional

Maybe these are related:
#585 #586

image

@Finii Finii added the ⚠ bug label Dec 1, 2021
@Finii Finii changed the title Iosevka Mono variants broken Some Iosevka variants broken Dec 10, 2021
@Finii
Copy link
Collaborator Author

Finii commented Dec 10, 2021

An especially severe problem have these two fonts:

iosevka-heavyoblique
iosevka-term-heavyoblique

After (any, even manual) patching, the patched result can not even be opened with fontforge(*).

I even tried the newest release v11.2.0.

This has nothing to do with the ligature removal (opposed to what I wrote above).

Note the error message in the bottom:

image

(*) Opened programatically via Python. It opens in interactive fontforge after a long long wait.

@GulajavaMinistudio
Copy link

I'm also facing same issue. I patch Iosevka 11.2.3 and want to keep built in ligature on Iosevka . Successfully patched with some error warning like this. "Internal error: Internal Error: Attempt to output 66740 into a 16-bit field. It will be truncated and the file may not be useful."

Screen Shot 2021-12-29 at 07 34 53

And after install it on Linux Ubuntu, and try in VS Code text editor, the patched font didn't work properly. It not show ligature. But if i use unpatched Iosevka font, it work properly.

@Finii
Copy link
Collaborator Author

Finii commented Dec 30, 2021

Hmm, at this point fontforge creates the message (inside putshort()) (i.e. this is the caller).

image

@Finii
Copy link
Collaborator Author

Finii commented Jan 3, 2022

Well, it has nothing to do with our patching.

While I can open src/unpatched-fonts/Iosevka/Regular/iosevka-term-regular.ttf and generate a new unchanged font in fontforge interatively, I can not do that in Python mode:

image

@Finii
Copy link
Collaborator Author

Finii commented Jan 3, 2022

WHAT THE F***!

It does save correctly if we supply an empty flag tuple (instead of no flags):

#!/usr/bin/env python
import fontforge

font = fontforge.open('src/unpatched-fonts/Iosevka/Regular/iosevka-term-regular.ttf')
font.generate('iosevka_blah.ttf', flags=tuple())

😭

@Finii
Copy link
Collaborator Author

Finii commented Jan 3, 2022

/* Legacy screw ups mean that opentype & apple bits don't mean what */
/*  I want them to. Python users should not see that, but fix it up */
/*  here */
if ( (iflags&0x80) && (iflags&0x10) )   /* Both */    
    iflags &= ~0x10;
else if ( (iflags&0x80) && !(iflags&0x10)) /* Just opentype */
    iflags &= ~0x80;
else if ( !(iflags&0x80) && (iflags&0x10)) /* Just apple */
    /* This one's set already */;
else
    iflags |= 0x90;
    
[...]

                fm_flag_apple = 0x10,
                fm_flag_opentype = 0x80,

If we set "apple" and "opentype" -> fm_flag_opentype
If we set "opentype" -> 0
If we set "apple" -> fm_flag_apple
If we set nothing -> fm_flag_apple | fm_flag_opentype

@Finii
Copy link
Collaborator Author

Finii commented Jan 3, 2022

If we set no flags by setting an empty tuple the correction will happen and we in fact set apple | opentype.

That will later be translated here:

old_sfnt_flags = 0;
switch ( fmflags&(fm_flag_apple|fm_flag_opentype) ) {
  case fm_flag_opentype:
    old_sfnt_flags |= ttf_flag_applemode|ttf_flag_otmode;
  break;
  case fm_flag_apple|fm_flag_opentype:
    /* Neither */;
  break;
  case fm_flag_apple:
    old_sfnt_flags |= ttf_flag_applemode;
  break;
  case 0x00:
    old_sfnt_flags |= ttf_flag_otmode;
  break;
  default:
  break;
}

If we set "apple" and "opentype" -> fm_flag_opentype -> ttf_flag_applemode | ttf_flag_otmode
If we set "opentype" -> 0 -> ttf_flag_otmode
If we set "apple" -> fm_flag_apple -> ttf_flag_applemode
If we set nothing -> fm_flag_apple | fm_flag_opentype -> 0

If we set no flags by ... just not mentioning, we use -1 as flags and the correction does never happen (instead we use "Default to what we did last time" 😬)
Which means

int old_sfnt_flags = ttf_flag_otmode;

@Finii
Copy link
Collaborator Author

Finii commented Jan 3, 2022

Interactive mode (default) works, because it uses apple mode and not opentype.

image

It fails with the same error if opentype is selected.
So I would say this is an fontforge error, and we can not really do something about it (apart from fixing upstream).

Finii added a commit that referenced this issue Jan 3, 2022
[why]
For some reason fontforge is unable to pack all the lookup tables of
Iosevka back into the generated font, although they fit obviously in the
source font.

The lookup tables will become garbled and fontforge says something like:

  Attempt to output 67666 into a 16-bit field.
  It will be truncated and the file may not be useful.

As this also happens if we do nothing, just open and generate the font
without touching, there is not much we can do.

[how]
The only thing we can do is reduce the number of lookup tables that are
in the font file. This is somewhat brute, but at least we end up with a
usable font, albeit with a little bit less features.

Unfortunately the SS (Style Set) tables are not enough, but the CV
(Character Variants) suffice. So all CV lookups are riped out.

Fixes: #694

Signed-off-by: Fini Jastrow <ulf.fini.jastrow@desy.de>
Finii added a commit to Finii/fontforge that referenced this issue Jan 11, 2022
[why]
Sometimes, especially when generating fonts with a lot ligatures,
fontforge can not create the font file correctly. A typical error
message looks like this:

  Attempt to output 68142 into a 16-bit field.
  It will be truncated and the file may not be useful.

This will already happen when the font is opened and generated without
any change. But the generated font must contain the lookups of course.

[how]
When the lookup tables are written we need to check if the indices can
be represented by 16 bit values. Big tables will often be longer.

In the file first is comes the lookups table that points to all lookups.
Then follow the lookups themselves.

When a lookup would violate the 16 bit boundary we need a different and
longer (!) entry in the lookups table, so all lookups will be shifted.
For this reason the function
  tottfgpos.c:g___FigureExtensionSubTables()
does this calculation recursively. Increasing the size of the lookups
table step by step for each lookup, and then checking the previous
lookups again (because they shifted).

Unfortunately the check if the 16 bit boundary would be violated does
not take the grown lookups table into account. Under some circumstances
this means that a lookup, that fittet just-so before is now shifted
because of the bigger lookups table and will then be too far away (i.e.
index > 16 bit).

In the calculation the `len` (size increase) of the lookups table needs
to be included.

[note]
In the `fontforge/tottfgsub.c`:

        g___FigureExtensionSubTables()

        3179: if ( sub->subtable_offset+offset>65535 )
        3218: sub->subtable_offset += len;

and later:

        dumpg___info()

        3410: efile=g___FigureExtensionSubTables(all,offset,is_gpos);
        3423: putshort(g___,offset+sub->subtable_offset);

On line 3179 is decided if a figure-extension has to be used, from the
original offsets sum.

One offset is then changed on line 3218.

On line 3423 we push out the changed offset sum as 16 bits, but did not
check if the changed (3218) offset values would fit.

There is this comment nearby in the code

        /* Offset to lookup data which is in the temp file */
        /* we keep adjusting offset so it reflects the distance between */
        /* here and the place where the temp file will start, and then */
        /* we need to skip l->offset bytes in the temp file */
        /* If it's a big GPOS/SUB table we may also need some extension */
        /*  pointers, but FigureExtension will adjust for that */

[note]
This fix has been triggered by
ryanoasis/nerd-fonts#694

Reported-by: Gulajava Ministudio <GulajavaMinistudio>
Reported-by: Rui Ming (Max) Xiong <xsrvmy>
Signed-off-by: Fini Jastrow <ulf.fini.jastrow@desy.de>
jtanx pushed a commit to fontforge/fontforge that referenced this issue Jan 21, 2022
[why]
Sometimes, especially when generating fonts with a lot ligatures,
fontforge can not create the font file correctly. A typical error
message looks like this:

  Attempt to output 68142 into a 16-bit field.
  It will be truncated and the file may not be useful.

This will already happen when the font is opened and generated without
any change. But the generated font must contain the lookups of course.

[how]
When the lookup tables are written we need to check if the indices can
be represented by 16 bit values. Big tables will often be longer.

In the file first is comes the lookups table that points to all lookups.
Then follow the lookups themselves.

When a lookup would violate the 16 bit boundary we need a different and
longer (!) entry in the lookups table, so all lookups will be shifted.
For this reason the function
  tottfgpos.c:g___FigureExtensionSubTables()
does this calculation recursively. Increasing the size of the lookups
table step by step for each lookup, and then checking the previous
lookups again (because they shifted).

Unfortunately the check if the 16 bit boundary would be violated does
not take the grown lookups table into account. Under some circumstances
this means that a lookup, that fittet just-so before is now shifted
because of the bigger lookups table and will then be too far away (i.e.
index > 16 bit).

In the calculation the `len` (size increase) of the lookups table needs
to be included.

[note]
In the `fontforge/tottfgsub.c`:

        g___FigureExtensionSubTables()

        3179: if ( sub->subtable_offset+offset>65535 )
        3218: sub->subtable_offset += len;

and later:

        dumpg___info()

        3410: efile=g___FigureExtensionSubTables(all,offset,is_gpos);
        3423: putshort(g___,offset+sub->subtable_offset);

On line 3179 is decided if a figure-extension has to be used, from the
original offsets sum.

One offset is then changed on line 3218.

On line 3423 we push out the changed offset sum as 16 bits, but did not
check if the changed (3218) offset values would fit.

There is this comment nearby in the code

        /* Offset to lookup data which is in the temp file */
        /* we keep adjusting offset so it reflects the distance between */
        /* here and the place where the temp file will start, and then */
        /* we need to skip l->offset bytes in the temp file */
        /* If it's a big GPOS/SUB table we may also need some extension */
        /*  pointers, but FigureExtension will adjust for that */

[note]
This fix has been triggered by
ryanoasis/nerd-fonts#694

Reported-by: Gulajava Ministudio <GulajavaMinistudio>
Reported-by: Rui Ming (Max) Xiong <xsrvmy>
Signed-off-by: Fini Jastrow <ulf.fini.jastrow@desy.de>
Omnikron13 pushed a commit to Omnikron13/fontforge that referenced this issue May 31, 2022
[why]
Sometimes, especially when generating fonts with a lot ligatures,
fontforge can not create the font file correctly. A typical error
message looks like this:

  Attempt to output 68142 into a 16-bit field.
  It will be truncated and the file may not be useful.

This will already happen when the font is opened and generated without
any change. But the generated font must contain the lookups of course.

[how]
When the lookup tables are written we need to check if the indices can
be represented by 16 bit values. Big tables will often be longer.

In the file first is comes the lookups table that points to all lookups.
Then follow the lookups themselves.

When a lookup would violate the 16 bit boundary we need a different and
longer (!) entry in the lookups table, so all lookups will be shifted.
For this reason the function
  tottfgpos.c:g___FigureExtensionSubTables()
does this calculation recursively. Increasing the size of the lookups
table step by step for each lookup, and then checking the previous
lookups again (because they shifted).

Unfortunately the check if the 16 bit boundary would be violated does
not take the grown lookups table into account. Under some circumstances
this means that a lookup, that fittet just-so before is now shifted
because of the bigger lookups table and will then be too far away (i.e.
index > 16 bit).

In the calculation the `len` (size increase) of the lookups table needs
to be included.

[note]
In the `fontforge/tottfgsub.c`:

        g___FigureExtensionSubTables()

        3179: if ( sub->subtable_offset+offset>65535 )
        3218: sub->subtable_offset += len;

and later:

        dumpg___info()

        3410: efile=g___FigureExtensionSubTables(all,offset,is_gpos);
        3423: putshort(g___,offset+sub->subtable_offset);

On line 3179 is decided if a figure-extension has to be used, from the
original offsets sum.

One offset is then changed on line 3218.

On line 3423 we push out the changed offset sum as 16 bits, but did not
check if the changed (3218) offset values would fit.

There is this comment nearby in the code

        /* Offset to lookup data which is in the temp file */
        /* we keep adjusting offset so it reflects the distance between */
        /* here and the place where the temp file will start, and then */
        /* we need to skip l->offset bytes in the temp file */
        /* If it's a big GPOS/SUB table we may also need some extension */
        /*  pointers, but FigureExtension will adjust for that */

[note]
This fix has been triggered by
ryanoasis/nerd-fonts#694

Reported-by: Gulajava Ministudio <GulajavaMinistudio>
Reported-by: Rui Ming (Max) Xiong <xsrvmy>
Signed-off-by: Fini Jastrow <ulf.fini.jastrow@desy.de>
@Finii Finii mentioned this issue Aug 19, 2022
2 tasks
@Finii
Copy link
Collaborator Author

Finii commented Aug 19, 2022

Fixed via #884

@github-actions
Copy link
Contributor

This issue has been automatically locked since there has not been any recent activity (i.e. last half year) after it was closed. It helps our maintainers focus on the active issues. If you have found a problem that seems similar, please open a new issue, complete the issue template with all the details necessary to reproduce, and mention this issue as reference.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 12, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants