Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v3.000; 706b2b2-release not working properly in intellij on Windows #345

Closed
jorgheymans opened this issue Oct 24, 2017 · 53 comments
Closed
Assignees

Comments

@jorgheymans
Copy link
Contributor

jorgheymans commented Oct 24, 2017

I installed the latest release using the installer and restarted my machine. In intellij i am now getting garbled text:

hack3-intellij

I reverted by shift-deleting the font from the windows font dir, then copying the ttf files of the 2.020 release into the windows font dir and restarting intellij.

Interesting perhaps is that in the windows font dir, the 3.0 font shows up like this:

image

whereas the 2.020 shows up like this:

image

This is on windows 10, latest intellij using jdk8.

@chrissimpkins chrissimpkins self-assigned this Oct 24, 2017
@chrissimpkins chrissimpkins changed the title 3.0 not working properly in intellij 3.0 not working properly in intellij on Windows Oct 24, 2017
@chrissimpkins
Copy link
Member

Thanks Jorg. I can confirm that the render is OK on OS X with CLion and PyCharm using the following configurations:

PyCharm 2017.2.3
Build #PY-172.3968.37, built on September 1, 2017
JRE: 1.8.0_152-release-915-b11 x86_64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Mac OS X 10.12.6
CLion 2017.2.2
Build #CL-172.3968.17, built on August 22, 2017
JRE: 1.8.0_152-release-915-b11 x86_64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Mac OS X 10.12.6

Looks like this is going to be a Windows x Java renderer issue. Available to help troubleshoot the Java end of things?

@jorgheymans
Copy link
Contributor Author

image

sure how do you suggest troubleshooting ? With the previous release the issue got fixed eventually in the jdk, looks like now it'll have to be fixed in the font.

@chrissimpkins
Copy link
Member

chrissimpkins commented Oct 24, 2017

@jorgheymans To confirm you are using the latest release of the Windows installer for Hack?

@texhex are you seeing the same display of incorrect glyphs on the icons for Hack Italic/Bold Italic/Bold on Windows? (See OP above)

@jorgheymans As a quick test, can I ask you to switch to a syntax highlighter that DOES NOT render italics? Please let me know if this fixes the issue. It looks like at least a subset of the problems are occurring in text that the syntax highlighter renders in italic.

After the above, can I ask you to provide the following for one or two code points where you see these errors:

  1. expected Unicode code point (or feel free to copy/paste glyph in Markdown)
  2. displayed Unicode code point (or feel free to copy/paste what is rendered in Markdown)
  3. Hack variant set that your syntax highlighter is using to render the incorrect Unicode code point

Not clear what the issue is but this will provide some data to get started looking into it.

Thanks!

@texhex
Copy link
Member

texhex commented Oct 24, 2017

I think if both IntelliJ and Hack had an Facebook account, their relationship status would be "It's complicated".

I do not see the rather strange glyphs on either my two machines in the "Fonts" applet (Windows 10 1703 and 1709). That's very odd.

@jorgheymans Could you maybe try the Windows installer a test run, just to be sure that is not again some Windows internal caching issue? Or did you tried that already?

@zhimoe
Copy link

zhimoe commented Oct 24, 2017

I think it is just font cache, if u delete font from Windows font folder, then you may encounter this, just use everything software search Hack u may see some Hack cache font, delete all of ttf start with Hack then reinstall font.
sometimes rebuild font cache can help but sometime may not.
it is a bug with Windows and jetbrain IDE, not font.

hope can help, I taste font a lot so encountered this too many times

@jorgheymans
Copy link
Contributor Author

@texhex I did use the latest installer the first time yes, but subsequent tries were done by just deleting the font from the windows font dir.

@jorgheymans
Copy link
Contributor Author

@chrissimpkins indeed non-italics render fine, it's just an observation though because I cannot control the use of italics in the syntax highlighter. An example is a "javadoc" comment which is always rendered italic. It starts with /** and ends with */ . Using hack, the / is rendered as ?, the asterisk is rendered as what looks to be a superscript 4.

image

@jorgheymans
Copy link
Contributor Author

Also, i can reproduce it using this tool https://github.com/fruel/JavaFontRendererTest , it's probably #152 all over again.

@chrissimpkins
Copy link
Member

the / is rendered as ?, the asterisk is rendered as what looks to be a superscript 4.

Great, this is helpful. Thanks. I have a hunch that I will investigate. Will let you know when I have more information.

@chrissimpkins chrissimpkins changed the title 3.0 not working properly in intellij on Windows v3.000; 706b2b2-release not working properly in intellij on Windows Oct 25, 2017
@chrissimpkins
Copy link
Member

To add to the OS X specific information here, I can verify that regular + italic variants render without issues using v3.000; 706b2b2-release builds (current v3.000 tagged release) following font cache clear + reboot on OS X with the following PyCharm font settings (with Java dependencies in #345 (comment))

screen1

screen2

@chrissimpkins
Copy link
Member

chrissimpkins commented Oct 28, 2017

@jorgheymans can I ask you to try these glyphs to see how they render without any other glyphs (i.e. test them individually):

  • type lower case a and confirm that it renders as lower case a when converted to italics with the syntax highlighter
  • type upper case C and confirm that it renders as upper case C when converted to italics
  • type zero 0 and confirm that it renders as zero 0 when converted to italics
  • type period . and see if it renders as a colon : when converted to italics

then try the following

  • type period . followed by any string of glyphs and see if all glyphs following the period render as inappropriate glyphs when rendered in italic

It is strange. My initial hunch was that the Java renderer may be using the glyph order to make the transition between regular Unicode code points and italic code points on Windows and there seems to be an association. There is a pattern to the two glyphs that you pointed out in #345 (comment). The glyphs at those points are in a different glyph order in the regular and italic sets. The glyph rendered in the italic set is not the same order position (where it would still be incorrectly displayed relative to the glyph that you want) but rather lies at glyph position + 1 for both of them. For the initial animation that you displayed, the order differences between the two sets begins at the period used to identify the method name and I am wondering if that may be throwing the remainder of the string rendering for the line off for some reason. I haven't looked in detail at the upper and lower case letters for that method name to confirm the pattern yet. I don't yet know the relationship to the incorrect text displayed in the popups.

Let's see what happens when you try the above glyphs. If it confirms the pattern we can look into the glyph order issue in the source.

@chrissimpkins
Copy link
Member

@anthrotype @moyogo does this project https://github.com/unified-font-object/ufoNormalizer automate the modification of glyph order to maintain consistency in UFO source across all font variants?

@moyogo
Copy link

moyogo commented Oct 28, 2017

No, the ufoNormalizer doesn't change a UFO glyph order.

@chrissimpkins
Copy link
Member

@moyogo thanks Denis. Is there a way to create a consistent glyph order in any of the font editors?

@anthrotype
Copy link

yes, if you use the "public.glyphOrder" key in the UFO lib.plist
http://unifiedfontobject.org/versions/ufo3/lib.plist/

@chrissimpkins
Copy link
Member

@anthrotype thanks Cosimo. In Glyphs do you need to manually reorder or is it automated when you save the full UFO? I am guessing the former since these UFO were initially saved during the conversion from sfd with Glyphs.

@anthrotype
Copy link

There should be a glyphOrder custom parameter

@chrissimpkins
Copy link
Member

@anthrotype 👍 ty! will look into it

@chrissimpkins
Copy link
Member

@anthrotype @moyogo have either of you ever come across an issue like in the OP? Bizarre and only found on Windows with the Java renderer.

@anthrotype
Copy link

No, never. It’s weird

@chrissimpkins
Copy link
Member

chrissimpkins commented Oct 28, 2017

@anthrotype OK, I feel less inadequate at least :)

Clearly, I blame the compiler ;)

@chrissimpkins
Copy link
Member

@xdahu can you verify this claim:

it is a bug with Windows and jetbrain IDE, not font.

Is there any information out there re: this problem or can you confirm that you experienced it as well and that there is workaround to fix it? We had a similar issue that developed with our former build process and it was resolved on the Java side. It seems we caused it to raise its ugly head again with this transition in source + build.

@jorgheymans
Copy link
Contributor Author

jorgheymans commented Oct 28, 2017 via email

@chrissimpkins
Copy link
Member

chrissimpkins commented Oct 28, 2017

@jorgheymans interesting. If you can find the path to the cached fonts in the editor (if JetBrains uses caching), you can check the versions with this tool:

https://github.com/source-foundry/font-v

$ font-v report [font path]

That will give you a report of our version string. You should see the following for the current release of Hack:

Version 3.000; 706b2b23b-release; ttfautohint (v1.6) -l 6 -r 50 -G 200 -x 10 -H 181 -D latn -f latn -m "Hack-Regular-TA.txt" -R "Hack-Regular.ttf" -Z 0 -w G -W -t -X ""

Unfortunately, I don't know about the font management in their editors...

Maybe you could confirm your system install of the appropriate Hack version (and importantly that there are no duplicates, this is the point that @texhex and @xdahu were making, and is common on Win) and then find an approach to clear the JetBrains font cache?

@chrissimpkins
Copy link
Member

Let us know if you find an approach that eliminates the problem and we will update our docs. Confirming also that you had a previous version of Hack installed on your home system, installed the current release with the Win installer, and are not seeing the problem? Also confirming that you are using the same syntax highlighter in your home install of IntelliJ?

@haktrik
Copy link

haktrik commented Oct 29, 2017

Same problem I've with Rider of Jetbrain.
Three rows:
0123456789
abcdefghij
ABCDEFGHIJ
image
It's look somehow offset characters.

I'm using Windows 10
image
And Rider:
image
I have now installed build v3.001, but with v.3000 I got same issue.

@jorgheymans
Copy link
Contributor Author

jorgheymans commented Oct 30, 2017

@chrissimpkins

type lower case a and confirm that it renders as lower case a when converted to italics with the syntax highlighter

yes it does

type upper case C and confirm that it renders as upper case C when converted to italics

yes it does

type zero 0 and confirm that it renders as zero 0 when converted to italics

no, it comes out as a greek char (see below)

type period . and see if it renders as a colon : when converted to italics

no, it comes out as comma

type period . followed by any string of glyphs and see if all glyphs following the period render as inappropriate glyphs when rendered in italic

some characters render inappropriate, some are fine

Here is the 'conversion' chart in case it helps:

image

@jorgheymans
Copy link
Contributor Author

added U+0114 and U+0115, all sets Ĕ ĕ

these come out fine

@jorgheymans
Copy link
Contributor Author

Maybe you could confirm your system install of the appropriate Hack version (and importantly that there are no duplicates, this is the point that @texhex and @xdahu were making, and is common on Win) and then find an approach to clear the JetBrains font cache?

The only way i'm checking that there are no duplicates is by looking in C:\Windows\Fonts , and there i'm only ever seeing 1 hack font, with each time 4 files in it.

@jorgheymans
Copy link
Contributor Author

jorgheymans commented Oct 30, 2017

i found the issue :/ When i looked at c:\windows\fonts using total commander i saw this:

image

I have no idea how these old "linegap25" variants got there, perhaps from running some tests lasts year. Anyway I removed them and now all is fine 🥇

@haktrik have a look if your issue is similar

(@texhex should the installer check for this ? Hack-* should really only return 4 files ...)

@chrissimpkins
Copy link
Member

@jorgheymans good news!

Let's see what @texhex has to say about the Win installer. Tex, are you reading file paths or OpenType table names (or hash digests or some other approach) to determine duplicates?

@haktrik let us know if this fixes your issue as well.

All, please let me know (or PR in) how we should update the documentation for those who decide to install and upgrade manually on Windows (despite our suggestion to use Win installer) in order to address this issue.

@texhex
Copy link
Member

texhex commented Oct 30, 2017

@jorgheymans @chrissimpkins And just when I thought I know all possibilities how Windows can screw up font registration...

We already make sure that the TTF file names in \Fonts (Directory) Hash-match the files we want to install and that the registry information points to the correct files. Besides deleting the _0, _1 TTF files, we do not touch any "unexpected" files. However, if I know what other TTF file names were in used during the Hack development, I can kill them during install.

The question would be how the registry path HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Fonts looked for Hack prior to this error? Do you see any more entries that start with "Hack" that the ones that should be there?

As I understood it, this is the only path Windows checks to find out which fonts are registered. Were those "-linegap25" registered as "Hack" so Windows somehow mixed Hack normal and Hack linegap25 together?

@jorgheymans: Do you see any chance to try to reproduce this (of course not on your primary system)?

As an alternative, @chrissimpkins do you maybe still have those special TTF file so I could try to reproduce it here? When I understood what has happened there, I could maybe develop counter measures.

@chrissimpkins
Copy link
Member

chrissimpkins commented Oct 30, 2017

@texhex Those were created with our font-line tool. They have modified linespacing metrics compared with the release fonts, but have the same font names in the OpenType tables. They would be "seen" by the OS as duplicate fonts if inspected via OpenType table naming, but not when file paths are evaluated. I am guessing the OS generally uses the former.

I've attached new versions of the 25% UPM modified fonts (these are from v3.000 releases) to this post as zip and tar.gz archives. You should be able to replicate the issue with these.

linegap25.zip
linegap25.tar.gz

@chrissimpkins
Copy link
Member

Perhaps we should consider automatically changing the font family name in addition to the file path with that tool? In addition to this Windows x Java renderer issue, that would also address reserved font name licensing issues for users who modify fonts that include these. We could try that script that we PR'd into the FSCW repo Tex. This would provide a negative control for these files to confirm that the same file with modifications of the name table nameID 1,4,6 allow Windows to detect these as different fonts (this should be the case).

@jorgheymans
Copy link
Contributor Author

@texhex difficult to reproduce i'm afraid. Also that registry entry you mention does not exist on my workstation.

Actually, what made me look in the directory with total commander was the JavaFontRenderer tool, which displays at the bottom the physical path of the font file used.

image

@texhex
Copy link
Member

texhex commented Oct 31, 2017

Thanks @jorgheymans @chrissimpkins. I tried to reproduce it here but failed at failing the font registration.

When Hack 3.0 is normally installed and I then manually copy the linegap25.ttf to the Fonts applet, the only font registration that was damaged in the Registry was Hack (Regular). It looks like this after the installation:

hack-linegap-registry

Also, the Fonts applet show Hack normally, not with the broken glyphs as in the case of Jorg. So I can not fully reproduce the problem you had.

My best guess right now is that Java programs seem to check both the C:\Windows\Fonts folder AND the registry and try to make sense of that data. For all defects like this, the font display was fine in Windows programs (e.g. Notepad++) but broken in Java, which makes me think that this could be the case. But I have no evidence to prove this.

@jorgheymans As I understood it, you installed Hack 3.0 with the installer and after that the problem appeared. Could you maybe attach the file C:\Program Files\Hack Windows Installer\Log-FontData.txt so we can check what the registry said about your Hack file prior to installation?

@chrissimpkins I have opened a new issue in the Hack-Windows-Installer Repo. Could you please document there all TTF file names that were or are in use, even if only temporary? I would add them to our delete list so when the installer is run, they will get deleted and hopefully prevent these issues.

I also added issue 15 so the installer will check if a font registration for FONTNAME (Regular) exists and delete it on installation. This should prevent cases like the above where we do not register the font with (Regular), but Windows itself did so.

@chrissimpkins
Copy link
Member

Could you please document there all TTF file names that were or are in use, even if only temporary?

@texhex done!

@chrissimpkins
Copy link
Member

chrissimpkins commented Oct 31, 2017

Any utility in adding the ability to search for installed fonts that carry the same font family name string in their OpenType tables as part of the FSCW tool? Is there a consistent directory structure that can be searched for font installations across all users on Windows?

Could either provide a warning with the file path displayed to the user or could do what the operating system should be doing on new installs and clean these up for users.

@jorgheymans
Copy link
Contributor Author

jorgheymans commented Nov 1, 2017

@texhex here is the log

---BeforeInstallAction START---
--------------------------------
Font name.....: Hack fonts
Script version: 2.03
Setup version.: 1.4.1
Font version..: 3.000
Local time....: 2017-24-10 09:42
Fonts folder..: C:\windows\Fonts
Dest folder...: C:\Program Files\Hack Windows Installer
--------------------------------
---HASH CALCULATION---
Calculating hash for Hack-Bold.ttf
   File from setup: 2d5c7e8b0091cce6a9adf71f7c8992365d486f70
   File in \fonts : a7bb6faacd609145b55ed15ca238755544c03af5
Calculating hash for Hack-Regular.ttf
   File from setup: 5eceb4a3b9bc49d23da0499317f4bed5d5a0b4bb
   File in \fonts : 664cfe2a64de1486c0ace8073ceeb6d9281e8b78
Calculating hash for Hack-BoldItalic.ttf
   File from setup: c05822be3b154476970241d6bddc0fe1662639da
   File in \fonts : c428004a2fe3570450c6d03442052b1a9989c58b
Calculating hash for Hack-Italic.ttf
   File from setup: 0c5a8a3396eaa2c49e1c317b47afee5b08a5a27a
   File in \fonts : efdae4b94858b98eab6dcf2cb8e3cc3d28263cc2
----------------------
Checking for differences between fonts in setup and this system
IsSetupFontSameAsInstalledFont(): Hack-Bold.ttf
   Checking for font name in registry: Hack Bold (TrueType)
   Font name found
   Checking file name in registry. Expected: Hack-Bold.ttf
   File name matches
   Hash values (Setup/Windows): 2d5c7e8b0091cce6a9adf71f7c8992365d486f70 / a7bb6faacd609145b55ed15ca238755544c03af5
   File hash is different!
   Found difference for file Hack-Bold.ttf
   Installation required.
---BeforeInstallAction END---
IsSetupFontSameAsInstalledFont(): Hack-Bold.ttf
   Checking for font name in registry: Hack Bold (TrueType)
   Font name found
   Checking file name in registry. Expected: Hack-Bold.ttf
   File name matches
   Hash values (Setup/Windows): 2d5c7e8b0091cce6a9adf71f7c8992365d486f70 / a7bb6faacd609145b55ed15ca238755544c03af5
   File hash is different!
IsSetupFontSameAsInstalledFont(): Hack-Regular.ttf
   Checking for font name in registry: Hack (TrueType)
   Font name found
   Checking file name in registry. Expected: Hack-Regular.ttf
   File name matches
   Hash values (Setup/Windows): 5eceb4a3b9bc49d23da0499317f4bed5d5a0b4bb / 664cfe2a64de1486c0ace8073ceeb6d9281e8b78
   File hash is different!
IsSetupFontSameAsInstalledFont(): Hack-BoldItalic.ttf
   Checking for font name in registry: Hack Bold Italic (TrueType)
   Font name found
   Checking file name in registry. Expected: Hack-BoldItalic.ttf
   File name matches
   Hash values (Setup/Windows): c05822be3b154476970241d6bddc0fe1662639da / c428004a2fe3570450c6d03442052b1a9989c58b
   File hash is different!
IsSetupFontSameAsInstalledFont(): Hack-Italic.ttf
   Checking for font name in registry: Hack Italic (TrueType)
   Font name found
   Checking file name in registry. Expected: Hack-Italic.ttf
   File name matches
   Hash values (Setup/Windows): 0c5a8a3396eaa2c49e1c317b47afee5b08a5a27a / efdae4b94858b98eab6dcf2cb8e3cc3d28263cc2
   File hash is different!
IsSetupFontSameAsInstalledFont(): Hack-Bold.ttf
   Checking for font name in registry: Hack Bold (TrueType)
   Font name found
   Checking file name in registry. Expected: Hack-Bold.ttf
   File name matches
   Hash values (Setup/Windows): 2d5c7e8b0091cce6a9adf71f7c8992365d486f70 / a7bb6faacd609145b55ed15ca238755544c03af5
   File hash is different!
IsSetupFontSameAsInstalledFont(): Hack-Regular.ttf
   Checking for font name in registry: Hack (TrueType)
   Font name found
   Checking file name in registry. Expected: Hack-Regular.ttf
   File name matches
   Hash values (Setup/Windows): 5eceb4a3b9bc49d23da0499317f4bed5d5a0b4bb / 664cfe2a64de1486c0ace8073ceeb6d9281e8b78
   File hash is different!
IsSetupFontSameAsInstalledFont(): Hack-BoldItalic.ttf
   Checking for font name in registry: Hack Bold Italic (TrueType)
   Font name found
   Checking file name in registry. Expected: Hack-BoldItalic.ttf
   File name matches
   Hash values (Setup/Windows): c05822be3b154476970241d6bddc0fe1662639da / c428004a2fe3570450c6d03442052b1a9989c58b
   File hash is different!
IsSetupFontSameAsInstalledFont(): Hack-Italic.ttf
   Checking for font name in registry: Hack Italic (TrueType)
   Font name found
   Checking file name in registry. Expected: Hack-Italic.ttf
   File name matches
   Hash values (Setup/Windows): 0c5a8a3396eaa2c49e1c317b47afee5b08a5a27a / efdae4b94858b98eab6dcf2cb8e3cc3d28263cc2
   File hash is different!
  Changes detected, require reboot

@jorgheymans
Copy link
Contributor Author

jorgheymans commented Nov 1, 2017

@texhex about the font applet, it gives different results depending on how you invoke it. If you specify the location of the font on the command line you are guaranteed to get the correct font, however if you don't and then choose a font from the fontpicker dialog you get more into the windows font api (i think) and you should be able to reproduce it if you managed to create the correct buggy font listing in the windows/font directory.

@texhex
Copy link
Member

texhex commented Nov 1, 2017

@jorgheymans Thanks, this is interesting as NONE of the Hack font registration pointed to the -linegap files. This could be another indicator that Java is trying to pull the fonts directly from C:\windows\fonts. But the strange glyphs in the font applet do not make sense in this context. This is rather confusing.

All linegaps variants will now be killed upon installation, see the commit for details.

@texhex
Copy link
Member

texhex commented Nov 1, 2017

Hack Windows Installer 1.4.2 is out, this version should address all the issues that were described here.

@chrissimpkins
Copy link
Member

Should we update documentation here in any way due to this issue?

@chrissimpkins
Copy link
Member

@haktrik see #345 (comment)

@texhex
Copy link
Member

texhex commented Nov 2, 2017

@chrissimpkins Which issue of the many Jorg has discovered : )?

@chrissimpkins
Copy link
Member

@texhex ha! Are there any that warrant new docs to address for other users? @jorgheymans thoughts?

@jorgheymans
Copy link
Contributor Author

jorgheymans commented Nov 2, 2017 via email

@chrissimpkins
Copy link
Member

Sounds good. Thanks for reporting it Jorg. OK to close this?

@texhex
Copy link
Member

texhex commented Nov 2, 2017

Fine with me.

@chrissimpkins
Copy link
Member

Thanks all! Nice work @texhex!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

7 participants