Letter spacing/kerning/padding issues with Lato

[chancancode]

Cross-posting from actions/runner-images#385.

We have a cron job that builds the Ember tutorial daily (source here), running on GitHub Actions. The idea is that the build will follow the steps in the tutorial the same way an end-user would (generate an app, editing the generated files, etc) and capture the command line output along the way, as well as taking screenshots of the app being built using puppeteer, which uses Chromium under-the-hood.

Since sometime last week (last good build, first bad build), things started to render differently in puppeteer in a subtle but noticeable way. You can see for yourself here, using the "Swipe" or "Onion Skin" mode to review the differences.

TL;DR fonts a rendering differently (different letter spacing, kerning, padding?):

My first thought is that perhaps this is due to a Chrome upgrade on the virtual environment. However, I don't think that's the case. We install puppeteer via yarn with a lockfile. Puppeteer (unlike puppeteer-core) bundles its own version of Chromium, so I don't think we are using the system Chrome at all. And the lockfile in yarn should ensure we are getting the same version of puppeteer (and therefore Chromium) across the build.

Over the weekend, a colleague mentioned that a client had a similar issue last week as well. They were also using the Lato font served by Google fonts, and other system/self-hosted fonts did not have the same issue. Their setup is similar to ours, but using the Percy visual review service instead of a custom puppeteer setup.

I noticed that there were some commits on this repo around that time last week. However, the commits do not seem to be modifying things related to the Lato font, at least not directly. Is it possible that those commits triggered a new build/release/deploy (if that's a thing), and somehow caused a regression with unrelated fonts? (#2328 seems to suggest that something like this could happen?)

[chancancode]

Hi @laerm0! Thanks for taking a look at the issue. Is there anything I can do to help confirm the issue? I would love to reproduce this actually, but I'm not sure if there is a a way to access "the fonts from last month (or from before the current deploy)". Are the old artifacts cached/saved somewhere that I can access?

[laerm0]

Well, Lato hasn't been updated on the repo for 3 years, so I'm definitely thinking it's not an issue on the Google Fonts side. And as for the software you're using, I have no experience with it or access to it, so I'm afraid you're on your own there… Sorry I can't be of further help.

[chancancode]

Right, I saw that the lato font in the repository hasn’t been updated, but is it possible that a different change in the repository would trigger a new “build” for the whole repository?

I was trying to understand how the build/deployment works, but as far as I can tell it’s not documented and probably managed internally inside google.

I know it’s a long shot, but the reason I suspect this could be the case is 1) another completely related project had the same problem with the same google font at around the same time 2) Fira Code was also not updated recently, but there was a new issue with that font in a similar timeframe in #2328, and the issue was caused/fixed by (what appears to be) some kind of build pipeline issues.

[chancancode]

As far as the software I was using that caused me to notice the problem, it basically amounts to opening the browser, navigating to a page that uses the Lato font via google fonts, then taking a screenshot of the result. The difference is pretty subtle, so probably wouldn’t have been noticed or reported by humans, but something definitely changed.

[noinkling]

I'm doing some visual regression testing too (on a personal project with a more ad-hoc setup), and have also noticed some differences.

Screenshots and diff

Top screenshot was taken on Chrome 77 (in August last year), bottom was taken today on Chrome 80 (I don't run these tests regularly so any change may not have been recent). Any pixels that have changed are represented as red in the middle.

So, the text is obviously a little taller (1 pixel), throwing off the shape of the letters (look at the difference in that "g" especially). Also the button has lost a pixel of width. It seems very possible in my case that this could just be an (undesirable) result of rendering differences between Chrome versions.

However there's also the fact that the last "t" in the blue text now links up with the "i". font-variant-ligatures is the default (normal) in the latter screenshot, and presumably the same in the earlier one - I now need to change it to none to remove the link again. I'm not sure if this is a font change or a Chrome change.

I notice there is also a case of this in @chancancode's second image: have a look at the "ti" in "assertions" vs the one in "visiting".

This is the CSS import statement I'm using (via a library), which hasn't changed:

@import url('https://fonts.googleapis.com/css?family=Lato:400,700,400italic,700italic&subset=latin&display=swap');

Hopefully this is somewhat helpful information.

[serg-admin]

I can confirm. We have issue with spasing Lato font too. If poll *.ttf files from google/fonts directly issue dissapear. Version from archive have wider space than google version.

[chancancode]

@serg-admin Interesting! I am still quite interested in figuring this out.

Can you elaborate on “poll from google/fonts directly” vs “versions from archive”? Do you have both versions saved somewhere for comparison?

For what it’s worth, this definitely still switched back and forth a few more times since I originally opened the ticket.

[vlad-roid]

@chancancode have a look, I've attached a ZIP with one Lato-Regular TTF served from the github archive here, and another served from https://www.googleapis.com/webfonts/v1. They're basically completely different, just look at this comparison when opening both in Windows Font Viewer (left is github, right is webfonts):

Well, Lato hasn't been updated on the repo for 3 years, so I'm definitely thinking it's not an issue on the Google Fonts side.

@laerm0 that can't be right. Please compare and find and replace the one that's outdated.

[chancancode]

The kind of difference your screenshot shows would likely explain the kind of shifts I was seeing in my original post. The direction of the shifts seems to line up.

I also see that the metadata said one of the is 1.x and the other is 2.x. Seems like the one on the left is supposed to be served but the right is currently on the CDN (until it changes back again - perhaps it even depends on the CDN node you ended up on?)

If we can find the issue that causes the 1.x version to be propagated to the CDN perhaps the mystery can finally be solved.

@laerm0 sorry for the ping, but since this issue is so difficult to pin down and now we have an active lead, would you be able to investigate a bit on your end?

[chancancode]

Pinging @rsheeter as well as they investigated and fixed a similar issue in #2328

[chancancode]

Pining @davelab6 @m4rc1e to give this a little more visibility as they seem to have knowledge in this area as well

[kainosnoema]

We ran into this as well, discovered the hard way that Google intentionally rolled back the v1 -> v2 upgrade a long time ago (mentioned here) and they're still serving v1 to this day. We had to download v2 from this repo and create our own web fonts from them....