Improve fonts generation: use fonttools to generate (reproducible) fonts #54

ylemkimon · 2018-07-24T17:52:41Z

This PR is part of #12.

Reproducible build: Fixes Use FontForge's SOURCE_DATE_EPOCH for repeatable builds #15
- Set created/modified timestamps to now or SOURCE_DATE_EPOCH, which is set to the last commit date of src/fonts in buildFonts.sh
- Remove FontForge timestamp(FFTM) table, Please do not put timestamps in saved (SFD) files fontforge/fontforge#2711
- Set font version from the version of package.json
Calculate ascent/descent using fonttools, remove fix_os2_metrics.rb and Ruby
Fix OS/2 and hhea metrics using fonttools, remove modify_ascent_descent.py
Generate WOFF and WOFF2 using fonttools, remove woff-tools and woff2_compress, combine targets into phony fonts target
Update copyright year

I compared the output using ttx and differences other than above were maxInstructionDefs in head, which probably was a bug in previous fontforge and few metrics, resulting in (sub)pixel changes but nothing visible.

* set created/modified timestamp * remove fontforge timestamp(FFTM) and GDEF table * set font version * fix metrics * generate WOFF, WOFF2

…nttools

kevinbarabash

I'm super excited about this change. I think it's going to unlock a lot of doors for us. Requesting a way to avoid updating font files with no changes to the glyphs themselves.

kevinbarabash · 2018-09-03T22:57:27Z

buildFonts.sh

@@ -72,10 +73,11 @@ if [[ $(docker images "$IMAGE" | wc -l) -lt 2 ]]; then
 fi

 CMDS="set -ex
+export SOURCE_DATE_EPOCH=${LAST_COMMIT_DATE}


Won't this still result in all files being modified (albeit only on commit) whenever someone makes any change? It would be nice if we only updated font files where we changed something. I think there might be some way to do that while also bumping the timestamp in those files, but it might be easier to just set SOURCE_DATE_EPOCH to be a constant.

src/fonts/OTF/TeX/Makefile

kevinbarabash · 2018-09-03T23:02:13Z

src/fonts/OTF/TeX/generate_fonts.py

+    version = json.load(f)['version']
+
+font['name'].setName(unicode('Version ' + version), 5, 3, 1, 1033)
+font['name'].setName(unicode('Version ' + version), 5, 1, 0, 0)


What's 5, 3, 1, 1033 and 5, 1, 0, 0 about?

@kevinbarabash They are name ID, platform ID, encoding ID, and language ID of NAME table, respectively. We have two tables, (3, 1, 1033) and (1, 0, 0), which are Unicode variants. (See Space.ttx)

kevinbarabash · 2018-09-03T23:05:58Z

src/fonts/OTF/TeX/generate_fonts.py

+font_file = sys.argv[1]
+font_name = os.path.splitext(os.path.basename(font_file))[0]
+
+font = TTFont(font_file, recalcBBoxes=False, recalcTimestamp=False)


Thanks for figuring out all of these TTFont options to produce a reproducible build.

src/fonts/OTF/TeX/generate_fonts.py

kevinbarabash · 2018-09-03T23:14:06Z

src/fonts/OTF/TeX/Makefile

-	mkdir -p ttf
-	rm -f ttf/*
+.PHONY: fonts
+fonts: ff


I wonder if we could rewrite this rule to be a per .ff rule since makeFF outputs multiple files to ff/KaTeX_$font.ff. This would allow us to only update those .ttf, .woff, and .woff2 files that have meaningful changes.

We'd have to check in the .ff files themselves which isn't the worst. Long term we might look at splitting up makeFF. Another short term approach we might look at is setting the default timestamp to be whatever it was in the file from the last commit... write the file, if there are changes, update the timestamp to be the current timestamp and update the file with that timestamp.

kevinbarabash · 2018-09-03T23:15:09Z

src/fonts/OTF/TeX/generate_fonts.py

+
+# save WOFF2
+font.flavor = 'woff2'
+font.save(os.path.join('woff2', font_name + '.woff2'), reorderTables=None)


I'm glad that FontTools makes it easy to save all these different formats.

ylemkimon · 2018-09-04T01:56:54Z

@kevinbarabash I was thinking of moving MakeFF and possibly Makefile to JS, and this PR is temporary (intermediate) step. Do you think we should leave Makefile and use make?

kevinbarabash

@ylemkimon I'd like to move things to JS as well. Since this is an intermediary step we can leave deciding whether or not to update individual font files to a future PR.

kevinbarabash · 2018-09-04T02:03:22Z

We should probably merge #50 first.

ylemkimon · 2018-09-04T02:05:15Z

@kevinbarabash ~~As circleci/test is not required check yet, I think we can merge this first.~~

kevinbarabash · 2018-09-04T03:30:23Z

Yay!

ylemkimon added 3 commits July 24, 2018 23:19

Upgrade fonttools to 3.28.0

c2377ad

Use fonttools to...

2ecbe23

* set created/modified timestamp * remove fontforge timestamp(FFTM) and GDEF table * set font version * fix metrics * generate WOFF, WOFF2

Add Space.otf to .gitignore

176bf6f

ylemkimon mentioned this pull request Jul 24, 2018

Improve fonts generation: move ascent/descent calculation into python #52

Closed

ylemkimon changed the title ~~Use fonttools to generate (reproducible) fonts~~ Improve fonts generation: use fonttools to generate (reproducible) fonts Jul 24, 2018

ylemkimon added this to In progress in Improve fonts generation via automation Jul 24, 2018

ylemkimon added a commit to ylemkimon/katex-fonts that referenced this pull request Jul 24, 2018

Build docker image for KaTeX#54

e4d5dcd

Merge branch 'master' of https://github.com/KaTeX/katex-fonts into fo…

926fd99

…nttools

ylemkimon force-pushed the fonttools branch from 87bc967 to 926fd99 Compare July 24, 2018 18:13

Build fonts

cbb5599

ylemkimon mentioned this pull request Jul 24, 2018

Test KaTeX/katex-fonts#48,49,54 KaTeX/KaTeX#1494

Closed

ylemkimon added the GH Review: need review label Aug 5, 2018

ylemkimon mentioned this pull request Aug 27, 2018

Improve fonts generation: remove unused codes #48

Merged

Merge branch 'master' into fonttools

7dae414

ylemkimon mentioned this pull request Sep 3, 2018

Fix typo: "Caligraphic" -> "Calligraphic". #55

Closed

kevinbarabash suggested changes Sep 3, 2018

View reviewed changes

kevinbarabash added GH Review: needs revision and removed GH Review: need review labels Sep 3, 2018

kevinbarabash assigned ylemkimon Sep 3, 2018

kevinbarabash mentioned this pull request Sep 3, 2018

Improve fonts generation: remove Math-Regular font #49

Closed

Remove unused tools

2546ef0

kevinbarabash approved these changes Sep 4, 2018

View reviewed changes

kevinbarabash added blocked GH Review: accepted and removed GH Review: needs revision labels Sep 4, 2018

ylemkimon added 2 commits September 4, 2018 11:07

Merge branch 'master' into fonttools

d882900

Update CircleCI config

e26dd21

Merge branch 'master' into fonttools

33d0c35

kevinbarabash added GH Review: needs revision and removed GH Review: accepted blocked labels Sep 4, 2018

Update Docker image

2e2ecf7

ylemkimon force-pushed the fonttools branch from dca4c20 to 2e2ecf7 Compare September 4, 2018 02:50

ylemkimon added 2 commits September 4, 2018 11:51

Merge branch 'master' into fonttools

592fac3

Merge branch 'master' into fonttools

44133db

kevinbarabash merged commit 93cbc5f into KaTeX:master Sep 4, 2018

Improve fonts generation automation moved this from In progress to Done Sep 4, 2018

kevinbarabash added GH Review: accepted and removed GH Review: needs revision labels Sep 4, 2018

kevinbarabash mentioned this pull request Dec 5, 2019

Merging katex-fonts into the main KaTeX repo #72

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve fonts generation: use fonttools to generate (reproducible) fonts #54

Improve fonts generation: use fonttools to generate (reproducible) fonts #54

ylemkimon commented Jul 24, 2018 •

edited

Loading

kevinbarabash left a comment •

edited

Loading

kevinbarabash Sep 3, 2018

kevinbarabash Sep 3, 2018

ylemkimon Sep 4, 2018

kevinbarabash Sep 3, 2018

kevinbarabash Sep 3, 2018

kevinbarabash Sep 3, 2018

kevinbarabash Sep 3, 2018

ylemkimon commented Sep 4, 2018

kevinbarabash left a comment

kevinbarabash commented Sep 4, 2018 •

edited

Loading

ylemkimon commented Sep 4, 2018 •

edited

Loading

kevinbarabash commented Sep 4, 2018

Improve fonts generation: use fonttools to generate (reproducible) fonts #54

Improve fonts generation: use fonttools to generate (reproducible) fonts #54

Conversation

ylemkimon commented Jul 24, 2018 • edited Loading

kevinbarabash left a comment • edited Loading

Choose a reason for hiding this comment

kevinbarabash Sep 3, 2018

Choose a reason for hiding this comment

kevinbarabash Sep 3, 2018

Choose a reason for hiding this comment

ylemkimon Sep 4, 2018

Choose a reason for hiding this comment

kevinbarabash Sep 3, 2018

Choose a reason for hiding this comment

kevinbarabash Sep 3, 2018

Choose a reason for hiding this comment

kevinbarabash Sep 3, 2018

Choose a reason for hiding this comment

kevinbarabash Sep 3, 2018

Choose a reason for hiding this comment

ylemkimon commented Sep 4, 2018

kevinbarabash left a comment

Choose a reason for hiding this comment

kevinbarabash commented Sep 4, 2018 • edited Loading

ylemkimon commented Sep 4, 2018 • edited Loading

kevinbarabash commented Sep 4, 2018

ylemkimon commented Jul 24, 2018 •

edited

Loading

kevinbarabash left a comment •

edited

Loading

kevinbarabash commented Sep 4, 2018 •

edited

Loading

ylemkimon commented Sep 4, 2018 •

edited

Loading