Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

193/194 build, dev slide, DSIGInfo tool, and some 2015 spec updates for OS/2 and CBDT/CBLC #1

Closed
wants to merge 110 commits into from

Conversation

@HinTak
Copy link

@HinTak HinTak commented Nov 12, 2015

Known issues are:

  1. The DSIGInfo tool is not hooked up to be built by make yet - the build instruction is a long one-line at the top of the file itself. Also, none of README/dev slide mentions the DSIGInfo tool or related matter.
  2. The OS/2 2015 spec update requires further action of rebuilding the help file and the resource file OTFontFileVal/OTFontFileVal.ValStrings.resources . Until then, if you test a font with a OS/2 table below 5, it would correctly warn that the version is not up to date, but mistakenly say the latest is 3.
  3. The 5 commits at the top of HEAD, 3 of them bundled binaries for convenience, 1 for a small subset of Mono's code require for DSIGinfo (and a small modification of Mono's code on top from me), are not mine; but they do have a license compatible for re-distributing under the MIT license. Skip them if you are uncomfortable.

FontVal-RELEASE_2015_11_12 tarballs and binaries are finally without a password at
http://sourceforge.net/projects/hp-pxl-jetready/files/private-test-data/

HinTak added 30 commits Nov 12, 2015
bug in CmdLineInterface.cs in add-table mode
adding verbose inteface and hook up to -quiet option
adding IsTestingTable interface

use IsTestingTable to do better verbose
adding +raster-tests option

adding IsTestingRaster and using it
…port, and comments.

Comments about the CMD using in-memory report.

more annoyance about in-memory report, and comments

check-for-permissions-to-write-to-a-directory-or-file

better answer to test directory writability
runtime type check on Compat.OTFontFile.Rasterizer.FreeType.Library

adding check for SharpFont.Library
… chinese names

better work-around for Library/Fonts/AdobeKaitiStd-Regular.otf mac names

apparently 10003 = 20949 and 10008 = 20936?

the rest of substitutions from Encoding.WindowsCodePage property
Older Mac fonts have older names for the EBLC and EBDT tables.

Apple LiGothic Medium.ttf
Apple LiSung Light.ttf
AppleMyungjo.ttf
BiauKai.ttf
Cochin.ttc
Gungseouche.ttf
HeadlineA.ttf
Hei.ttf
Kai.ttf
NISC18030.ttf
OsakaMono.ttf
Osaka.ttf
PCmyoungjo.ttf
Pilgiche.ttf
Symbol.ttf

better fixes for old aliases
balld__.ttf
ballo__.ttf
bernf__.ttf
blipp__.ttf
bodbc__.ttf
btbd___.ttf
btlt___.ttf
btmd___.ttf
btul___.ttf
btxl__.ttf
croobie.ttf
eprg___.ttf
fat.ttf
flair__.ttf
gloo-gun.ttf
linea__.ttf
mickey.ttf
minnie.ttf
plbdc__.ttf
pl__x__.ttf
porkh.ttf
porky's.ttf
tcm____.ttf
walba__.ttf
CodePage-related source files from master

perl -pi -e 'if ($_ =~ /0x/) {$_ = uc $_; s(0X)(0x)g;};' Compat/cp*.cs

correct mistake in 5765acc4961ab616454c68715bdf373953696b65 for perl uppercase conversion

sync to win 8.1 cp

new from win 8.1 sync

adding headers and footers to code page 10001-10008

that comment is wrong

adding boilerplate to CPInfo.cs
adding boilerplate to Win32APIs.cs
Most of Freetype-based rasterer tests

do not re-define difference

adding boilerplate to Compat.cs
adding patch descriptions

about FREETYPE_PATCH

changing to FreeType

comments about Meaningless WSF shortcut bug.

adding resgen hidden dependency bug. comment

details about wine bug 708

mono codepage 10008 bug

Meaningless WSF shortcut bug filed

mono xsd bug id

resgen hidden dependency bug filed

more notes

adding notes on Extend ComputeMetrics

remove done codepage investigation

typo

CFF Incomplete notes

update README.txt for freetype info

Misc-updates to README's

updating README-extra.txt

README-hybrid.txt

more about the bybrid branch

more update about the bybrid branch

updating README.txt
…apple bloc/bdat issue

Part of the old commit.

commit 0902901c85e6fb9ef1ddcf39328943cde78dc8cb
Author: Hin-Tak Leung <htl10@users.sourceforge.net>
Date:   Wed Aug 26 09:44:09 2015 +0100

    Apple LiGothic Medium seems to have strange EBDT/EBLC tables - re-visit later
freetype-win64.patch

updating patch to freetype 2.6.1
…nt release is out.

0001-adding-ComputeMetrics.patch
dos2unix FontVal/FontVal.Form1.resx FontVal/FontVal.FormAbout.resx FontVal/FontVal.ResultsForm.resx Glyph/NS_Glyph.GErrStrings.resx
This shortcut does nothing and clutter up the GUI on Mono. Strangely
.NET is not affected. Filed bug at Mono bugzilla for "incompatible behavior".
4 new entries

regenerated OTFontFileVal/atoms.cs
new FTVersion property for the rasterer

put warning inside val_hdmx.cs

put version check inside LTSH

version check for VDMX

no need to write version and error to console anymore
…rrent test.

these two seem to be cut-and-paste typos

these seems to be typos
@HinTak
Copy link
Author

@HinTak HinTak commented Feb 24, 2016

I have been thinking about withdrawing the pull request for a while.

There are many ways Microsoft folks can contribute: of course they can put some dev resources and hours to it, or commission outsiders (not necessarily me, but that would be welcomed) doing some work; two unique ways they can contribute is (1) supplying a less-antique binary than 2003 - HinTak#5 - I know they updated the code as recent as after the 2009 2nd edition opentype spec, (2) help fix visual studio building so that more Window centric devs can contribute -HinTak#8 - . There are about 20 opened issues at the moment, some involves updating the build instruction docs, reading the specs and documenting what needs to be done . Many do not require a lot of special knowledge, just time.There are discussions going on in the Opentype mailing list they can participate, and there are past design decision they can clarify.

And while it isn't useful for practical purposes, providing binaries based on the stock august 11 upload, just to demonstrate that they can cope with using the open mono + GNU make build chain, instead of visual studio which they had been doing for 13 years, would be assuring too.

Hopefully we would see better participation from Microsoft folks.

There exists an implementation of the rasterisation test in my hard disk, and a few other bits on CFF also.

Loading

@HinTak HinTak closed this Feb 24, 2016
@HinTak
Copy link
Author

@HinTak HinTak commented Mar 7, 2016

This year's Google Summer of Code is starting to accept student application in a week or two. For those who are unfamiliar with it, GSoC is a scheme where Google sponsors about 1000 students around the world each year - for just over 10 years now - to work on a project of the student's choice, under the guidance of an expert in that area, for about 3 months in the summer. (I think the students get $5000 US, which is not bad for any summer job, especially when it doesn't even require leaving the house!)

The Linux Foundation has been a mentoring organization for almost every year, and is in this year. I mentored a few students under the open-printing initiative for a few years, as well as a couple of students working on various parts of the linux kernel in the past also.

The Microsoft Font Validator can go under Linux Foundation's openprinting. The Mono Project was a mentor organization in the past, so if they are in this year also, student can apply under the mono project too.
It is allowed to apply under two orgs, which can increase the student's chance of being accepted.

The student application windows period is 14 March to 25 March.

So if you work for an educational organization, or know students with suitable skills interested in that, please forward.

Loading

@davelab6
Copy link
Contributor

@davelab6 davelab6 commented Mar 7, 2016

Loading

@HinTak
Copy link
Author

@HinTak HinTak commented Apr 15, 2016

Re-reading some of the e-mails, I am reminded by @davelab6 , who is honest but kind in the best possible way, that maybe I have not explained and communicated well to @aaronbell why I really want a more "recent" internal binary, whichever 'more recent' means. If that's the case, apologies.

The issue I am facing from my experience with FreeType vs MS renderer on LTSH/hdmx/VDMX is that there are going to be differences. Some maybe my lack of understanding of freetype, (e.g. I was initially very excited by freetype giving perfect "hdmx" calculations, until I learned that freetype is optimized to just read the table and tell me, unless told not to, so of course the agreement was perfect!), bugs in freetype, bugs in my new code, and bug in the 2003 binary, bug in the specific font, or just "behavior differences". But I need to understand and account for all the differences until I am happy that all the differences are well-understood. If I get a 'close enough' answer, I can move to the next task. If they differ, I 'll dig.

Having a buggy 2003 binary actually means wasting me time debugging the 2003 binary to make sure that the difference is because I am correct with the new freetype code, and the old 2003 binary is wrong. So it takes extra time.

Besides causing me extra time, if microsoft is afraid that I might learn something I shouldn't, denying me a more recent and less buggy binary will result in exactly what they are afraid of: I may have to dig inside the proprietary 2003 binary to understand that it is buggy, and how buggy in exactly what way, after exhausting the other possibilities like modifying my own new code etc. So putting me in a scenario where I need to dig inside the proprietary 2003 binary to understand why and in what way it is buggy, may actually lead me to learn something I should not. If the new implementation agrees with the old, and the old is correct , I have no reason to dig. The worst is a 3-way disagreement: freetype said one thing, the old binary says another, and the built-in LTSH/hdmx/VDMX tables in the font says a 3rd answer (which should be 'correct' according to the last font editing apps that wrote them). There are indeed 3 way disagreements, even now, of those tables, on the common platform fonts. I keep notes about which of the three is correct, and try to tick the disagreements off one by one.

And I do not look forward to a 3-way disagreement in rasterisation, which is far more complex than calculating the tables.

So, I hope this is a better explanation why I really like a less antique binary for private reference.

Loading

@davelab6
Copy link
Contributor

@davelab6 davelab6 commented Apr 15, 2016

dig inside the proprietary 2003 binary

What does the binary EULA say about that?

Loading

@Pomax
Copy link

@Pomax Pomax commented Apr 15, 2016

presumably "reverse engineering is illegal", which is likely technically overruled by EU law that protects reverse engineering for public research purposes, but could equally well still lead to lawsuits.

Loading

@davelab6
Copy link
Contributor

@davelab6 davelab6 commented Apr 15, 2016

could equally well still lead to lawsuits

Indeed. Even if not, that kind of thing certainly makes rightsholders unhappy. Even just publicly discussing that kind of thing as a possibility is hostile.

Since Hin-Tak's goal is to be given something by a rightsholder, then doing things that make them less than joyful undermines the possibility the goal can be achieved at all.

Loading

@davelab6
Copy link
Contributor

@davelab6 davelab6 commented Apr 15, 2016

I hope this is a better explanation why I really like a less antique binary for private reference

It does explain why your life would be improved by MS sharing the binary, but it isn't clear how MS benefits. I can't see anything in your comment about the benefits to MS of releasing the binary. Maybe this is obvious to you and implicit, but it would help me to understand by stating it explicitly.

There is also a wider context to your request: Since you have - so far - not co-operated with MS, by not signing their CLA, then I am not sure anyone at MS is going to think too hard on any suggestion you make.

Eventually if you annoy anyone too much then they will just write off everything about you and not even reply to your messages any more; and if you keep pushing on them, then they might retaliate.

Please be respectful, offer value, and explain why what you are doing/offering is valuable to other people; and please do not threaten, and be willing to accept 'no' as an answer :)

Loading

@HinTak
Copy link
Author

@HinTak HinTak commented Apr 16, 2016

Hmm, okay, I thought the message was meant to make the situation better, not worse. Anyway, my goal is to make the new code covers no less than all the grounds that the old covers; at least judging from some of the reactions of the users over the last few months, 'why isn't X there?' , where X had been the XML report viewers a few times, and the rasterization test at least once as I can recall. (despite both being mentioned as 'not included' both initially and repeated a few times there after).

Perhaps the new code is too similar to old that users' expectation seems a bit high :-(.

Loading

@HinTak
Copy link
Author

@HinTak HinTak commented Apr 16, 2016

I would also like to say it again that, reading lengthy legal document is hard work; reading a lengthy legal document with sufficient attention to details to decide whether to agree with it or not, is even harder work. So being handed a piece of legal document to read, feels a lot like piling on more home work, and unnecessary one too, as a punishment for being hard-working.

Loading

@HinTak
Copy link
Author

@HinTak HinTak commented Apr 16, 2016

And I hope nobody is suggesting one should sign any legal documents without reading every word carefully, and thinking about it.

Loading

@HinTak
Copy link
Author

@HinTak HinTak commented Apr 18, 2016

@davelab6 , @Pomax : ask a Mozilla developer who has ever tried to figure out why the Adobe flash plug-in crashes firefox, for example. He or she might know a thing or two that Adobe does not want people know. The process is just called 'debugging proprietary plug-ins without the source'. It is not a pleasant process, and few people are motivated or have the skills to do it. If the proprietary plug-in is not buggy, there is no need to spend time on it. And as long as there are users who want to watch flash movies, and as long as the Adobe flash plug-in is the only plug-in to provide that functionality, there is a need to go in that direction. Please do not try to call the process a different name.

As for the CLA situation: 5 months ago ( #1 (comment) ) on 24th Nov, I already said, it is not important, and it is waiting for another interesting fork. If somebody comes up with interesting enhancements that I like to include, and he/she is unwilling to sign the CLA. That would polute my fork, then I have to forfeit the CLA too. If Microsoft comes up with interesting enhancement I like to include, and the CLA allows a merge, it would go in the other direction. Until another interesting enhancement happens outside of my fork that I need to merge, it is simply not important for me to read the CLA. And not reading the CLA means not deciding. If you want me to make a decision whether to sign the CLA or not, please come up with an interesting enhancement that I like to have. Keep on asking will not change the situation.

Loading

@davelab6
Copy link
Contributor

@davelab6 davelab6 commented Apr 18, 2016

it isn't clear how MS benefits

Please do answer this

I already said, it is not important

This is the problem, Hin-Tak. What is important is not for you to dictate, but emerges out of the shared values of the people involved. Aaron tried to see if the MS CLA could be avoided, and it could not. It is important to him that contributors sign the CLA, and your unwillingness to do so shows that you don't care about his needs; so it is no wonder that he appears uninterested in caring about your needs.

Loading

@HinTak
Copy link
Author

@HinTak HinTak commented Apr 18, 2016

Putting my enhancement out there is already a significant contribution. Extra work reading a lengthy legal document is extra work.

Microsoft folks can help a merge by adding enhancements that are worth merging with. Other contributors may have enhancements I like to incorporate with, and that are in conflict with a CLA. Until another interesting fork happens, both of these are still possible - @davelab6 : I am not dictating, I am leaving the options open. The possibility that others may contribute which are in conflict with my CLA, is slightly higher than Microsoft contributing to the future development, although both are small.

As @Pomax already responded, he cannot contribute while signing a CLA; and I cannot take on his possible change if I have a CLA in place; so this is leaving the option open for him to contribute: I am allowed to take up his change, if the change is interesting enough, and I do not need to worry about the joint branch poluting and violating my CLA.

Loading

@HinTak
Copy link
Author

@HinTak HinTak commented Apr 18, 2016

Please try not to use words like 'unwilling', 'dictate' or other emotionally charged labeling like 'threats' etc. Not deciding is simply that, not deciding, and leaving the options and possibilities open. I have been trying to explain how and when an decision can be made, and it is quite tying.

FWIW, I have been asking for code contributions from the community too, not just Microsoft; and trying to make it easier for people to contribute too, like reviving the support for building with the more common visual studio. I don't personally use MS VS, but there are people who use it daily, far more than mono.

So I am still waiting for an interesting enhancement outside of my fork to happen.

Loading

@davelab6
Copy link
Contributor

@davelab6 davelab6 commented Apr 18, 2016

Not deciding is simply that, not deciding

That's not how social reality works.

Loading

@Pomax
Copy link

@Pomax Pomax commented Apr 19, 2016

I certainly know how frustrating it can be to seemingly get ignored while having the best intentions, but one thing to remember is that Microsoft is not people. It's easy to forget when we're discussing on Github in a PR with comments by Microsoft employees, but @aaronbell, the awesome person, doesn't really get to speak as Microsoft, the company who owns the Font Validator code, when it comes the kind of legal issues associated with the process you're talking about, @HinTak.

It would be incredibly valuable if you could have a look at the newer code to see if oddities you're seeing in a three-way parsing difference can be disambiguated, but this is not something Microsoft, the company, can easily do. As a multi billion dollar company, even if Aaron wanted to help, unless he's a Microsoft VP, he probably cant: Microsoft is so large that it has to go with what its lawyers say they can do, because if they don't they can lose legal standing and could open themselves up to all kinds of IP losses. That's where good faith comes in.

When I represent myself, I will not sign a CLA, because I am not willing to operate on the legal level required for this project. In consequence, I made a choice to not contribute any code, and am unlikely to ever do so. However, if I were as driven as you were to fix up the Font Validator to conform to the modern spec, then I probably would sign that CLA because the world as a whole benefits from that far more than I am inconvenienced by it (I need to be pretty damn invested for that to happen, but it has happened before). Of course, when I do, I would also ask that my code be reviewed not just as code but also from a legal standpoint, because I do not have the resources to guarantee everything I wrote was IP unencumbered.

I would, in that position, also love to see Microsoft's newer code to solve the problem you're having, but asking for that, without showing good will first (companies are terrified of the pandora's box that gets opened when a traditionally closed source project goes open source) and making them feel like my contributions are valuable to their company. Not just the people in the team whose repository I'm contributing to, but the actual company itself. They need to see that the work I did, for free, for my benefit, benefits them, too, and companies don't value words much: a CLA, a signature on an NDA, an agreement of collaboration, etc. are the kind of things that open the doors behind which the people can be found that might help.

Is it frustrating that this is how things work? Absolutely. Should we change that? As a bit of an idealist yes, I think we should. Should we force that? As a realist, no. Making demands, forcing hands, without showing willingness to cooperate in a language that companies (not people) understand just gets great ideas shut down, so in this case I'd suggest a brief respite to gather some thoughts, see if for the amount of work you did you might be willing to sign the CLA, and allow me to make that more interesting for you: I would be happy to send another donation if it makes the difference between the development you've put in so far getting cut off over this, and showing Microsoft that not just you, by I as well, want to make an effort to meet them where they are, so that we can find a way forward that is beneficial to everyone, not just "everyone except Microsoft".

Loading

@HinTak
Copy link
Author

@HinTak HinTak commented Apr 19, 2016

That's not how social reality works. 

Not sure what you mean by that. I am, however, quite tired of being the only person making interesting enhancements on this project. If forfeiting the CLA is a direction to encourage contributions from people who are against it to contribute, that's an option and possibility I'd like to leave open.

Again, this is not a "threat". Signing the CLA to merge with a Microsoft enhancement, or merge with somebody else's contribution who is against the CLA, are the two future possibilities. Until another interesting fork happens, still there. Until then, there is no need to spend time reading a lengthy legal document to decide.

Loading

@HinTak
Copy link
Author

@HinTak HinTak commented May 7, 2016

Font Validator gains python scripting ability, and ttc-splitter/merger scripts.

Announcement on
http://typedrawers.com/discussion/1558/microsoft-font-validator-gains-python-scripting-ability-and-ttc-splitter-merger-scripts

binary:
[1] https://sourceforge.net/projects/hp-pxl-jetready/files/Microsoft Font Validator/FontVal-empy-bin-2016_05_07.zip/download

code:
[2] https://github.com/HinTak/Font-Validator/tree/embedding-ironpython

Loading

@HinTak
Copy link
Author

@HinTak HinTak commented Jul 11, 2016

I have re-implemented (part of) the last missing piece, the rasterization test, which tests for hinting problems. One announcement is up at -
http://typedrawers.com/discussion/1673/font-validator-with-rasterization-i-e-hinting-test-out-now

code and binaries at the usual places, and please feel free to click the donate button above the download links. I'll prepare a Mac OS X package in a couple of days.

Loading

@davelab6
Copy link
Contributor

@davelab6 davelab6 commented Jul 11, 2016

On 10 July 2016 at 23:58, HinTak notifications@github.com wrote:

I'll prepare a Mac OS X package in a couple of days.

Will you take in the commits that are on Georg Seifert's branch?

Loading

@HinTak
Copy link
Author

@HinTak HinTak commented Jul 11, 2016

I realized that the need for a more up-to-date binary is only to create quality analysis reports on fonts of my choice to look at (versus many bogus warnings/errors from the 2003 binary). If I cannot have quality, large quantity of interesting, if many bogus, reports would do too:

http://typedrawers.com/discussion/1658/old-ms-fontval-reports-to-spare-or-computer-time-to-make-new-ones

and even that isn't needed - to get interesting reports, it isn't necessary to analyze on all sizes - the interesting hinting errors are in the smaller sizes, and one (out of size 4 to 72 plus a few above) would do very well.
Thus I have cut possibly 3-4 weeks of test run time of all 2000 fonts on a typical linux system down to 7 hours. Am collecting reports on Apple shipped fonts now, and there are many more interesting reports to look at.

FYI, only about 2/3 of linux shipped fonts pass (expected to be a bit lower than microsoft's own, around 4/5). And quite interestingly, The proportion of Apple shipped fonts are lower.

The current list of interesting errors with accompanying interesting fonts to look at - i.e. potential hinting errors that could be detected - are up at ( HinTak#5 ).

Loading

@HinTak
Copy link
Author

@HinTak HinTak commented Jul 11, 2016

The process of implementing the rasterization test is simply looking at the analysis report from the older binary, which says exactly which byte at which glyph is problematic. Then look at how FreeType behaves around those bytes at those glyphs. Some of the reports are wrong, but so far, most of the analysis are correct, and they turn out to be (silent) workaround for buggy fonts in FreeType's code through out past history - half of it corresponds to bug reports submitted to FreeType's bug tracker e.g. - "I have this buggy font which causes FreeType to choke - can you have a look?"

So nothing like cool reverse-engineering going on there re-implementing the rasterization test. Just looking at interesting reports which say a font is problematic at a particular glyph at a particular byte with a particular problem, and see how FreeType behaves there. Often enough, I saw a comment in FreeType code at exactly that spot, with a comment 'some fonts are so brain-damaged, we try to cope this way'. My diagnostics patch is just a 'tell me about it at this spot'.

Loading

@HinTak
Copy link
Author

@HinTak HinTak commented Jul 14, 2016

I have tagged and uploaded another version, 4 days after the last one - it is mostly about exciting major enhancement that you should have ASAP.

Loading

@HinTak
Copy link
Author

@HinTak HinTak commented Jul 18, 2016

July 14 was b41. I have tagged and signed for the first time b54 as "FontVal-2.0.0".

dotnet/mono binaries, also signed (hurray!)
https://sourceforge.net/projects/hp-pxl-jetready/files/Microsoft%20Font%20Validator/

When I get round to build the mac os x binaries I'd write properly.

Loading

@HinTak
Copy link
Author

@HinTak HinTak commented Jul 31, 2016

Cut-and-paste and repeating my own writing isn't much fun, so I'd just say I have gpg-signed the 2.0 commit tags, (there are two, one on the py branch) , and am serious about it.

I refer people to the sort-of release overview:
http://typedrawers.com/discussion/1717/font-validator-2-0-arrived

and the summary test result on the libre font part of ~5000 fonts
http://typedrawers.com/discussion/1716/libre-font-test-results-and-hall-of-shame-list
and the details up at
http://htl10.users.sourceforge.net/tmp/FontVal-test-results-2016July/

And the usual, please make a donation at https://sourceforge.net/p/hp-pxl-jetready/donate/ , and if you use it for work, please consider asking your employers to.

For @schriftgestalt mostly : you want the "FontVal-2.0.0-py-osxbin.tgz" and extract ".bin/FontValidator" as well as the stand-alone libfreetype (I don't do libpng.dylib any more) from
https://sourceforge.net/projects/hp-pxl-jetready/files/Microsoft%20Font%20Validator/misc/
and presumably you want to update and offer "+raster-tests" up. Other than those two (no more libpng.dylib, and the "+raster-tests") it should work like it did. While you are at it, you want "FontVal-2.0.0-helpsrc.zip" also - there are a dozen additional figures and docs for the glyf test, since the last time you got hold of the unpacked help src.

Loading

prepare referenced this issue in prepare/Font-Validator Dec 9, 2017
FontVal 2.1.x no longer need these separate freetype/png
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

7 participants