Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fontbakery: init at 0.10.40 #228099

Merged
merged 23 commits into from
Dec 3, 2023
Merged

fontbakery: init at 0.10.40 #228099

merged 23 commits into from
Dec 3, 2023

Conversation

danc86
Copy link
Contributor

@danc86 danc86 commented Apr 25, 2023

Description of changes

Font Bakery is a command-line tool for checking the quality of font projects.

It's used for example in the build process for the Inter font family.

This PR also includes a number of new Python packages which are dependencies of fontbakery.

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • [ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.05 Release Notes (or backporting 22.11 Release notes)
    • [ ] (Package updates) Added a release notes entry if the change is major or breaking
    • [ ] (Module updates) Added a release notes entry if the change is significant
    • [ ] (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@danc86
Copy link
Contributor Author

danc86 commented Apr 25, 2023

Result of nixpkgs-review pr 228099 run on x86_64-linux 1

85 packages built:
  • dehinter (python310Packages.dehinter)
  • dehinter.dist (python310Packages.dehinter.dist)
  • font-v (python310Packages.font-v)
  • font-v.dist (python310Packages.font-v.dist)
  • fontbakery (python310Packages.fontbakery)
  • fontbakery.dist (python310Packages.fontbakery.dist)
  • opentype-sanitizer
  • python310Packages.axisregistry
  • python310Packages.axisregistry.dist
  • python310Packages.babelfont
  • python310Packages.babelfont.dist
  • python310Packages.beziers
  • python310Packages.beziers.dist
  • python310Packages.collidoscope
  • python310Packages.collidoscope.dist
  • python310Packages.commandlines
  • python310Packages.commandlines.dist
  • python310Packages.fontfeatures
  • python310Packages.fontfeatures.dist
  • python310Packages.gflanguages
  • python310Packages.gflanguages.dist
  • python310Packages.glyphsets
  • python310Packages.glyphsets.dist
  • python310Packages.glyphtools
  • python310Packages.glyphtools.dist
  • python310Packages.kurbopy
  • python310Packages.kurbopy.dist
  • python310Packages.opentypespec
  • python310Packages.opentypespec.dist
  • python310Packages.ots-python
  • python310Packages.ots-python.dist
  • python310Packages.rstr
  • python310Packages.rstr.dist
  • python310Packages.sre-yield
  • python310Packages.sre-yield.dist
  • python310Packages.stringbrewer
  • python310Packages.stringbrewer.dist
  • ufolint (python310Packages.ufolint)
  • ufolint.dist (python310Packages.ufolint.dist)
  • python310Packages.vharfbuzz
  • python310Packages.vharfbuzz.dist
  • python310Packages.youseedee
  • python310Packages.youseedee.dist
  • python311Packages.axisregistry
  • python311Packages.axisregistry.dist
  • python311Packages.babelfont
  • python311Packages.babelfont.dist
  • python311Packages.beziers
  • python311Packages.beziers.dist
  • python311Packages.collidoscope
  • python311Packages.collidoscope.dist
  • python311Packages.commandlines
  • python311Packages.commandlines.dist
  • python311Packages.dehinter
  • python311Packages.dehinter.dist
  • python311Packages.font-v
  • python311Packages.font-v.dist
  • python311Packages.fontbakery
  • python311Packages.fontbakery.dist
  • python311Packages.fontfeatures
  • python311Packages.fontfeatures.dist
  • python311Packages.gflanguages
  • python311Packages.gflanguages.dist
  • python311Packages.glyphsets
  • python311Packages.glyphsets.dist
  • python311Packages.glyphtools
  • python311Packages.glyphtools.dist
  • python311Packages.kurbopy
  • python311Packages.kurbopy.dist
  • python311Packages.opentypespec
  • python311Packages.opentypespec.dist
  • python311Packages.ots-python
  • python311Packages.ots-python.dist
  • python311Packages.rstr
  • python311Packages.rstr.dist
  • python311Packages.sre-yield
  • python311Packages.sre-yield.dist
  • python311Packages.stringbrewer
  • python311Packages.stringbrewer.dist
  • python311Packages.ufolint
  • python311Packages.ufolint.dist
  • python311Packages.vharfbuzz
  • python311Packages.vharfbuzz.dist
  • python311Packages.youseedee
  • python311Packages.youseedee.dist

Copy link
Contributor

@doronbehar doronbehar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great PR overall! General tip: Because there are so many new packages, it'd be nice to test them on Darwin with ofborg as well. You could trigger that automatically if all the commits' messages' titles will be change from python3Packages... to python311Packages.... Only the latest is directly evaluated by Hydra, and ofborg counts on that as well.

Also, sorry for waiting so long for a review. We'll sure get all of these packages as quick as possible.

P.S There's a small, trivial merge conflict.

@@ -13,13 +14,26 @@ buildPythonPackage rec {
hash = "sha256-fdI4CBUSMbdKW0qg4y22wjWI6bhotgDkQdFMc9X00as=";
};

patches = [
./0001-use-packaged-unicode-data.patch
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Add a comment near the patch, and squash this commit to the
  2. Instead of the substituteInPlace used in preBuild, use the Nix function substituteAll.
  3. Squash this commit to the previous, init commit.


# Package has no unit tests, also it requires network access so we can't
# try to exercise it in checkPhase.
pythonImportsCheck = [ "youseedee" ];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, doCheck = false won't hurt.

# https://github.com/simoncozens/fontFeatures/pull/60#issuecomment-1518963962
# Have to write out the -k option instead of using disabledTests because
# of the test parameter.
"-k 'not (test_round_trip and mark_attachment)'"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to use disabledTests here instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, that didn't work because it's a parametrized test... but the test is passing in the latest upstream release now.

pytestCheckHook
uharfbuzz
youseedee
];

pythonRelaxDeps = [ "protobuf" ];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

@@ -1508,6 +1508,8 @@ with pkgs;

font-v = with python3Packages; toPythonApplication font-v;

fontbakery = with python3Packages; toPythonApplication fontbakery;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this package supposed to be available as a python module as well?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It required as a module for https://github.com/notofonts/notobuilder/ which we would need if we ever want to build noto-fonts from source

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the packages in this PR should be available as Python modules (since a lot of font build processes will depend on them that way, even if they're really a command line tool like fontbakery). Plus for all those packages where upstream's docs give some indication that the tool is designed to be invoked from the command line, I've exported them as Python applications as well.

];

postInstall = ''
install -D -m0644 snippets/fontbakery.bash-completion \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: It might be safer to use installShellFiles.

Comment on lines 50 to 60
# Fix tests to work with ufo2ft >= 2.29.0
# https://github.com/googlefonts/fontbakery/pull/4126
./0001-fix-tests-to-work-with-ufo2ft-2.29.0.patch
# Fix tests when not inside a fontbakery git checkout
# https://github.com/googlefonts/fontbakery/pull/4125
./0002-don-t-assume-the-tests-are-always-running-in-a-git-c.patch
# Mock HTTP requests in tests (note we still have to skip some below)
# https://github.com/googlefonts/fontbakery/pull/4124
./0003-use-not-listed_on_gfonts_api-condition-instead-of-no.patch
./0004-use-requests-instead-of-download_file-in-github_gfon.patch
./0005-mock-HTTP-requests-in-tests.patch
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests that have open PRs, please use fetchpatch. I guess many of these patches are already merged upstream so hopefully this will look different after you'd revisit this PR.

@danc86
Copy link
Contributor Author

danc86 commented Nov 7, 2023

Thanks for the review, and for the tip about python310Packages in the commit messages.

I'll rebase and update this PR in the next few days when I get some spare time.

@danc86
Copy link
Contributor Author

danc86 commented Nov 25, 2023

I've rebased this PR, fixed the conflicts, and amended it as follows:

  • latest upstream release of all packages
  • added shaperglot, new dependency of fontbakery
  • squashed all changes into each "init" commit
  • used explicit doCheck = true
  • used substituteAll nix function on nixpkgs-specific patches and commented them
  • switched to upstream Cargo.lock for kurbopy
  • moved opentype-sanitizer under pkgs/by-name and removed rec in favour of mkDerivation (finalAttrs:
  • used fetchpatch for the one remaining open PR on fontbakery
  • used python311Packages in all the commit messages, to trigger ofborg properly

@danc86
Copy link
Contributor Author

danc86 commented Nov 25, 2023

Result of nixpkgs-review pr 228099 run on x86_64-linux 1

89 packages built:
  • dehinter (python311Packages.dehinter)
  • dehinter.dist (python311Packages.dehinter.dist)
  • font-v (python311Packages.font-v)
  • font-v.dist (python311Packages.font-v.dist)
  • fontbakery (python311Packages.fontbakery)
  • fontbakery.dist (python311Packages.fontbakery.dist)
  • opentype-sanitizer
  • python310Packages.axisregistry
  • python310Packages.axisregistry.dist
  • python310Packages.babelfont
  • python310Packages.babelfont.dist
  • python310Packages.beziers
  • python310Packages.beziers.dist
  • python310Packages.collidoscope
  • python310Packages.collidoscope.dist
  • python310Packages.commandlines
  • python310Packages.commandlines.dist
  • python310Packages.dehinter
  • python310Packages.dehinter.dist
  • python310Packages.font-v
  • python310Packages.font-v.dist
  • python310Packages.fontbakery
  • python310Packages.fontbakery.dist
  • python310Packages.fontfeatures
  • python310Packages.fontfeatures.dist
  • python310Packages.gflanguages
  • python310Packages.gflanguages.dist
  • python310Packages.glyphsets
  • python310Packages.glyphsets.dist
  • python310Packages.glyphtools
  • python310Packages.glyphtools.dist
  • python310Packages.kurbopy
  • python310Packages.kurbopy.dist
  • python310Packages.opentypespec
  • python310Packages.opentypespec.dist
  • python310Packages.ots-python
  • python310Packages.ots-python.dist
  • python310Packages.rstr
  • python310Packages.rstr.dist
  • python310Packages.shaperglot
  • python310Packages.shaperglot.dist
  • python310Packages.sre-yield
  • python310Packages.sre-yield.dist
  • python310Packages.stringbrewer
  • python310Packages.stringbrewer.dist
  • python310Packages.ufolint
  • python310Packages.ufolint.dist
  • python310Packages.vharfbuzz
  • python310Packages.vharfbuzz.dist
  • python310Packages.youseedee
  • python310Packages.youseedee.dist
  • python311Packages.axisregistry
  • python311Packages.axisregistry.dist
  • python311Packages.babelfont
  • python311Packages.babelfont.dist
  • python311Packages.beziers
  • python311Packages.beziers.dist
  • python311Packages.collidoscope
  • python311Packages.collidoscope.dist
  • python311Packages.commandlines
  • python311Packages.commandlines.dist
  • python311Packages.fontfeatures
  • python311Packages.fontfeatures.dist
  • python311Packages.gflanguages
  • python311Packages.gflanguages.dist
  • python311Packages.glyphsets
  • python311Packages.glyphsets.dist
  • python311Packages.glyphtools
  • python311Packages.glyphtools.dist
  • python311Packages.kurbopy
  • python311Packages.kurbopy.dist
  • python311Packages.opentypespec
  • python311Packages.opentypespec.dist
  • python311Packages.ots-python
  • python311Packages.ots-python.dist
  • python311Packages.rstr
  • python311Packages.rstr.dist
  • shaperglot (python311Packages.shaperglot)
  • shaperglot.dist (python311Packages.shaperglot.dist)
  • python311Packages.sre-yield
  • python311Packages.sre-yield.dist
  • python311Packages.stringbrewer
  • python311Packages.stringbrewer.dist
  • ufolint (python311Packages.ufolint)
  • ufolint.dist (python311Packages.ufolint.dist)
  • python311Packages.vharfbuzz
  • python311Packages.vharfbuzz.dist
  • python311Packages.youseedee
  • python311Packages.youseedee.dist

@danc86 danc86 changed the title fontbakery: init at 0.8.11 fontbakery: init at 0.10.40 Nov 26, 2023
@doronbehar
Copy link
Contributor

Sorry for not responding to this. I haven't forgotten this PR. Hopefully will find time to approve it tomorrow or next week.

Copy link
Contributor

@doronbehar doronbehar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very well written! Thanks for this big contribution, and again sorry for this one more iteration. I will have time tomorrow for sure to merge this if these small issues will be fixed!

pkgs/development/python-modules/glyphsets/default.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/babelfont/default.nix Outdated Show resolved Hide resolved
unzip
];
preCheck = ''
unzip -o $dist/*.whl
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm interesting. Could you explain this a bit in a comment?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit of a dodgy hack and I shouldn't have left it in (not without a comment).

The actual problem is the classic pytest import confusion. Because of pytest's manipulations of sys.path, the test process ends up importing kurbopy/__init__.py from the source directory, which does not contain the built .so, and so it fails like this:

/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_basic.py:1: in <module>
    from kurbopy import Point, Vec2, TranslateScale
kurbopy/__init__.py:1: in <module>
    from .kurbopy import Point as Point
E   ModuleNotFoundError: No module named 'kurbopy.kurbopy'

We want it to import kurbopy from the nix store via PYTHONPATH instead.

So this hack was stealing $dist and using it to populate the source directory with a usable copy of the built .so, but a proper solution would be to make pytest import kurbopy from the installed directory like a proper installCheckPhase is supposed to do.

Pytest has an --import-mode option which should help with that kind of mess. But no matter what I do, sys.path always imports from . first. I think it's because pytestCheckHook does python -m pytest, and that's one of the side effects noted in the docs.

Anyway, I can just hack it to delete the module from the source directory in preCheck, so that it imports the installed copy from the nix store instead. With a comment explaining what's going on. :-)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyway, I can just hack it to delete the module from the source directory in preCheck, so that it imports the installed copy from the nix store instead. With a comment explaining what's going on. :-)

Thanks for the thorough explanation @danc86 . Could you do me please 1 more favor? In the very well written comment, could you mention that this is a common issue in nixpkgs and it is investigated in #255262 ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR should be ready after issue #255262 is mentioned.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I've reworded the comment to mention #255262.

pkgs/development/python-modules/shaperglot/default.nix Outdated Show resolved Hide resolved
@doronbehar
Copy link
Contributor

Also, there's a merge conflict, now I'm noticing - probably fixable easily.

@danc86
Copy link
Contributor Author

danc86 commented Dec 3, 2023

I've rebased this and amended it as follows:

  • format = "pyproject" -> pyproject = true
  • added a comment to glyphsets explaining why pythonRelaxDepsHook doesn't work
  • handle pytest import path problems in kurbopy in a cleaner way, with an explanatory comment

@doronbehar
Copy link
Contributor

Building fontbakery now on my machine, unfortunately ofborg didn't build it and all other packages by itself for all platforms, although the commit messages should have triggered it - perhaps it is because there are too many new packages. Will merge immediately afterwards.

@danc86
Copy link
Contributor Author

danc86 commented Dec 3, 2023

Thanks for the reviews. :-)

@doronbehar doronbehar merged commit fab84c7 into NixOS:master Dec 3, 2023
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants