build system improvements #1750
Conversation
|
For libk2pdfopt, I think the external project should switch to the official source archive + an overlay directory (similarly to how lunarsvg is handled). That's how I handle it on my meson branch too. |
benoit-pierre
force-pushed
the
pr/build_system_improvements
branch
from
0a39610
to
6332e2a
Compare
|
Right, emulator builds don't go through docker, so meson is missing… |
benoit-pierre
force-pushed
the
pr/build_system_improvements
branch
from
7218840
to
c16004c
Compare
|
fixup commits aren't supported by GH rebase btw (unless they added support in the last few months), so for that a force push will be required. Edit: at a glance this looks great, but I don't know if I'll have time to look at it properly this week. Hopefully I'll find the time to give it a quick try tonight though. :-) |
|
Not competent and experienced enough to judge. Some comments and questions: 40b323b leptonica: are these new patches ? Taken from libkopt2pdf project? Is there some history? 934627b zlib: drop of patches intentional? (I've seen here and there some added comments that sounded (wording, vocabulary) like @NiLuJe 's :) I've seen some of these removed from other commits. You did just moved things around? Or are you beginning to sound like @NiLuJe ? :)) About using tarball and specifying 733bbbb 8ff3898 ae75343 5919035 any reason for the reordering of $(LIBXYZ), or just for balance of the text? ae75343 some ellipsis 043d3c9 So, what's our official policy ? Do we love CMake or not ? :) 8499524 669045d So, this moves some bits to (and requires?) meson. And asking again #1697 (comment) :
Why all the pain of this PR?
I must say it's quite painful to update LunaSVG. I have a local git clone of the upstream repo (not pushed anywhere) where I keep my history of changes, and just upload the extended.patch here. I'm not certain it's the best way to handle that. |
I believe that's simply a port of the current behavior, not a behavioral change.
The projects themselves have introduced meson. (I didn't double check but I know at least some of them did.) |
I just extracted the changes to leptonica from the version of k2pdfopt being used. No relevant changes to tesseract (only patches for old Windows versions, e.g. XP, or debug traces).
Yes: those are autotools specific.
Some.
Probably for the same reasons: PTSD from to much CMake :P
You have to fill-in the md5sum manually, yes. Note however that it's optional, so you can:
CMake automatically adds it, to this was done to reduce differences with the final version (produce the same binaries as much as possible to avoid regressions, or at least the same
Same as above.
Wrong commit?
I certainly don't, but it still better than make or autotools (for some definition of better).
Yeah, it can be hell. But to be fair, the current build system is not doing itself any favor by installing each external project to a dedicated directory, instead of a common staging directory that can be more easily searched by CMake. That simplify things a lot. Even if you still have to contend with the numerous ways a CMake project has to find particular dependency (and the potential differences between CMake versions, as in how the initial call to |
No, it's a change. Same with not compiling gettext, and then relying on
Yes, meson is the default build system of those projects. |
Right, in lunasvg's case, there's some patching to the source involved, but not for libk2pdfopt: the overlay is independent. Of course you still need to update the leptonica / tesseract patches (if any), and the overlay code itself. |
I don't like it either, but it's required for proper dependencies handling (particularly on incremental parallel builds). And there's no dependency file equivalent for by-products. Initially, I had setup every external project to always build (see But thanks to the sources depfile trick, there's no need for And there might be a way to avoid the need for using by-products by moving the crengine library, koreader libraries, and miscellaneous executables builds to dedicated external projects. I need to investigate. |
|
I also need someone with access to macOS to test. I checked the resulting executable/libraries have the right runtime path / soname information, and that's it. I think all the |
Indeed :/ Can't find it, and not looking again at these 180 commits :) |
Once upon a time we had a GLIB (without libiconv and gettext, except on MacOS) and a GLIB_STATIC (for Android, with libiconv). If that's no longer the case it would require investigation into why that was changed. At a glance it looks like the relevant conditions were dropped in #1690, meaning that was the change in behavior, not this, imho. |
benoit-pierre
force-pushed
the
pr/build_system_improvements
branch
2 times, most recently
from
17df59c
to
2bec084
Compare
I assume that for chrpath anything will do. That is, it's been at 0.16 for years, unless the new 0.17 offers something we need. But is the same true for Meson? |
|
Those projects that use meson are usually pretty conservative (same with WrapDB): ▹ grep meson_version build/x86_64-pc-linux-gnu-debug/cmake/*/source/meson.build
build/x86_64-pc-linux-gnu-debug/cmake/fribidi/source/meson.build
2: meson_version : '>= 0.54')
build/x86_64-pc-linux-gnu-debug/cmake/freetype2/source/meson.build
27: meson_version: '>= 0.55.0',
build/x86_64-pc-linux-gnu-debug/cmake/harfbuzz/source/meson.build
2: meson_version: '>= 0.55.0',
build/x86_64-pc-linux-gnu-debug/cmake/proxy-libintl/source/meson.build
3: meson_version : '>= 0.46.0',
build/x86_64-pc-linux-gnu-debug/cmake/glib/source/meson.build
3: meson_version : '>= 0.47.0', |
With that last commit, it's actually not needed anymore: no need to list all build by-products for each external projects anymore, each project actual install rules are used. I moved the crengine build to a dedicated external project. Same with the koreader libraries / executable. (But I kept support for things like chrpath was only needed to cleanup meson build runtime path before manually "installing" to staging, that's unnecessary now that meson install rules are used. I'll have to check if the macOS runtime path is still correct however: from past experience with my meson branch, custom runtime path entries are stripped on install. |
Not conservative enough for the Ubuntu 20.04 repos. :-) But perhaps the time is right to switch to 22.04, if Android wants to participate. |
|
We can always install from pip, even if that will not always give us the last version (e.g. the kindle image uses Python 3.6, so meson 0.61.5 is the latest supported version). |
|
Of course we can, but I'd like to avoid that if we don't have to. |
|
What about the appimage? What's the oldest target? |
This is close to the same statement ftr. I want them on 20.04 (or maybe 22.04) but the Android build mysteriously crashes in CI. We have yet to test if the update from a while back solves that though. The 0.3.0 image (with 20.04) has been available for immediate use for all but Android for a year. ^_^ |
|
OK. One thing I'd like to PR, is the test suite improvements I made on my meson branch: isolating tests, and then using meson as a test runner (each spec file run is isolated to its own process, tests run in parallel). It's much faster, and you don't have to work around the lack of real isolation in busted. |
That's basically what I was referring to above by saying maybe the time has come for 22.04. For anything else it really doesn't matter all that much, but I consider lower versions (when reasonable) more advantageous since they ensure it's easy for people to start hacking. Keeping them in line with the AppImage is two birds one stone. |
|
Ubuntu 20.04 has meson 0.53.2, 22.04: 0.61.2. |
|
Right, but FreeType would like 0.55. |
|
Yep, so update to 22.04, or pip install. ¯\(ツ)/¯ |
Don't build `SDL2main` library (we don't need it).
Ensure concurrent access to the clone directory work as expected (either from compiling for different platforms in parallel, or when using the same source for different external projects, like fbdepth / fbink).
Since it does not rely on MuPDF dependencies.
Fix corner case when using `make curl-download` from scratch.
Add a dedicated `generate` step, rather than use `patch`: allow for manual patching before actually generating the required files.
Needed for some rare DJVU files using JPEG encoding.
Necessary if ninja is still used by some external project build systems (e.g. meson) even when using CMake's make generator.
Reduce the toolchain file to its essentials, as setting a `CXX` variable breaks detection of the C++ compiler supported features (Cf. https://gitlab.kitware.com/cmake/cmake/-/issues/22125#note_947613).
benoit-pierre
force-pushed
the
pr/build_system_improvements
branch
from
9316664
to
eb2565e
Compare
Done. |
|
@NiLuJe You said you wanted a merge commit? |
There was a problem hiding this comment.
Reviewed 2 of 102 files at r11, 2 of 34 files at r12, 99 of 99 files at r13, all commit messages.
Reviewable status:complete! all files reviewed, all discussions resolved (waiting on @benoit-pierre)
Rework the build system to make it faster and better.
TODO:
[ ] update docker images (chrpath+meson)Highlights:
steps as much as possible (e.g. all download / patch steps can be executed in
parallel).
build steps (e.g.:
make koreader-creto build onlylibkoreader-cre.soand allits dependencies,
make leptonica-reto rebuild leptonica, etc…). Top level targetscan be serialized and are forwarded to CMake (so its possible to do
make clean zlibfor example).
thirdparty/*/build/…,move the external projects build directories under the machine specific build
directory: reduce the depth of the build tree, and simplify clean rules.
directory: simplify dependency lookups in external projects (including CMake).
to pass parameters to CMake and simplify building CMake based external projects.
luajsondependency, don't use luarocks for building/installing LUAexternal projects: faster builds, no need for network, all dependencies explicitly
accounted for, simpler install tree (no need for the
rocksdirectory anymore).-las as well as-jwhen running make / ninja to manage system load.build by their respective external projects.
during the next build.
Build times (on my machine: -j8 / -j8 -l8, from scratch: after
make distclean):(÷ 2.5)(÷ 6.4)(÷ 2.2)(÷ 6.6)(÷ 2.3)(÷ 6.4)Disk usage (downloads + build tree(s))
Notes:
This change is