-
Notifications
You must be signed in to change notification settings - Fork 235
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Move to a GNUMakefile-based build system #795
Comments
An example of such a build system in a well-known project is in the Linux man-pages project. https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/README |
man-pages is not ideal example It would be better to reference projects which
|
The Linux man-pages project does indeed build binaries. It builds the example programs included in the EXAMPLES sections in man2 and man3 pages. Not only builds binaries, but also runs many many linters on the source code of those programs. Probably more than many programs out there. :) In fact, I would include those linters in the shadow build system, which would be a great thing. There are no libraries, though. I have some library projects where the makefiles do build shared and static libraries, but as they are only personal projects done for fun, I don't really maintain them so well, so I don't think it would be good showing them as state-of-the-art Makefiles (there are many things I'd improve on them, but I don't have the time to do so). I don't check for existence of libraries, either, as I don't write variable dependencies, but it would be possible to do so. Something using
I never used their build system, so I can't talk of them. Here's one of my library projects. Don't judge the the missing features too badly, as I don't have time or motivation to do it as good as I'd like to. :) |
@alejandro-colomar please do not do this. The autotools helps us enforce GNU conventions, and Automake solves 100s of problems that reinventing yourself isn't worth it. Every time someone tries to reinvent Automake, it goes horribly wrong. |
Here's one of my biggest concerns with Autotools: Let's say I add files With a properly-written Makefile, that problem simply doesn't exist, as sources and dependencies are recalculated from the repository itself. Of course, when writing a Makefile, I try to follow GNU conventions as closely as possible.
I'm not a maintainer of the project. I'll do this, but will propose it for public discussion. I understand that the discussion may end up in dropping the patches.
If you list them, I promise to not introduce them. :)
I wonder if this happens in the man-pages case. If so, I'm interested in fixing any issues. Please report them if there are. |
Here are some things people always fail at:
Just have a look at bzip2 in Gentoo and the hacks we need to employ to make it work as a "simple Makefile" that constantly causes issues for us. There is no such thing as a simple Makefile. I understand that people hate Automake, but your NIH'd makefiles are worse, I can guarantee that from 10 years of Gentoo packaging experience. If you want to move away from Autotools, rather consider something like Meson that avoids all of these legacy pitfalls. |
This is the most basic case and you'd expect "simple Makefiles" to work fine here, but that's not the case we should be using to evaluate the choice of build system. Because build systems help solve portability problems in non-trivial cases. Keep in mind that this is purely a development issue - and IMO not a particularly problematic one, as with bisecting, you can use |
Thanks.
Yes, it's a bit difficult. It took me some time to get something reliable,
A bit generic. All I can say is that I try to follow them as much as possible.
The Linux man-pages' GNUMakefile, all builds are VPATH builds. And it works fine, AFAIK. :)
I've never done this, but having a correct distcheck: dist
tar xf ...
cd ...
$(MAKE) installcheck
$(MAKE) uninstall
$(MAKE) distclean
$(MAKE) dist
This is relatively easy. In the man-pages I have many flags variables, and they work fine, AFAIK.
libtool is usable from a Makefile, isn't it? Use it in the rules for building library files as you'd use $(CC) otherwise. Never done this, though. I'd need to try to see if it's problematic.
No python here.
If someone wants to do that, I'm fine with it. I just don't have the experience with those tools, and have it with GNU make(1). |
That still takes the extra time that configure + make takes over a make that only remakes the necessary things. And when rebasing and changing small things, this issue also appears. Yes, the issue is only suffered by developers, and not distributors, I admit. You'll probably have much different concerns to those I have. But since you do package the Linux man-pages, I ask you to report any distribution inconveniences of that Makefile. I admit it's not distributing a library or anything similarly complex, but please report anything from there that is substandard to you. Then I can work based on that to try to write a Makefile up to those expectations. If it doesn't meet them, feel free to express it and reject it. |
if you just changed |
I didn't know that, and don't think it's trivial information that the common programmer knows. But okay, it can be learnt. But still, |
Here's a practical test using https://git.kernel.org/pub/scm/linux/kernel/git/kay/libabc.git:
and then emulate adding a new file to
notice how Automake-generated files know exactly what to regenerate (i.e. |
First I build from a clean state:
For calibration, let's see how a no-op alx@asus5775:~/src/shadow/shadow/STRLCPY$ time make -j |& wc -l; echo $?
129
real 0m0.156s
user 0m0.118s
sys 0m0.041s
0 Now I move part of alx@asus5775:~/src/shadow/shadow/STRLCPY$ cp lib/must_be.h lib/must_be_helper.h
alx@asus5775:~/src/shadow/shadow/STRLCPY$ vi !$
vi lib/must_be_helper.h
alx@asus5775:~/src/shadow/shadow/STRLCPY$ vi lib/must_be.h
alx@asus5775:~/src/shadow/shadow/STRLCPY$ vi lib/Makefile.am And then let's try the least intrusive rebuild (that I know of as of now): alx@asus5775:~/src/shadow/shadow/STRLCPY$ time automake |& wc -l; echo $?
0
real 0m0.973s
user 0m0.950s
sys 0m0.038s
0
alx@asus5775:~/src/shadow/shadow/STRLCPY$ time ./config.status |& wc -l; echo $?
41
real 0m0.680s
user 0m0.697s
sys 0m0.171s
0
alx@asus5775:~/src/shadow/shadow/STRLCPY$ time make -j |& wc -l; echo $?
7826
real 1m21.713s
user 4m44.514s
sys 0m18.418s
0 I really hope I don't understand autotools and there's really a hidden way to do it correctly, because 1m22s just for this is huge. |
And for comparison, here's a project of mine (a C library), which uses a single Makefile. Build and lint from a clean state alx@asus5775:~/src/alx/libc/libc-mem$ git clean -dffx |& wc -l; echo $?
1
0
alx@asus5775:~/src/alx/libc/libc-mem$ time make -j build lint |& wc -l; echo $?
127
real 0m2.893s
user 0m8.902s
sys 0m1.803s
0 A calibration with a no-op make: alx@asus5775:~/src/alx/libc/libc-mem$ time make -j build lint |& wc -l; echo $?
14
real 0m0.143s
user 0m0.205s
sys 0m0.105s
0 Now move some trivial stuff to a helper file: alx@asus5775:~/src/alx/libc/libc-mem$ cp include/c/mem/cpy/mempcpy.h include/c/mem/cpy/mempcpy_helper.h
alx@asus5775:~/src/alx/libc/libc-mem$ vi !$
vi include/c/mem/cpy/mempcpy_helper.h
alx@asus5775:~/src/alx/libc/libc-mem$ vi include/c/mem/cpy/mempcpy.h And rebuild what needs to be built: alx@asus5775:~/src/alx/libc/libc-mem$ time make -j build lint |& wc -l; echo $?
25
real 0m0.527s
user 0m1.292s
sys 0m0.313s
0 Only the stuff that really needed it has been updated. |
I've reconsidered. I didn't remember that I already considered this some time ago. srcdir$ make EXTRA_CFLAGS=-O3 builddir=.foo/O3
srcdir$ make EXTRA_CFLAGS=-O0 builddir=.foo/O0 (This specific case is unnecessary in my Makefile, as I build at all optimization levels in a loop, just to catch more warnings --and as a side effect, be able to easily compare the different assemblies that the compiler produces at different optimization levels; it's an interesting feature--, but it's to show a trivial example.) |
FWIW, I'm using meson with great success for all my userspace projects. I'll never again consider using autotools, cmake nor plain make. |
Please run |
No, it's not supported - you have to regenerate libtool using configure. Standalone libtool has various problems. |
Okay, and why? For what are we gaining? Having this conversation every time someone thinks autotools is pointless is not a great use of time. The only issue here is you've found regenerating autotools to be a pain for development, to which I've noted that the times given seem unusual and not consistent with my experience.
|
I used Where is I wouldn't mind doing |
Thanks, this does seem to work. alx@asus5775:~/src/shadow/shadow/master$ git clean -dffx |& wc -l; echo $?
0
0
alx@asus5775:~/src/shadow/shadow/master$ time ./autogen.sh |& wc -l; echo $?
348
real 0m12.749s
user 0m10.225s
sys 0m2.010s
0
alx@asus5775:~/src/shadow/shadow/master$ time make -j |& wc -l; echo $?
8072
real 1m21.154s
user 4m45.685s
sys 0m17.558s
0
alx@asus5775:~/src/shadow/shadow/master$ time make -j |& wc -l; echo $?
129
real 0m0.162s
user 0m0.143s
sys 0m0.022s
0 Now add stuff: alx@asus5775:~/src/shadow/shadow/master$ cp lib/must_be.h lib/must_be_helper.h
alx@asus5775:~/src/shadow/shadow/master$ vi !$
vi lib/must_be_helper.h
alx@asus5775:~/src/shadow/shadow/master$ vi lib/must_be.h
alx@asus5775:~/src/shadow/shadow/master$ vi lib/Makefile.am
alx@asus5775:~/src/shadow/shadow/master$ time make -j |& wc -l; echo $?
245
real 0m4.849s
user 0m10.063s
sys 0m1.960s
0 Remove stuff: alx@asus5775:~/src/shadow/shadow/master$ rm lib/must_be_helper.h
alx@asus5775:~/src/shadow/shadow/master$ git restore lib/must_be.h
alx@asus5775:~/src/shadow/shadow/master$ vi lib/Makefile.am
alx@asus5775:~/src/shadow/shadow/master$ time make -j |& wc -l; echo $?
245
real 0m4.701s
user 0m10.044s
sys 0m1.804s
0 Move a source: alx@asus5775:~/src/shadow/shadow/master$ mv lib/agetpass.c lib/agetpass2.c
alx@asus5775:~/src/shadow/shadow/master$ vi lib/Makefile.am
alx@asus5775:~/src/shadow/shadow/master$ time make -j |& wc -l; echo $?
172
real 0m3.132s
user 0m5.036s
sys 0m1.260s
0 Really, thank you very much! |
I'm still unsure we can add linters that work similar to how I do it in the man-pages (that is, that make is aware of what's been linted so far), but I'll try to do it keeping autotools. |
Another question: How can one build out-of-tree? The first step, autoreconf, doesn't seem to work out of tree. alx@asus5775:~/src/shadow/shadow/master$ git clean -dffx |& wc -l; echo $?
0
0
alx@asus5775:~/src/shadow/shadow/master$ git restore . |& wc -l; echo $?
0
0
alx@asus5775:~/src/shadow/shadow/master$ mkdir .tmp
alx@asus5775:~/src/shadow/shadow/master$ cd .tmp/
alx@asus5775:~/src/shadow/shadow/master/.tmp$ ../autogen.sh
autoreconf: export WARNINGS=
autoreconf: error: 'configure.ac' is required
alx@asus5775:~/src/shadow/shadow/master/.tmp$ cat ../autogen.sh
#! /bin/sh
autoreconf -v -f --install || exit 1
./configure \
CFLAGS="-O2 -Wall" \
--enable-lastlog \
--enable-man \
--enable-maintainer-mode \
--enable-shared \
--without-libpam \
--with-selinux \
"$@"
alx@asus5775:~/src/shadow/shadow/master/.tmp$ autoreconf -v -f --install .. |& wc -l; echo $?
73
0
alx@asus5775:~/src/shadow/shadow/master/.tmp$ git status --ignored | head
On branch master
Your branch is up to date with 'alx/master'.
Ignored files:
(use "git add -f <file>..." to include in what will be committed)
../ABOUT-NLS
../Makefile.in
../aclocal.m4
../autom4te.cache/
../compile It generates the stuff in the source tree, not in the build tree. |
AFAIK you can't make |
Huh, this is interesting. I'd always seen |
Hmm, since autoreconf doesn't modify git files, and I don't need to run it often, I can live with it. I'll complain if/when I find any inconveniences. :) |
Link: <shadow-maint#795> Cc: Sam James <sam@gentoo.org> Signed-off-by: Alejandro Colomar <alx@kernel.org>
This allows to do the following: ~/src/shadow/shadow/master$ mkdir .tmp/ && cd .tmp/ ~/src/shadow/shadow/master/.tmp$ ../autogen.sh Link: <shadow-maint#795> Cc: Sam James <sam@gentoo.org> Signed-off-by: Alejandro Colomar <alx@kernel.org>
This allows to do the following: ~/src/shadow/shadow/master$ mkdir .tmp/ && cd .tmp/ ~/src/shadow/shadow/master/.tmp$ ../autogen.sh Link: <shadow-maint#795> Reviewed-by: Sam James <sam@gentoo.org> Signed-off-by: Alejandro Colomar <alx@kernel.org>
Glad to read that Alejandro's problems have been solved, and that he's improved his workflow. I still think that we need to improve our building workflow, but we can discuss the possibilities at some other time. |
This allows to do the following: ~/src/shadow/shadow/master$ mkdir .tmp/ && cd .tmp/ ~/src/shadow/shadow/master/.tmp$ ../autogen.sh Link: <#795> Reviewed-by: Sam James <sam@gentoo.org> Signed-off-by: Alejandro Colomar <alx@kernel.org>
At the first-ever shadow sprint? :) |
What do you mean? |
I have this wish.
Benefits:
Only need to know one language to understand the entire build system: GNU Make. Using autotools means knowing portable make(1) --whatever that is--, and the several autotools languages. Alternatives like cmake(1) would require learning a completely different language. Everyone knows (basic) make(1), and learning the GNU features only takes a little bit of effort, compared to other tools.
One-step builds. This is the greatest feature of it. No more
./configure
(or equivalent, like./auto...
orcmake ...
). make(1) is really in control of the entire build, and so you don't need tomake clean
or re./configure
after certain changes. make(1) simply does whatever needs to be done, as part of dependencies, and skips what needs not be done.I understand some fear of having a handwritten Makefile. Please submit a list of problems you expect of such an implementation, so I make sure that it's free of them. I'll work in a branch, and expect it to take a year or so, so you'll be able to see how it works long before proposing a PR.
Cc: @ikerexxe
Cc: @lslebodn
Cc: @thesamesam
Link: #791
Link: Paul's Rules of Makefiles https://make.mad-scientist.net/papers/rules-of-makefiles/
Link: Metaprogramming Make https://make.mad-scientist.net/category/metaprogramming/
Link: Advanced Auto-Dependency Generation https://make.mad-scientist.net/papers/advanced-auto-dependency-generation/
The text was updated successfully, but these errors were encountered: