Trying to build for alpine docker (musl libc)

[diginc]

We talked about this is chat but I thought I should probably put everything in an issue for full documentation of things attempted/results, and to let others voice opinions. I have been trying to get my docker upgraded and part of that includes getting FTL in my primary docker tag, which is alpine traditionally. Now I could drop alpine support but the size differences is pretty big between alpine and debian and I'm sure I wouldn't be the only one disappointed.

The first approach I decided on trying was just use the musl-gcc wrapper application, which fills in the necessary paths and such for GCC compiling for musl. I didn't get very far in fixing the first compilation error, where it cannot seem to find the qsort's 4th arg type of __compar_fn_t

musl-gcc -c -o obj/main.o main.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -pie -fPIE -g3 -rdynamic -fno-omit-frame-pointer
musl-gcc -c -o obj/structs.o structs.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -pie -fPIE -g3 -rdynamic -fno-omit-frame-pointer
musl-gcc -c -o obj/log.o log.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -pie -fPIE -g3 -rdynamic -fno-omit-frame-pointer
musl-gcc -c -o obj/daemon.o daemon.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -pie -fPIE -g3 -rdynamic -fno-omit-frame-pointer
musl-gcc -c -o obj/parser.o parser.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -pie -fPIE -g3 -rdynamic -fno-omit-frame-pointer
musl-gcc -c -o obj/signals.o signals.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -pie -fPIE -g3 -rdynamic -fno-omit-frame-pointer
musl-gcc -c -o obj/socket.o socket.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -pie -fPIE -g3 -rdynamic -fno-omit-frame-pointer
musl-gcc -c -o obj/request.o request.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -pie -fPIE -g3 -rdynamic -fno-omit-frame-pointer
request.c: In function ‘getTopDomains’:
request.c:293:55: error: ‘__compar_fn_t’ undeclared (first use in this function)
   qsort(temparray, counters.domains, sizeof(int[2]), (__compar_fn_t)cmpdesc);
                                                       ^
request.c:293:55: note: each undeclared identifier is reported only once for each function it appears in
request.c:293:69: error: expected ‘)’ before ‘cmpdesc’
   qsort(temparray, counters.domains, sizeof(int[2]), (__compar_fn_t)cmpdesc);
                                                                     ^
request.c:295:69: error: expected ‘)’ before ‘cmpasc’
   qsort(temparray, counters.domains, sizeof(int[2]), (__compar_fn_t)cmpasc);
                                                                     ^
request.c: In function ‘getTopClients’:
request.c:400:54: error: ‘__compar_fn_t’ undeclared (first use in this function)
  qsort(temparray, counters.clients, sizeof(int[2]), (__compar_fn_t)cmpasc);
                                                      ^
request.c:400:68: error: expected ‘)’ before ‘cmpasc’
  qsort(temparray, counters.clients, sizeof(int[2]), (__compar_fn_t)cmpasc);
                                                                    ^
request.c: In function ‘getForwardDestinations’:
request.c:458:58: error: ‘__compar_fn_t’ undeclared (first use in this function)
  qsort(temparray, counters.forwarded+1, sizeof(int[2]), (__compar_fn_t)cmpdesc);
                                                          ^
request.c:458:72: error: expected ‘)’ before ‘cmpdesc’
  qsort(temparray, counters.forwarded+1, sizeof(int[2]), (__compar_fn_t)cmpdesc);
                                                                        ^
Makefile:44: recipe for target 'obj/request.o' failed
make: *** [obj/request.o] Error 1

I messed with that for a while but C is not my forte and I couldn't seem to google my way past that type error so I next tried a glibc static binary...it seems to actually start up inside of my alpine docker but it does have these 2 warnings, which I'm not sure if that'll cause problems down the road or not...this is more an FYI.

Note: -pie -fPIE are replaced by -static -fPIC to get this build.

root@d5f38ecee56c:/usr/src/myapp# make clean ; make
rm -f obj/*.o pihole-FTL
gcc -c -o obj/main.o main.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -static -fPIC -g3 -rdynamic -fno-omit-frame-pointer
gcc -c -o obj/structs.o structs.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -static -fPIC -g3 -rdynamic -fno-omit-frame-pointer
gcc -c -o obj/log.o log.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -static -fPIC -g3 -rdynamic -fno-omit-frame-pointer
gcc -c -o obj/daemon.o daemon.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -static -fPIC -g3 -rdynamic -fno-omit-frame-pointer
gcc -c -o obj/parser.o parser.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -static -fPIC -g3 -rdynamic -fno-omit-frame-pointer
gcc -c -o obj/signals.o signals.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -static -fPIC -g3 -rdynamic -fno-omit-frame-pointer
gcc -c -o obj/socket.o socket.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -static -fPIC -g3 -rdynamic -fno-omit-frame-pointer
gcc -c -o obj/request.o request.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -static -fPIC -g3 -rdynamic -fno-omit-frame-pointer
gcc -c -o obj/grep.o grep.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -static -fPIC -g3 -rdynamic -fno-omit-frame-pointer
gcc -c -o obj/setupVars.o setupVars.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -static -fPIC -g3 -rdynamic -fno-omit-frame-pointer
gcc -c -o obj/args.o args.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -static -fPIC -g3 -rdynamic -fno-omit-frame-pointer
gcc -c -o obj/flush.o flush.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -static -fPIC -g3 -rdynamic -fno-omit-frame-pointer
gcc -c -o obj/threads.o threads.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -static -fPIC -g3 -rdynamic -fno-omit-frame-pointer
gcc -c -o obj/gc.o gc.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -static -fPIC -g3 -rdynamic -fno-omit-frame-pointer
gcc -c -o obj/config.o config.c -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -static -fPIC -g3 -rdynamic -fno-omit-frame-pointer
gcc -v -I. -Wall -Wextra -Wno-unused-parameter -D_FILE_OFFSET_BITS=64 -fstack-protector -D_FORTIFY_SOURCE=2 -O3 -Wl,-z,relro,-z,now -static -fPIC -g3 -rdynamic -fno-omit-frame-pointer   -o pihole-FTL obj/main.o obj/structs.o obj/log.o obj/daemon.o obj/parser.o obj/signals.o obj/socket.o obj/request.o obj/grep.o obj/setupVars.o obj/args.o obj/flush.o obj/threads.o obj/gc.o obj/config.o -pthread
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc/x86_64-unknown-linux-gnu/4.9.4/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: /usr/src/gcc/configure --disable-multilib --enable-languages=c,c++,fortran,go
Thread model: posix
gcc version 4.9.4 (GCC)
COMPILER_PATH=/usr/local/libexec/gcc/x86_64-unknown-linux-gnu/4.9.4/:/usr/local/libexec/gcc/x86_64-unknown-linux-gnu/4.9.4/:/usr/local/libexec/gcc/x86_64-unknown-linux-gnu/:/usr/local/lib/gcc/x86_64-unknown-linux-gnu/4.9.4/:/usr/local/lib/gcc/x86_64-unknown-linux-gnu/
LIBRARY_PATH=/usr/local/lib/gcc/x86_64-unknown-linux-gnu/4.9.4/:/usr/local/lib/gcc/x86_64-unknown-linux-gnu/4.9.4/../../../../lib64/:/lib/x86_64-linux-gnu/:/lib/../lib64/:/usr/lib/x86_64-linux-gnu/:/usr/local/lib/gcc/x86_64-unknown-linux-gnu/4.9.4/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-I' '.' '-Wall' '-Wextra' '-Wno-unused-parameter' '-D' '_FILE_OFFSET_BITS=64' '-fstack-protector' '-D' '_FORTIFY_SOURCE=2' '-O3' '-static' '-fPIC' '-g3' '-rdynamic' '-fno-omit-frame-pointer' '-o' 'pihole-FTL' '-pthread' '-mtune=generic' '-march=x86-64'
 /usr/local/libexec/gcc/x86_64-unknown-linux-gnu/4.9.4/collect2 -plugin /usr/local/libexec/gcc/x86_64-unknown-linux-gnu/4.9.4/liblto_plugin.so -plugin-opt=/usr/local/libexec/gcc/x86_64-unknown-linux-gnu/4.9.4/lto-wrapper -plugin-opt=-fresolution=/tmp/ccqMEy2b.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-lc -m elf_x86_64 -static -o pihole-FTL /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o /usr/local/lib/gcc/x86_64-unknown-linux-gnu/4.9.4/crtbeginT.o -L/usr/local/lib/gcc/x86_64-unknown-linux-gnu/4.9.4 -L/usr/local/lib/gcc/x86_64-unknown-linux-gnu/4.9.4/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib/gcc/x86_64-unknown-linux-gnu/4.9.4/../../.. -z relro -z now obj/main.o obj/structs.o obj/log.o obj/daemon.o obj/parser.o obj/signals.o obj/socket.o obj/request.o obj/grep.o obj/setupVars.o obj/args.o obj/flush.o obj/threads.o obj/gc.o obj/config.o --start-group -lgcc -lgcc_eh -lpthread -lc --end-group /usr/local/lib/gcc/x86_64-unknown-linux-gnu/4.9.4/crtend.o /usr/lib/x86_64-linux-gnu/crtn.o
obj/daemon.o: In function `getUserName':
/usr/src/myapp/daemon.c:225: warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
obj/parser.o: In function `resolveHostname':
/usr/src/myapp/parser.c:618: warning: Using 'gethostbyaddr' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

working I guess?

bash-4.3# cat /etc/issue
Welcome to Alpine Linux 3.5
Kernel \r on an \m (\l)
bash-4.3# telnet 127.0.0.1 4711
>top-clients
0 2 172.17.0.5
1 2 ::1
---EOM---

>forward-dest
0 10 8.8.4.4
1 2 8.8.8.8
---EOM---

I'll keep you posted as I go down the static glibc road more since I'm in over my head with musl libc compilation which sounds like it may require some rewriting.

[DL6ER]

@diginc How do you use/install musl on Debian/Ubuntu? I need this for testing whether #41 resolves the initial problems you have been seeing.

Concerning the static approach, as you have seen, the getpwuid and gethostbyaddr functions will be undefined. Not sure why FTL is even running for you on a system w/o glibc, but FTL will not be able to

• determine the username under which it is running (uncritical)
• any host names for clients and forward destinations (more critical)

You can see that in your example that your forward-destinations are shown as e.g. 8.8.8.8 while an FTL instance with working name resolution would show google-public-dns-a.google.com here.

Hence, I'm not a very big fan of the static road.

[diginc]

I was using ubuntu but I think debian has the same packages I used. I installed musl-gcc package and it pulled in the required libraries I believe, then invoked musl-gcc instead of gcc

[DL6ER]

Okay, did you do that on Raspberry hardware or does docker (however it does it) not require architecture specific packages? I don't seem to find musl-gcc on Raspbian.

[diginc]

I built it on normal x86 ubuntu xenial.

[DL6ER]

Compilation works with musl-gcc and current development code (after having merged #41 into development).

root@vultr:~/FTL# readelf -a pihole-FTL | grep interpreter
      [Requesting program interpreter: /lib/ld-musl-x86_64.so.1]

[diginc]

Looking good building off development and copying it into my alpine docker.

bash-4.3# telnet 127.0.0.1 4711
>top-clients
0 14 ::1 localhost
---EOM---

>forward-dest
0 11 8.8.4.4 google-public-dns-b.google.com
1 10 8.8.8.8 google-public-dns-a.google.com
2 8 ::1 local
---EOM---

>top-domains
0 8 diginc.us
1 2 analytics.google.com
2 2 google.com
---EOM---

[DL6ER]

PR #44 will add a musl-x86_64 binary to our releases