vagrant ubuntu 14 build error

[pankajtakawale]

I'm following "Building on Linux" using vagrant
http://osquery.readthedocs.io/en/latest/development/building/

The 'make' command is giving following linking error.
[ 81%] Linking CXX executable osquery_tables_tests
/usr/local/osquery/bin/ld.lld: error: failed to open osquery_tables_tests: Invalid argument
clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [osquery/osquery_tables_tests] Error 1
make[2]: *** [osquery/CMakeFiles/osquery_tables_tests.dir/all] Error 2
make[1]: *** [all] Error 2
make: *** [all] Error 2

[obelisk]

Ah this is an issue we've been having since we rebuilt the world. We're still investigating but it has to do with it being a mounted shared folder.

As a stop gap, if you copy the vagrant folder somewhere else and build, it will work

[theopolis]

At the end of all the compiling it looks like:

vagrant@ubuntu1704-64:/vagrant/build/zesty/osquery$ "/usr/local/osquery/bin/ld.lld" -pie -export-dynamic -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -export-dynamic -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o example_extension.ext /usr/local/osquery/opt/glibc/lib/Scrt1.o /usr/local/osquery/opt/glibc/lib/crti.o /usr/local/osquery/opt/gcc/lib64/gcc/x86_64-unknown-linux-gnu/5.3.0/crtbeginS.o -L/usr/local/osquery/legacy/lib -L/usr/local/osquery/lib -L/usr/local/osquery/opt/glibc/lib -L/usr/local/osquery/opt/gcc/lib64/gcc/x86_64-unknown-linux-gnu/5.3.0 -L/usr/local/osquery/opt/gcc/lib64/gcc/x86_64-unknown-linux-gnu/5.3.0/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/local/osquery/opt/gcc/lib64/gcc/x86_64-unknown-linux-gnu/5.3.0/../../.. -L/usr/local/osquery/Cellar/llvm/4.0.0_101/bin/../lib -L/lib -L/usr/lib -plugin /usr/local/osquery/Cellar/llvm/4.0.0_101/bin/../lib/LLVMgold.so -plugin-opt=mcpu=x86-64 -plugin-opt=thinlto CMakeFiles/example_extension.dir/examples/example_extension.cpp.o -whole-archive libosquery.a -no-whole-archive -lpthread -lz -Bstatic -lgflags -lthrift -llz4 -Bdynamic -ldl -Bstatic -lboost_system-mt -lboost_filesystem-mt -lboost_thread-mt -lboost_context-mt -Bdynamic -lrt -lc -Bstatic -lglog -llzma -lbz2 -Bdynamic -zrelro -znow -lgcc_s --build-id -v -Bstatic -lstdc++ -Bdynamic -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/local/osquery/opt/gcc/lib64/gcc/x86_64-unknown-linux-gnu/5.3.0/crtendS.o /usr/local/osquery/opt/glibc/lib/crtn.o
LLD 4.0.0
/usr/local/osquery/bin/ld.lld: error: failed to open example_extension.ext: Invalid argument
vagrant@ubuntu1704-64:/vagrant/build/zesty/osquery$ "/usr/local/osquery/bin/ld.lld" -pie -export-dynamic -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -export-dynamic -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o /tmp/example_extension.ext /usr/local/osquery/opt/glibc/lib/Scrt1.o /usr/local/osquery/opt/glibc/lib/crti.o /usr/local/osquery/opt/gcc/lib64/gcc/x86_64-unknown-linux-gnu/5.3.0/crtbeginS.o -L/usr/local/osquery/legacy/lib -L/usr/local/osquery/lib -L/usr/local/osquery/opt/glibc/lib -L/usr/local/osquery/opt/gcc/lib64/gcc/x86_64-unknown-linux-gnu/5.3.0 -L/usr/local/osquery/opt/gcc/lib64/gcc/x86_64-unknown-linux-gnu/5.3.0/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/local/osquery/opt/gcc/lib64/gcc/x86_64-unknown-linux-gnu/5.3.0/../../.. -L/usr/local/osquery/Cellar/llvm/4.0.0_101/bin/../lib -L/lib -L/usr/lib -plugin /usr/local/osquery/Cellar/llvm/4.0.0_101/bin/../lib/LLVMgold.so -plugin-opt=mcpu=x86-64 -plugin-opt=thinlto CMakeFiles/example_extension.dir/examples/example_extension.cpp.o -whole-archive libosquery.a -no-whole-archive -lpthread -lz -Bstatic -lgflags -lthrift -llz4 -Bdynamic -ldl -Bstatic -lboost_system-mt -lboost_filesystem-mt -lboost_thread-mt -lboost_context-mt -Bdynamic -lrt -lc -Bstatic -lglog -llzma -lbz2 -Bdynamic -zrelro -znow -lgcc_s --build-id -v -Bstatic -lstdc++ -Bdynamic -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/local/osquery/opt/gcc/lib64/gcc/x86_64-unknown-linux-gnu/5.3.0/crtendS.o /usr/local/osquery/opt/glibc/lib/crtn.o
LLD 4.0.0

If the output path for LLD is within the /vagrant mount, this fails. If it's /tmp/ it works.

[theopolis]

Using strace to compare each run:

/tmp/example_extension.ext:

access("/tmp/example_extension.ext", F_OK) = -1 ENOENT (No such file or directory)
stat("/tmp/example_extension.ext", 0x7ffcfe42c218) = -1 ENOENT (No such file or directory)
lstat("/tmp/example_extension.ext", 0x7ffcfe42c218) = -1 ENOENT (No such file or directory)
open("/dev/urandom", O_RDONLY)          = 3
read(3, "p\267\372\201", 4)             = 4
close(3)                                = 0
open("/tmp/example_extension.ext.tmpbfc8bc3", O_RDWR|O_CREAT|O_EXCL|O_TRUNC, 0777) = 3
fallocate(3, 0, 0, 7853184)             = 0
mmap(NULL, 7853184, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x7fe0c866b000
close(3)                                = 0
munmap(0x7fe0c866b000, 7853184)         = 0
rename("/tmp/example_extension.ext.tmpbfc8bc3", "/tmp/example_extension.ext") = 0
write(1, "LLD 4.0.0\n", 10LLD 4.0.0
)             = 10
exit_group(0)                           = ?
+++ exited with 0 +++

example_extension.ext:

access("example_extension.ext", F_OK)   = -1 ENOENT (No such file or directory)
stat("example_extension.ext", 0x7ffd653e77e8) = -1 ENOENT (No such file or directory)
lstat("example_extension.ext", 0x7ffd653e77e8) = -1 ENOENT (No such file or directory)
open("/dev/urandom", O_RDONLY)          = 3
read(3, "\316T\271\352", 4)             = 4
close(3)                                = 0
open("example_extension.ext.tmp2368022", O_RDWR|O_CREAT|O_EXCL|O_TRUNC, 0777) = 3
fallocate(3, 0, 0, 7853184)             = -1 EOPNOTSUPP (Operation not supported)
fstat(3, {st_mode=S_IFREG|0775, st_size=0, ...}) = 0
fstatfs(3, {f_type=NFS_SUPER_MAGIC, f_bsize=4096, f_blocks=55805335, f_bfree=5651642, f_bavail=5651642, f_files=1000, f_ffree=1000, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_NODEV|ST_RELATIME}) = 0
mmap(NULL, 7853184, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = -1 EINVAL (Invalid argument)
close(3)                                = 0
write(2, "/usr/local/osquery/bin/ld.lld", 29/usr/local/osquery/bin/ld.lld) = 29
write(2, ": ", 2: )                       = 2
write(2, "error: ", 7error: )                  = 7
write(2, "failed to open ", 15failed to open )         = 15
write(2, "example_extension.ext", 21example_extension.ext)   = 21
write(2, ": ", 2: )                       = 2
write(2, "Invalid argument", 16Invalid argument)        = 16
write(2, "\n", 1
)                       = 1
madvise(0x7f4ad88ce000, 2162688, MADV_DONTNEED) = 0
madvise(0x7f4ad0d5c000, 1331200, MADV_DONTNEED) = 0
madvise(0x7f4ad0d30000, 180224, MADV_DONTNEED) = 0
munmap(0x7f4ade3f4000, 790528)          = 0
munmap(0x7f4ae2219000, 122280)          = 0
munmap(0x7f4ae033e000, 9690690)         = 0
munmap(0x7f4ae21f6000, 142000)          = 0
munmap(0x7f4ae21e1000, 84008)           = 0
munmap(0x7f4ae21ac000, 213400)          = 0
munmap(0x7f4ae01c1000, 1560382)         = 0
munmap(0x7f4ae219b000, 67552)           = 0
munmap(0x7f4ae2196000, 19472)           = 0
munmap(0x7f4ae2190000, 22214)           = 0
munmap(0x7f4ae2168000, 160912)          = 0
munmap(0x7f4ae211b000, 313734)          = 0
munmap(0x7f4ae210c000, 58992)           = 0
munmap(0x7f4adfff4000, 1886136)         = 0
munmap(0x7f4ae2105000, 24886)           = 0
munmap(0x7f4ae20de000, 156048)          = 0
munmap(0x7f4ae20ac000, 203930)          = 0
munmap(0x7f4ae2067000, 278900)          = 0
munmap(0x7f4adffbb000, 231578)          = 0
munmap(0x7f4adff5b000, 389248)          = 0
munmap(0x7f4adf1f6000, 14043592)        = 0
munmap(0x7f4adf165000, 591680)          = 0
munmap(0x7f4adf105000, 389248)          = 0
munmap(0x7f4adec79000, 4767666)         = 0
munmap(0x7f4adeaac000, 1886136)         = 0
munmap(0x7f4ae2060000, 24886)           = 0
munmap(0x7f4adea85000, 156048)          = 0
munmap(0x7f4adea25000, 389248)          = 0
munmap(0x7f4ade599000, 4767666)         = 0
madvise(0x7f4ad0aba000, 2580480, MADV_DONTNEED) = 0
madvise(0x7f4ad84f1000, 4050944, MADV_DONTNEED) = 0
madvise(0x7f4ad84c3000, 188416, MADV_DONTNEED) = 0
madvise(0x7f4ad84ac000, 94208, MADV_DONTNEED) = 0
madvise(0x7f4ad0021000, 11112448, MADV_DONTNEED) = 0
write(1, "LLD 4.0.0\n", 10LLD 4.0.0
)             = 10
rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0
close(1)                                = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
madvise(0x7f4ad8021000, 4763648, MADV_DONTNEED) = 0
brk(0x474f000)                          = 0x474f000
exit_group(1)                           = ?
+++ exited with 1 +++

The error when the output path is on a vboxsf-mounted directory:

fallocate(3, 0, 0, 7853184)             = -1 EOPNOTSUPP (Operation not supported)
fstat(3, {st_mode=S_IFREG|0775, st_size=0, ...}) = 0
fstatfs(3, {f_type=NFS_SUPER_MAGIC, f_bsize=4096, f_blocks=55805335, f_bfree=5651642, f_bavail=5651642, f_files=1000, f_ffree=1000, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_NODEV|ST_RELATIME}) = 0
mmap(NULL, 7853184, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = -1 EINVAL (Invalid argument)

[theopolis]

We need this patch: https://reviews.llvm.org/rL302263

[theopolis]

Applying that patch gives me:

open("example_extension.ext.tmp2ce0bfa", O_RDWR|O_CREAT|O_EXCL|O_TRUNC, 0777) = 3
fallocate(3, 0, 0, 7853184)             = -1 EOPNOTSUPP (Operation not supported)
fstat(3, {st_mode=S_IFREG|0775, st_size=0, ...}) = 0
fstatfs(3, {f_type=NFS_SUPER_MAGIC, f_bsize=4096, f_blocks=55805335, f_bfree=4879025, f_bavail=4879025, f_files=1000, f_ffree=1000, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_NODEV|ST_RELATIME}) = 0
ftruncate(3, 7853184)                   = 0
mmap(NULL, 7853184, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = -1 EINVAL (Invalid argument)

[theopolis]

vboxsf cannot handle MAP_SHARED: https://www.virtualbox.org/ticket/819
Here is the same issue described on SO: https://stackoverflow.com/questions/18420473/invalid-argument-for-read-write-mmap

[theopolis]

I think the TL;DR is this would go away with the release of LLVM 4.0.1, but trying to pull in https://reviews.llvm.org/D30010 too. I checked and VBOX is reporting 0x6969 (NFS) as the filesystem type.

[theopolis]

So, no, it will not go away in 4.0.1, there are no changes effecting the code-path to mmap. Any failure in mmap will fail the tool. We can change the flag to MAP_PRIVATE as a stop-gap. The alternative is some complicated logic to build in a temporary directory. That will be a bag of worms as we'll need to refactor the "out from under you" checks in extensions and worker-forking.