llvm generated code ~8-12% slower than gcc for sqlite3

[edwintorok]

Bugzilla Link	2174
Version	unspecified
OS	Linux
Attachments	bzip2 packed sqlite.c, bzip2 packed tst.sql used as input
CC	@asl,@bmrzycki,@lattner,@hiraditya,@sunfishcode

Extended Description

I adapted sqlite3's build system to llvm-test.

When running the test I get results like this:
Program | GCCAS Bytecode LLC compile LLC-BETA compile JIT codegen | GCC CBE LLC LLC-BETA JIT | GCC/CBE GCC/LLC GCC/LLC-BETA LLC/LLC-BETA
sqlite3 | 4.8396 1744588 5.3496 * 3.6930 | 7.08 8.29 8.68 * 13.63 | 0.85 0.82 n/a n/a

To reproduce you can either take the package from here and run 'make TEST=nightly report': http://edwintorok.googlepages.com/sqlite_lemon_llvmtest.tar.gz

Or follow these steps (I am attaching all needed files):
$ llvm-gcc -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=0 -DHAVE_FDATASYNC=0 -DHAVE_USLEEP=0 -DHAVE_LOCALTIME_R=0 -DHAVE_GMTIME_R=0 -DHAVE_READLINE=0 -I. -DSQLITE_THREADSAFE=0 -DSQLITE_OMIT_LOAD_EXTENSION=1 -I/home/edwin/llvm-svn/llvm/projects/llvm-test/MultiSource/Applications/sqlite3 -I/home/edwin/llvm-svn/llvm/projects/llvm-test/MultiSource/Applications/sqlite3 -I/home/edwin/llvm-svn/llvm/include -I/home/edwin/llvm-svn/llvm/projects/llvm-test/include -I../../..//include -I/home/edwin/llvm-svn/llvm/include -D_GNU_SOURCE -D__STDC_LIMIT_MACROS -DNDEBUG -O3 sqlite.c -emit-llvm -c -o sqlite3.bc && opt -std-compile-opts -f sqlite3.bc -o sqlite3_opt.bc && llvm-ld -O2 -native sqlite3_opt.bc -o sqlite3_llvm
$ gcc-4.3 -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=0 -DHAVE_FDATASYNC=0 -DHAVE_USLEEP=0 -DHAVE_LOCALTIME_R=0 -DHAVE_GMTIME_R=0 -DHAVE_READLINE=0 -I. -DSQLITE_THREADSAFE=0 -DSQLITE_OMIT_LOAD_EXTENSION=1 -I/home/edwin/llvm-svn/llvm/projects/llvm-test/MultiSource/Applications/sqlite3 -I/home/edwin/llvm-svn/llvm/projects/llvm-test/MultiSource/Applications/sqlite3 -I/home/edwin/llvm-svn/llvm/include -I/home/edwin/llvm-svn/llvm/projects/llvm-test/include -I../../..//include -I/home/edwin/llvm-svn/llvm/include -D_GNU_SOURCE -D__STDC_LIMIT_MACROS -DNDEBUG -O3 sqlite.c -o sqlite_native

$ time ./sqlite_native :memory: <tst.sql >/dev/null
real 0m1.589s
user 0m1.493s
sys 0m0.010s

$ time ./sqlite_llvm :memory: <tst.sql >/dev/null
real 0m1.970s
user 0m1.890s
sys 0m0.013s

[edwintorok]

When running the test I get results like this:
Program | GCCAS Bytecode LLC compile LLC-BETA compile JIT codegen | GCC
CBE LLC LLC-BETA JIT | GCC/CBE GCC/LLC GCC/LLC-BETA LLC/LLC-BETA
sqlite3 | 4.8396 1744588 5.3496 * 3.6930 | 7.08
8.29 8.68 * 13.63 | 0.85 0.82 n/a n/a

Using SVN r48684 for llvm.
$ uname -a
Linux lightspeed2 2.6.25-rc4-00134-g84c6f60 #​4 Sun Mar 9 19:40:34 EET 2008 x86_64 GNU/Linux

$ gcc-4.3 -v
Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure linux gnu
Thread model: posix
gcc version 4.3.1 20080321 (prerelease) (Debian 4.3.0-2)

$ llvm-gcc -v
Using built-in specs.
Target: x86_64-unknown-linux-gnu
Configured with: ../llvm-gcc4.2/configure --prefix=/home/edwin/llvm-svn/obj42/../install --program-prefix=llvm- --enable-llvm=/home/edwin/llvm-svn/llvm/ --disable-shared --disable-multilib --enable-languages=c,c++ --disable-bootstrap : (reconfigured) ../llvm-gcc4.2/configure --prefix=/home/edwin/llvm-svn/obj42/../install --program-prefix=llvm- --enable-llvm=/home/edwin/llvm-svn/llvm/ --disable-shared --disable-multilib --disable-bootstrap --enable-languages=c,c++ --no-create --no-recursion
Thread model: posix
gcc version 4.2.1 (Based on Apple Inc. build 5555) (LLVM build)

[llvmbot]

Confirmed on Mac OS X. Interestingly, turning off LTO helps (make DISABLE_LTO=1 ...) in this case.

[lattner]

Edwin, can you see if this is still true?

[edwintorok]

Yes with 2.7 it is still ~8-12% slower.

my 2.7pre1 tests:
MultiSource/Applications/sqlite3/sqlite3 | 3.3002 1024048 4.9603 * 3.4162 | 3.7400 * 4.2500 * 8.3600 | n/a 0.88 n/a n/a

If I look at the i686-fnt buildbot:
MultiSource/Applications/sqlite3/sqlite3 | 10.5007 1614784 0.0040 * * | 8.3300 * 9.2700 * * | n/a 0.90 n/a n/a

And the x86_64-fnt buildbot:
MultiSource/Applications/sqlite3/sqlite3 | 4.3243 1619792 0.0040 * * | 4.5600 * 4.9400 * * | n/a 0.92 n/a n/a

[lattner]

Is this still a problem?

[bmrzycki]

I re-tested the original attached files (sqlite.c and tst.sql) with near-tip versions of GCC and LLVM on Linux x86_64.

The compilers:
CC=/tools/cross/llvm-3.9/2016-07-05-23-15-58-7440ba0-master/bin/x86_64-linux-clang
CC=/tools/native/x86_64/gcc-7.0/2016-07-05-18-51-38-039a1d4-master/bin/gcc

The compilation string:
$CC -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=0 -DHAVE_FDATASYNC=0 -DHAVE_USLEEP=0 -DHAVE_LOCALTIME_R=0 -DHAVE_GMTIME_R=0 -DHAVE_READLINE=0 -I. -DSQLITE_THREADSAFE=0 -DSQLITE_OMIT_LOAD_EXTENSION=1 -I/tmp/bmr/foo/MultiSource/Applications/sqlite3 -I/tmp/bmr/test-suite/MultiSource/Applications/sqlite3 -I/tmp/bmr/foo/MultiSource/Applications/lemon -I/tmp/bmr/test-suite/MultiSource/Applications/lemon -I/tmp/bmr/foo/../test-suite/include -I../../..//include -D_GNU_SOURCE -D__STDC_LIMIT_MACROS -DNDEBUG -O3 sqlite.c -o sqlite_bin

I renamed sqlite_bin to sqlite_gcc or sqlite_llvm for each respective compiler.

Next I ran the following shell script:
$ cat go.sh
#!/bin/bash

BIN="./sqlite_$1"

for i in $(seq 1 20); do
/usr/bin/time -f "%U" taskset -c 22 $BIN < tst.sql >/dev/null
done

And the output of 20 runs for each:
$ ./go.sh gcc 2>&1 | sort -n | uniq
0.54
0.55
0.57
1.00

$ ./go.sh llvm 2>&1 | sort -n | uniq
0.53
0.54
0.55
1.04

There looks to be no difference in execution (user) time.

[bwendling]

Looks to be good now. Closing.