rdkafka can't be built with native compilers

[shalstea]

We need to build opensource products with the native compiler, e.g. SUN's Workshop tools on Solaris machines not gcc/g++. The code uses gcc specific conventions and the mklove tools assume the same. It would be useful to convert this to autoconf and to make the code compiler agnostic.

Would you be open to take changes for this?

[edenhill]

The dev branch contains a bunch of portability fixes (and makes it easier to fix future portability issues), I think that branch would be a good starting point for porting to a pure Sun toolchain.

l did get librdkafka+mklove to build on Solaris x86, but that might have been with the GNU toolchain.

If you could set me up with an account on a box with the proper toolchain, or provide a virtualbox image for the same, I could do the porting work required.

I'm very reluctant to autoconf, it is just bad technology, mklove should be able to handle it.

[shalstea]

While autoconf is not a tool I particularly like it handles a lot of different cases that mklove currently does not. For example rdkafka doesn't build on our Linux / older gcc versions out of the box. Neither the dev branch or prod branch builds.

I upgraded to a more recent version of gcc and that builds the production branch. However in the dev branch I get: rdendian.h:96:2: error: #error Missing definition for be64toh

mklove doesn't seem to handle setting CFLAGS to values like "-D ABC=123" correctly. For mklove to work with a native compiler one needs to be able to specify and override all the compiler/linker options. I will try and play around some more.

[edenhill]

Yeah, I hear you, but I also think this can be fixed in mklove and librdkafka without too much effort given that I get access to the proper dev environment. Do you know if they are freely available?

[shalstea]

I will try and see. We, unfortunately, don't expose our machine environments for public consumption. We would also want to port to AIX which we still support. At this point I have confirmed that using gcc/g++ builds cleanly on Linux and Solaris for us. AIX is failing and I am investigating.

[edenhill]

Could you provide the exact build env names and version numbers and I'll try to see how close I can get?
There's been previous interest in AIX support, but without a development system it is hard to do the porting, autoconf or not.

[shalstea]

I actually got it to compile on AIX using the gcc compiler. Or at least most of the code. It is failing to build the .so - some gcc atomic ops failing to get resolved. I can give you a pull request with the changes which are not much. I haven't tested it yet but it does compile. Give me some more time to play with it.

[edenhill]

Awesome, I suggest you make your changes to 'dev' branch which has better porting support (and is the future of software!)

[shalstea]

I am working in the dev branch and am now struggling to get the code compiling using gcc 4.8.4 on linux with no modifications to the code. The current roadblocks are:

1. Missing definition for be64toh
2. implicit declaration of function 'strndup'

Can you fix these before I try and merge my changes in?

[edenhill]

dev branch builds fine on my Debian machine with gcc/g++ 4.8.4.
Can you provide the full ./configure and make output?

[shalstea]

I will do a few postings:  So here is the first.  The config output:

using cache file config.cache
checking for OS or distribution... ok (RedHatEnterpriseServer)
checking for C compiler from CC env... ok (cached)
checking for C++ compiler from CXX env... ok (cached)
checking for mbits compiler flag (-m64)... ok (cached)
checking executable ld... ok (cached)
checking executable nm... ok (cached)
checking executable objdump... ok (cached)
checking executable strip... ok (cached)
checking for pkgconfig (by command)... ok (cached)
checking for install (by command)... ok (cached)
checking for PIC (by compile)... ok (cached)
checking for GNU-compatible linker options... ok (cached)
checking for GNU linker-script ld flag... ok (cached)
checking for __atomic_32 (by compile)... ok (cached)
checking for __atomic_64 (by compile)... ok (cached)
checking for socket (by compile)... ok (cached)
parsing version '0x00080600'... ok (0.8.6)
checking for libpthread (by pkg-config)... failed
checking for libpthread (by compile)... ok
checking for zlib (by pkg-config)... failed
checking for zlib (by compile)... ok
checking for librt (by pkg-config)... failed
checking for librt (by compile)... ok
checking for nm (by env NM)... ok (cached)
checking for perl (by command)... ok (cached)
Generated rdkafka.pc
Generated Makefile.config
Generated config.h

Configuration summary:
  prefix                   /bb/mbig/mbig1016/librdkafka/linux
  ARCH                     x86_64
  CPU                      generic
  MBITS                    64
  MKL_DISTRO               RedHatEnterpriseServer
  CC                       /opt/bb/lib/gcc-4.8/bin/gcc
  CXX                      /opt/bb/lib/gcc-4.8/bin/g++
  CPPFLAGS                 -m64 -g -O2 -fPIC -Wall -Werror -Wfloat-equal -Wpointer-arith
  LDFLAGS                  -m64
  LD                       ld
  NM                       nm
  OBJDUMP                  objdump
  STRIP                    strip
  PKG_CONFIG               pkg-config
  INSTALL                  install
  LIB_LDFLAGS              -shared -Wl,-soname,$(LIBFILENAME)
  LDFLAG_LINKERSCRIPT      -Wl,--version-script=
  RDKAFKA_VERSION_STR      0.8.6
  LIBS                     -lpthread -lz -lrt
  CXXFLAGS                 -Wno-non-virtual-dtor
  SYMDUMPER                $(NM) -D
  exec_prefix              /bb/mbig/mbig1016/librdkafka/linux
  bindir                   /bb/mbig/mbig1016/librdkafka/linux/bin
  sbindir                  /bb/mbig/mbig1016/librdkafka/linux/sbin
  libexecdir               /bb/mbig/mbig1016/librdkafka/linux/libexec
  datadir                  /bb/mbig/mbig1016/librdkafka/linux/share
  sysconfdir               /bb/mbig/mbig1016/librdkafka/linux/etc
  sharedstatedir           /bb/mbig/mbig1016/librdkafka/linux/com
  localstatedir            /bb/mbig/mbig1016/librdkafka/linux/var
  libdir                   /bb/mbig/mbig1016/librdkafka/linux/lib
  includedir               /bb/mbig/mbig1016/librdkafka/linux/include
  infodir                  /bb/mbig/mbig1016/librdkafka/linux/info
  mandir                   /bb/mbig/mbig1016/librdkafka/linux/man
Generated config.cache

Now type 'make' to build

[shalstea]

uname -a output:
Linux nylxdev1 2.6.18-308.4.1.el5 #1 SMP Wed Mar 28 01:54:56 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

[shalstea]

Make output:
 make 
make[1]: Entering directory '/bb/mbigd/mbig1016/librdkafka/dev/librdkafka/src'
/opt/bb/lib/gcc-4.8/bin/gcc -MD -MP -m64 -g -O2 -fPIC -Wall -Werror -Wfloat-equal -Wpointer-arith  -c rdkafka.c -o rdkafka.o
In file included from rdkafka_proto.h:32:0,
                 from rdkafka_int.h:79,
                 from rdkafka.c:37:
rdendian.h:95:2: error: #error Missing definition for be64toh
 #error Missing definition for be64toh
  ^
../mklove/Makefile.base:71: recipe for target 'rdkafka.o' failed
make[1]: *** [rdkafka.o] Error 1
make[1]: Leaving directory '/bb/mbigd/mbig1016/librdkafka/dev/librdkafka/src'
Makefile:14: recipe for target 'libs' failed
make: *** [libs] Error 2

[shalstea]

If you want to suggest something to try or send me code changes I can give it a try.

[edenhill]

Can you look in your /usr/include/endian.h and see what the prereqs for be64toh() is?
On Debian/Ubuntu it depends on __USE_MISC which is defined by _DEFAULT_SOURCE in features.h

Do you know if there are there vbox or vagrant images available for something similar to your system?

[edenhill]

I can probably do a CentOS image, but which version?

[shalstea]

So our older version (which I am testing on) doesn't even have any of the be* macro(s) defined. On a more recent version they exist. So you will need a fallback case. This did compile in your prod branch.

[edenhill]

I've reproduced this issue on CentOS 5:
https://build.opensuse.org/package/live_build_log/home:edenhill/librdkafka/CentOS_CentOS-5/x86_64

I'll grab an image and fix it.

[shalstea]

Cool. When I hacked around that problem to see what else would come up I quickly came across strndup missing as well. Once you push changes back let me know and I will continue trying on my end.

As a FYI to get this to build on AIX required changes to 8 files (small changes).

[shalstea]

Do you have a rough ETA on when you will push changes back to dev with the fixes for older Linux envs?

[shalstea]

Looking at the dev branch - native compiler issues:

1. mklove/modules/configure.base - line 888 -Wall -Werror is gcc specific
2. mklove/modules/configure.good_cflags - all these settings are gcc specific
3. src/rd.h attribute and __builtin_constant are gcc specific so the lines should be changed to be:

#ifdef GNUC
#define RD_UNUSED attribute((unused))
#define RD_PACKED attribute((packed))
#define RD_IS_CONSTANT(p) __builtin_constant_p((p))
#else
#define RD_UNUSED
#define RD_PACKED
#define RD_IS_CONSTANT(p)
#endif

1. line 217 of src/rdkafka.h has attribute((unused)) this needs to be removed or ifdef'ed as this is a gcc idiom
2. mklove/modules/configure.atomics the __ATOMIC_SEQ_CST is gcc specific.

[shalstea]

I am not sure how to properly change the mklove code to say if the compiler is gcc then do this else if the compiler xlc (AIX) do that, else if SUN's cc do xxx so that is why I posted the above. After I hacked around those issues I found the other stuff.

[shalstea]

Sorry for all the updates. I think this issue is now devolved into two separate problems:

1. The library doesn't compile cleanly on AIX (with gcc/g++) - I will put in a separate issue for that.
2. The mklove tools and code don't compile cleanly with native compilers (which is this issue).

My suggested fix for this issue is that when running configure a new option be provided to use the specified compiler/linker values and not intermix them with defaults. Part of the problem is that you are providing some defaults which are appended to by user specified values and the defaults are GCC specific. If we have the option to not use any defaults but to take what is specified then that might make it easier to implement.

The downside to this approach is that everyone who wants to use a native compiler has to know all their options well. And you may need to add a lot more options to configure to maximize the flexibility you need.

The alternative is to detect the compiler provided and set appropriate options internally (like you do for gcc). I can help with the AIX and SUN compilers. Let me know what approach you want to take and I will try and help out.

[edenhill]

Thanks for all the effort you are putting into this, very much appreciated.
The CentOS (or rather the old glibc endian.h) fix is on the way.

[shalstea]

Cool. Note that issue 328 is now for the AIX port. This one I will keep for the native compilers.

[edenhill]

I think the alternative approach is best:
provide proper flags per compiler (there aren't that many and the per-compiler modifications should be straight forward when we get this in place).
I.e., we provide an mkl var named "CC_TYPE", "CC_COMPAT", or something similar, which lets various parts of the mklove code assign different values depending on if it is set to "gcc", "clang" (which is gcc compatible for the most part), "sunsomething", "xlc", etc.

[edenhill]

be64toh and strndup are now fixed on the dev branch, can you give it a go?

[edenhill]

I merged master into dev as well.

[shalstea]

I picked up the dev branch and this builds cleanly on my gcc/linux version. However on sun (using gcc) it fails:
queue.h:38:23: fatal error: sys/cdefs.h: No such file or directory
#include <sys/cdefs.h>

If I ifdef this out for sun I get more problems, e.g. struct timespec undefined, strdup undeclared etc. There seem to be a lot of mismatched #include statements for Sun.

[shalstea]

Interesting - the master branch builds fine on both SunOS and Linux. So do you want me to merge my AIX changes into that or do you want to continue working on the dev branch and have me merge it there when you are done?

[shalstea]

No we can work off the dev branch. What would be helpful it to email me when you have made significant dev changes. I can pull and do test builds on various platforms. If you want to avoid cluttering this issue with a ton of more comments you can email me directly as shalstead@bloomberg.net

[edenhill]

Will do.
Are you planning on using the consumer in librdkafka too?

[shalstea]

Yes indeed. For the consumer we only plan to use it on Linux. For the publisher I need windows, aix, solaris and linux.

[edenhill]

Okay.

The consumer implementation will get an overhaul on the dev branch to fix a number of issues (mainly offset storage bugs) and get ready for the broker based consumer groups that will be available in Kafka 0.9.

I see that collectdwin is written in C#, how are you planning on integrating librdkafka in that project?

[shalstea]

I am not 100% sure since I am not a C# developer. C# can call C and we will just have to build the glue (dll ?) to do it. Collectdwin doesn't do dlopen / dlsym like the Unix variant.

[edenhill]

Sounds like proper C# bindings for librdkafka is what you need.

[shalstea]

Nice to have but not required we need to wrap it regardless to match the collectdwin plugin interface. Low priority. AIX, Windows and Native compiler support are higher priority from my perspective.

As an aside I need to disable the linker script support (WITH_LDS) at ./configure time in your master branch. Do you have an option to do this? On my solaris machine this is not supported.

[edenhill]

WITH_LDS: if the configure check succeeds but the actual use in the makefile fails I think the configure check should be fixed so it properly fails and disables WITH_LDS.
Will look into it.

[shalstea]

I will be on vacation for a bit and back on Aug 12th. It would be nice if there you could shoot to release something by Aug 17th. If not a stable dev version with support for windows, etc by then could be a fallback.

Also from a build perspective it might be useful to have separate targets for just the libs, examples and tests and then a build all to do everything. When we do packaging with only want to build the libs. Everything should of course build on all platforms and run correctly with appropriate config changes.

[shalstea]

How goes the next release (current dev branch) with AIX, Win32 and hopefully native compiler support? Do you have an ETA for it yet?

[edenhill]

Welcome back from vacation! I'm currently on vacation myself 'til monday.

Would rolling your own release off dev work for the time being?
I'm looking to merge dev down into master within a two week period.
After that I want some flight time on master before rolling a new release.

[shalstea]

If you are on vacation you shouldn't be so active on github ;-)

I presume that you believe your dev branch to be stable at this point. I will get a copy of this and try to do builds (using gcc) on our platforms (linux, aix and solaris) and see how that goes.

[shalstea]

Linux ============================

make tests: warning should be cleaned up. Note that everything else built cleanly and the tests ran and completed correctly.

../src/rd.h: In function 'rd_strndup':
../src/rd.h:110:2: warning: implicit declaration of function 'strndup' [-Wimplicit-function-declaration]
  char *n = strndup(s, len);

Solaris ============================

make failed building the shared library

/opt/bb/lib/gcc-4.8/bin/gcc -m64 -shared -Wl,-soname,librdkafka.so.1 -Wl,--version-script=librdkafka.lds rdkafka.o rdkafka_broker.o rdkafka_msg.o rdkafka_topic.o rdkafka_defaultconf.o rdkafka_timer.o rdkafka_offset.o rdkafka_transport.o rdcrc32.o rdgz.o rdaddr.o rdrand.o tinycthread.o rdlog.o snappy.o -o librdkafka.so.1 -lpthread -lz -lrt
ld: fatal: unrecognized option '--'
ld: fatal: use the -z help option for usage information
collect2: error: ld returned 1 exit status
../mklove/Makefile.base:79: recipe for target 'librdkafka.so.1' failed

However this command worked:

/opt/bb/lib/gcc-4.8/bin/gcc -m64 -shared -Wl,-soname,librdkafka.so.1 rdkafka.o rdkafka_broker.o rdkafka_msg.o rdkafka_topic.o rdkafka_defaultconf.o rdkafka_timer.o rdkafka_offset.o rdkafka_transport.o rdcrc32.o rdgz.o rdaddr.o rdrand.o tinycthread.o rdlog.o snappy.o -o librdkafka.so.1 -lpthread -lz -lrt

The examples did not build:

make[1]: Leaving directory '/bb/mbigd/mbig1016/librdkafka_dev/librdkafka/src-cpp'
make -C examples
make[1]: Entering directory '/bb/mbigd/mbig1016/librdkafka_dev/librdkafka/examples'
/opt/bb/lib/gcc-4.8/bin/gcc -m64 -g -O2 -fPIC -Wall -Wfloat-equal -Wpointer-arith -D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT -D__EXTENSIONS__ -I../src rdkafka_example.c -o rdkafka_example -m64 \
        ../src/librdkafka.a -lpthread -lz -lrt
Undefined                       first referenced
 symbol                             in file
socket                              ../src/librdkafka.a(rdkafka_broker.o)
getnameinfo                         ../src/librdkafka.a(rdaddr.o)
recvmsg                             ../src/librdkafka.a(rdkafka_transport.o)
sendmsg                             ../src/librdkafka.a(rdkafka_transport.o)
setsockopt                          ../src/librdkafka.a(rdkafka_transport.o)
connect                             ../src/librdkafka.a(rdkafka_transport.o)
getaddrinfo                         ../src/librdkafka.a(rdaddr.o)
freeaddrinfo                        ../src/librdkafka.a(rdaddr.o)
gai_strerror                        ../src/librdkafka.a(rdaddr.o)
ld: fatal: symbol referencing errors. No output written to rdkafka_example
collect2: error: ld returned 1 exit status

This can be fixed by adding -lsocket -lnsl to the linkline

The tests did not build. The compilation of test.c did not honor the -m64 option and linking fails:

/opt/bb/lib/gcc-4.8/bin/gcc -m64 -g -O2 -fPIC -Wall -Wfloat-equal -Wpointer-arith -D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT -D__EXTENSIONS__ -m64 -L../src -L../src-cpp 0002-unkpart.o 0011-produce_batch.o 0004-conf.o 0005-order.o 0007-autotopic.o 0013-null-msgs.o 0001-multiobj.o 0008-reqacks.o 0010-enforcereqacks.o 0014-reconsume-191.o 0006-symbols.o 0012-produce_consume.o 0003-msgmaxsize.o test.o -o merged -lrdkafka -lpthread -lz -lrt
ld: fatal: file test.o: wrong ELF class: ELFCLASS32
ld: fatal: file processing errors. No output written to merged
collect2: error: ld returned 1 exit status

Then the it failed to link due to the same missing -lsocket -lnsl

Finally once it linked it crashed on Solaris:

LD_LIBRARY_PATH=../src:../src-cpp ./run-test.sh merged
############## merged ################
### Running test merged in bare mode ###
Tests to run: all
================= Run test 0001_multiobj =================
Creating and destroying 100 kafka instances
Using topic "rdkafkatest_generic"
./run-test.sh: line 43:  1823 Bus Error               (core dumped) ./$TEST
###
### Test merged in bare mode FAILED! ###
###
Makefile:22: recipe for target 'all' failed
make: *** [all] Error 1

stack of crash

t@3 (l@3) signal BUS (invalid address alignment) in rd_kafka_broker_metadata_reply at 0x100027e9c
0x0000000100027e9c: rd_kafka_broker_metadata_reply+0x05c0:      st       %g1, [%g2 + 24]
(dbx) where           
current thread: t@3
=>[1] rd_kafka_broker_metadata_reply(0x1001428d0, 0x10014cdd0, 0x88, 0x4f05, 0x100143130, 0xffffffff7c9fb7ac), at 0x100027e9c 
  [2] rd_kafka_recv(0x1001428d0, 0x74f87, 0xb1cc5, 0x1001425f0, 0x1001431f0, 0x100143360), at 0x1000234dc 
  [3] rd_kafka_broker_io_serve(0x1001428d0, 0x1c866ea8, 0x3b9aca00, 0xb1cc5, 0x296597a357200, 0x1), at 0x100023dc0 
  [4] rd_kafka_broker_thread_main(0x1001428d0, 0xff000000, 0xffffffff7d384cc0, 0xa98fc448ac, 0xf4240, 0xa98fd38245), at 0x100024968 
  [5] _thrd_wrapper_function(0x100141620, 0x0, 0x0, 0x1000306b8, 0x1001428d0, 0x100024658), at 0x1000306cc 

I didn't try and do windows or AIX due to the above errors.

[shalstea]

I decided to give IBM a try.

• The code compiled but the library archiving commands are wrong.
• I also saw: Symbol visibility nm: librdkafka.so.1: 0654-207 The sort process was stopped prematurely.
• The tests built but failed and asserted: bootstrap: Receive failed: Bad file number

[shalstea]

I am trying to debug the Solaris (on Sparc) problem. Turning off optimization I can see that it is crashing in at reading through the topics - on the first one. It looks like the messsage is good.

0x0000000100029db8 in rd_kafka_metadata_handle (rkb=0x10015ec90, rko=0x10015f4f0, buf=0x1001641f0 "", size=20365) at rdkafka_broker.c:878
878                     _READ_I16A(md->topics[i].err);
(gdb) p md
$2 = (struct rd_kafka_metadata *) 0x100169190
(gdb) p *md
$3 = {broker_cnt = 3, brokers = 0x1001691f0, topic_cnt = 17, topics = 0x1001692a1, orig_broker_id = -1, orig_broker_name = 0x1001691c0 "iny1-cas-r1n06.inf.bloomberg.com:9092/bootstrap"}

Any suggestions? I may need to move the macro code inline to debug further.

[shalstea]

The problem appears to be when the assignment / write is done to the topics[0] fields inside the for loop on line 876 of rdkafka_broker.c (dev branch). I tried assigning 0 to both the err and partition_cnt fields and both crash. Kind of at a loss. The MSH_ALLOC calls seem to be returning reasonable values and I can print out the info in a debugger.

[edenhill]

I think the problem is with unaligned access, typical sparc thing. I'll push a fix

[shalstea]

Let me know when you push and I will give it a try. It would be useful to also:

1. For Solaris add -lsocket -lnsl to the link lines for your examples and tests
2. Modify the makefile for the tests to insure that the test.c is honoring the -m64 option if set.

[shalstea]

So - also please fix the library archiving for Solaris and AIX - it needs to be the following command
/opt/bb/lib/gcc-4.8/bin/gcc -m64 -shared -Wl,-soname,librdkafka.so.1 rdkafka.o rdkafka_broker.o rdkafka_msg.o rdkafka_topic.o rdkafka_defaultconf.o rdkafka_timer.o rdkafka_offset.o rdkafka_transport.o rdcrc32.o rdgz.o rdaddr.o rdrand.o tinycthread.o rdlog.o snappy.o -o librdkafka.so.1 -lpthread -lz -lrt

[shalstea]

The problem on Solaris (SPARC) is that the memory returned from the _MSH_ALLOC() macro is not 64 byle aligned. In rdkafka_broker.c (in dev branch) line 727-728 to be:

#define _MSH_ALLOC(PTR,LEN) do {
int __LEN = (LEN + (LEN % 64)); \

The first two tests then pass. The third test is failing - I am looking into it:
%3|1440437572.532|FAIL|rdkafka#producer-101|:9092/11: Send failed: Message too long

[shalstea]

When I ifdef out the 3rd test, the 5th test 0005_order fails with message too long as well:
%3|1440439253.318|FAIL|rdkafka#producer-103| :9092/11: Send failed: Message too long

So something more fundamental with the message is a problem.

[shalstea]

Magnus - I think you can go ahead and close this out once you have merged the dev branch changes you have made so far to the main branch. I will look to do some retesting and put in new issues, if I find any, off of that.

Do you have an ETA on when you will do your next release ?

[edenhill]

I have rewritten all the buffer handling to be alignment safe - no more crashes on Sparc, no more soft-alignment interrupts on ARM.
This will go in the oct15 branch that I will merge with master some time early october, a release is then imminent.
It'd be really helpful if you could give it a spin at some point in the future (we can sync up on a good such point).

[shalstea]

OK - let me know when you are ready... and from which branch to work off of.

[edenhill]

awesome, will do

[edenhill]

@shalstea Not sure if this is still on your radar, but could you give latest master a try?