Implement Lua bindings (design & specification)

[thelamb]

Introduction

I'd like to start a discussion on implementing Lua bindings for rdkafka. I will keep this first post up-to-date with the design choices that we make in the comments. So it's just a placeholder of all current decisions.

I will take care of the initial implementation, and maintenance of the bindings in the future if needed (to be discussed, see below).

At this point I'm not too experienced with rdkafka or Kafka itself, but I'd like to start documenting the design for the Lua bindings. As we go along I'll figure out the details.

Anyone with ideas/comments should feel free to participate. If I missed any major topics below please suggest to add them. When replying to a specific topic, please include the same title in your comment.

What is out there already?

• https://github.com/doujiang24/lua-resty-kafka

Example usage of lua-resty-kafka:

local producer = require "resty.kafka.producer"
local p = producer:new(broker_list, producer_config)
p:send(topic, key, message)

List of major choices we have to make

• Code location (separate repository or in edenhill/librdkafka)
• What to expose? The raw C functions, or CPP-like classes
• How to bind between Lua and C
  • Writing 'raw' bindings using the Lua API, or a helper library like OOLua (http://oolua.org/)
• Unit tests

Code location

There will be a separate Github repo

What to expose

TBD.

How to bind between Lua and C

TBD.

Unit tests

(TBD. I haven't really looked into the rdkafka tests, but I suppose we can replicate most of them in Lua).

[thelamb]

Code location

Many libraries choose to put bindings for other languages in a separate repository. To me, it makes more sense to have the bindings part of the edenhill/librdkafka sources.

I suppose the main thing to keep into account here is who will be the maintainer of the bindings and how we will take care of a change in rdkafka that breaks the Lua bindings code (and CI would fail).
For maintenance (e.g. responding to bug reports) I think there is no problem with having the code in the edenhill repository. If I maintain the bindings I'll just watch the rdkafka issues and send a pull request for fixes.

Breaking changes is a different problem as this puts a burden on you, @edenhill. So I'll leave it up to you to decide whether we put the bindings in the rdkafka repository or if we maintain a separate one.

[thelamb]

How to bind between Lua and C

The Lua API is not too friendly. In other projects where I interface between C++ and Lua I use 'OOLua'.
There are quite a few 'binding generators' out there but I found that OOLua is the only one that suited my needs. A useful comparison is here: http://realmensch.org/blog/fun-lua-bindings but keep in mind that this compares (and links to) an old version of OOLua. I prefer to use OOLua 2, even though it is still in beta. OOLua is by far the lightest (no dependency on other libraries like Boost, no extra compilation steps required).

Obviously the downside is that we introduce a dependency, which is far from ideal.
I am OK with writing the bindings without OOLua.

[thelamb]

What to expose

In my opinion it makes sense to expose the CPP interface to Lua, rather than the 'raw' C functions.
This way the interface would look similar to the lua-resty-kafka

[edenhill]

Lua bindings sounds great and is a welcomed addition to the set of client languages!

How to bind between Lua and C

While the C API is stable (and guaranteed to be both API and ABI stable) the C++ is still experimental and may thus still be modified if interface problems arise - this is a good thing in this context since it avoids workarounds in the Lua binding code if such problems are found. Having said that the C++ API has been out in the open for almost a year now without too many complaints - so it shouldn't be changed too lightheartedly.

Code location

You do have good points, but initially I'd probably want to see this as an external repository for two reasons:

• you will have quicker turn-around without having to wait for pull-request merge
• attribution

However, I plan on submoduling all known librdkafka bindings and writing a small generic test case for each one as a means of regression testing the library API, assumed behaviour, etc.

Thanks for your exhaustive specification and analysis, really looking forward to seeing this come to life.
Dont hesitate if you need assistance.

[DEvil0000]

vote for different repo (as a wrapper? i don't like to have lua depends)
vote for c & c++ (for me c++ is more important if i would ever use Lua again)

Lua vs. Python:
about 10 years ago i would have used Lua but now i would use python in projects. if i watch other projects it looks like they also prefere python (now) so is there any need for Lua?

[thelamb]

Thanks for your input @DEvil0000

The implementation will be in a separate repo (main post updated).

Lua vs. Python:
Every project has its own needs. Python is certainly popular and many new project choose to use it. This is however no argument to discard Lua. When it comes to performance, Lua is miles ahead of Python (and any other dynamic language that I tested). Especially since there is a very good JIT implementation. I work on a network intrusion detection system, using Python for our scripting framework would simply trash the throughput. If performance wasn't a factor then I probably also would have chosen for Python.

[DEvil0000]

@thelamb i don't think Lua is faster in general it depends on your code and how you use the language. i have seen some pretty fast python code as well. if it comes to more complex data structures i am pretty sure python is faster - even with multidimensional arrays python should be faster.

take a look here:
http://flux242.blogspot.de/2013/05/python-vs-perl-vs-lua-speed-comparison.html
and here:
http://lua-users.org/wiki/LuaVersusPython

[thelamb]

Sure, I didn't mean to say Lua is faster in any case. Though my wording might suggest that.

In my situation, the native -> dynamic transition (of which there are many) alone was enough to discard Python.

That test you link to is true for non-JIT Lua/Python indeed. I re-ran those tests with LuaJIT and PyPy back when I was investigating which language to choose, and Lua was faster in the JIT case:

JIT

dd if=/dev/urandom bs=1M count=10 | ./pypy ./test.py > /dev/null
10485760 bytes (10 MB) copied, 6.21522 s, 1.7 MB/s

dd if=/dev/urandom bs=1M count=10 | luajit-2.0.0-beta9 ./test.lua > /dev/null
10485760 bytes (10 MB) copied, 5.37819 s, 1.9 MB/s

Non-JIT:

dd if=/dev/urandom bs=1M count=10 | ./test.py > /dev/null
10485760 bytes (10 MB) copied, 11.0328 s, 950 kB/s

dd if=/dev/urandom bs=1M count=10 | ./test.lua > /dev/null
10485760 bytes (10 MB) copied, 12.821 s, 818 kB/s

Anyway, I'm sure that there are cases where other languages are faster than Lua(JIT). Any project should decide for itself whether Lua is a good fit. The fact that it was for me justifies a Lua wrapper for rdkafka IMHO :).

[davidxiaozhi]

we use the luajit send the message

[edenhill]

@thelamb any progress on the Lua bindings?

[thelamb]

@edenhill I haven't been able to work on it much because of a change in my position at work. My free time is spent getting to grips with new responsibilities at the moment. I'm not abandoning this work, but if you want we can close this ticket and I will submit a PR when I have something to show.

Something we'll do in short-term is implement SSL for librdkafka. It's a hard requirement for how we want to use kafka in the near future. Alternatives like VPN are not possible in the scenario we're in. I'm aware that SSL support is not official yet (https://issues.apache.org/jira/browse/KAFKA-1682) but there's a fork that implements it (https://github.com/relango/kafka/tree/kafka_security_0.8.2).

If you've already thought about how to implement SSL support in librdkafka we could coordinate so that the work we do is also useful for you.

[edenhill]

@thelamb Congrats on the new position!

Regarding SSL: I've refactored things a bit on the 'dev' branch (originally for win32 support), one thing being the socket stuff now abstracted a bit which makes adding SSL a bit simpler.
It is currently a compile time thing (windows defines or not), so it needs to be changed to a runtime thing.

[edenhill]

And yes, coordinating the SSL work would be great :)

[ediskandarov]

Hi guys.

My colleague developed producer luajit binding with ffi.

Project located at https://github.com/mistsv/luardkafka

Please feel free to fork and make contributions.

There is no unit tests but library was tested in production environment.

[edenhill]

@toidi Thanks for sharing, will add it to the list of bindings.

[assada]

WARNING: this repo is not maintained anymore

there is no library for lua that support ssl...