Lua Tools for BCC #457

vmg · 2016-03-29T16:50:08Z

As promised in #455.

This is a code dump of a Lua(JIT) wrapper that we've developed internally at GitHub. Some notes:

The code is Apache 2 licensed, like the rest of BCC.
The code contains a few third party helpers to make writing Lua probes easier. All the dependencies are MIT licensed and 100% compatible with the rest of the wrapper. All due attribution has been added to the corresponding files.
The wrapper has been put in src/lua, so it sits next to the src/python wrapper and the actual implementation.
Some of the existing Python tools have been rewritten to Lua as a sample of how to use the new APIs. They can be found under examples/lua.
The best place to learn how to write a probe using these bindings is memleak.lua. It's a full port of @brendangregg's memleak.py and should have identical behavior.

I'm aware this is a lot of code to go through. I'm not sure what else can I do to help you approve the binding and get it merged on the tree (or whether you trust my dubious Lua expertise, hehe), but I'm here to answer all your questions.

Thanks!

vmg · 2016-03-29T16:53:17Z

examples/lua/bashreadline.lua

+    print("%-9s %-6d %s" % {os.date("%H:%M:%S"), tonumber(event.pid), ffi.string(event.str)})
+  end
+
+  b:get_table("events"):open_perf_buffer(print_readline, "struct { uint64_t pid; char str[80]; }")


This is (obviously) a port of the existing bashreadline.py, and I'm particularly happy with it. Very little code, and it shows how you can pass a literal C definition for the structure we're getting out of the perf_buffer, and have the binding JIT compile it to native code, so the callback receives an event struct instead of a void pointer.

I think it's a clear win over the Python API!

looks great. was worried that this struct {pid, char str[80];} copy-paste will be an issue, but looks like other examples get fields into lua dynamically.

drzaeus77 · 2016-03-29T17:06:11Z

Will look through this in more detail, but first: where are the test(s)? :) Would be good to have at least one with the initial checkin to test the sanity of things like dependencies, file placement, etc.

vmg · 2016-03-29T17:08:10Z

Good point, hehe. I'm afraid most of the tests are ad-hoc scripts, so I guess my next thing in the TODO list is writing a more-or-less comprehensive suite in Lua that we can put on this PR. I'll get to it sometime this week.

brendangregg · 2016-03-29T17:11:37Z

Amazing work! bashreadline does make for a nice short example. BTW memleak.py is actually Sasha's, not mine.

drzaeus77 · 2016-03-29T17:27:00Z

First shot at running the example didn't work following the README. I'm sure I'll be able to get it working but probably some updates to the doc will be needed.

Running from src/lua:

$ sudo ./bcc-probe ../../examples/lua/task_switch.lua 
luajit: ./bcc-probe:31: module '../../examples/lua/task_switch' not found:
        no field package.preload['../../examples/lua/task_switch']
        no file './//////examples/lua/task_switch/init.lua'
        no file './//////examples/lua/task_switch.lua'
        no file '/usr/share/luajit-2.0.4///////examples/lua/task_switch.lua'
        no file '/usr/local/share/lua/5.1///////examples/lua/task_switch.lua'
        no file '/usr/local/share/lua/5.1///////examples/lua/task_switch/init.lua'
        no file '/usr/share/lua/5.1///////examples/lua/task_switch.lua'
        no file '/usr/share/lua/5.1///////examples/lua/task_switch/init.lua'
        no file './//////examples/lua/task_switch.so'
        no file '/usr/local/lib/lua/5.1///////examples/lua/task_switch.so'
        no file '/usr/lib/lua/5.1///////examples/lua/task_switch.so'
        no file '/usr/local/lib/lua/5.1/loadall.so'
        no file './.so'
        no file '/usr/local/lib/lua/5.1/.so'
        no file '/usr/lib/lua/5.1/.so'
        no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
        [C]: in function 'require'
        ./bcc-probe:31: in main chunk
        [C]: at 0x00404750

and switching to the examples directory:

$ sudo ../../src/lua/bcc-probe bashreadline.lua 
luajit: ../../src/lua/bcc-probe:19: module 'bcc' not found:
        no field package.preload['bcc']
        no file './bcc/init.lua'
        no file './bcc.lua'
        no file '/usr/share/luajit-2.0.4/bcc.lua'
        no file '/usr/local/share/lua/5.1/bcc.lua'
        no file '/usr/local/share/lua/5.1/bcc/init.lua'
        no file '/usr/share/lua/5.1/bcc.lua'
        no file '/usr/share/lua/5.1/bcc/init.lua'
        no file './bcc.so'
        no file '/usr/local/lib/lua/5.1/bcc.so'
        no file '/usr/lib/lua/5.1/bcc.so'
        no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
        [C]: in function 'require'
        ../../src/lua/bcc-probe:19: in main chunk
        [C]: at 0x00404750

vmg · 2016-03-29T17:38:45Z

@drzaeus77: Actually, there's an issue with the examples because the bcc-probe loader is not prepared to load relative paths. 😅

Let me fix this for you. :)

vmg · 2016-03-29T17:42:50Z

@drzaeus77: Fixed. Sorry for wasting your time. Our original deployment had our custom tools as modules. You should now be able to call bcc-probe from src/lua to run probes regardless of location.

Gonna get started with these tests because they're obviously necessary.

vmg · 2016-03-29T17:45:13Z

Amazing work! bashreadline does make for a nice short example. BTW memleak.py is actually Sasha's, not mine.

Thank you! I'd swear I saw memleak on one of your slides. I guess you were just using it an example. Sorry Sasha! :)

goldshtn · 2016-03-29T18:09:55Z

This is very impressive. I am barely conversational in Lua, but I can definitely understand the desire to use a JIT-compiled language. I wonder how we are going to move forward, though, with shared functionality such as the USDT* and Tracepoint classes. Is there a way for users of one language to benefit from shared code developed in the other language?

4ast · 2016-03-30T01:57:43Z

src/lua/bcc/bpf.lua

+    n = n + 1
+  end
+
+  assert(n == kprobe_count, "wtf")


meaningful error please

4ast · 2016-03-30T02:27:49Z

overall looks great. Would be even more awesome to have a static tool that can be built out of luajit-devel and libbcc libraries. Installing libbcc.so and python often gets in the way in production systems. If we can have standalone binary (with only external dependency on kernel headers) then it will go long way.
Please add the tests, so buildbots can make sure that stuff still works.

drzaeus77 · 2016-03-30T05:33:23Z

examples/lua/bashreadline.lua

+local ffi = require("ffi")
+
+return function(BPF)
+  local b = BPF:new{src_file="bashreadline.c", debug=true}


Two things:

We've found that inlining the C source contents (which you do in memleak.lua) is more portable. I would suggest adding relative path support or inlining everything, since a user needs to be in the same directory to be able to find bashreadline.c here.

debug is not a bool, but rather a set of flags. See Minor endian and debug enhancement #456

Thinking aloud here...if we do truly share the .c file implementation, then duplicating that code in the different bindings could become a bad thing. We should weigh that duplication against tool portability.

Possible solutions to a problem that only just barely exists yet: cmake prepocessing, .c implementation moving to libbcc, git precommit hook to detect out-of-sync files, etc.

I found the section on relative include later on in the code, so this is partially answered. If anyone else thinks we should worry about code duplication, let's discuss.

vmg · 2016-03-30T14:29:51Z

Hey all! I've pushed a small batch of tests (ported straight from the Python counterparts) and implemented a bunch of features which were missing to pass those tests. :)

Answering your questions now:

This is very impressive. I am barely conversational in Lua, but I can definitely understand the desire to use a JIT-compiled language. I wonder how we are going to move forward, though, with shared functionality such as the USDT* and Tracepoint classes. Is there a way for users of one language to benefit from shared code developed in the other language?

Thank you! 😍

I'm afraid I don't have a good answer regarding code reuse -- beside the obvious that really important features should be written in C/C++ and available in libbcc. There is no way to automagically reuse Python classes from Lua. I think that's OK. Maybe I've spent too much of my career writing cross-language bindings, but I feel like as long as they're (mostly) thin wrappers over the libbcc code, having diverging and idiomatic implementations is a healthy thing. It keeps the API design in check!

btw, python has the same simple approach to trace_pipe, but it can be used by multiple processes/tracers and can be in binary mode, so would be good to add a check that /sys/kernel/debug/tracing/trace_options has noraw, nobin and context-info set, otherwise regex from line 19 will be confused

Thanks for the pointer. I'll try to get this right in both Python and Lua tomorrow. 👌

Would be even more awesome to have a static tool that can be built out of luajit-devel and libbcc libraries

Totally doable. I've done this plenty of times before for deployments. The resulting binary is rather fat but only depends on libc, so it should be drag-and-droppable anywhere. All it takes is a simple C wrapper, and a small script to compile the Lua bytecode for the wrapper and stick it on the data section of the binary, and a Makefile rule to package it up. Quick and painless!

...Except I gotta port the Makefile rule to CMake. [sweats profusely]

Please add the tests, so buildbots can make sure that stuff still works.

Tests are in place. Now I just gotta learn how to wire them up so they run on CI. Let me tinker with this...

vmg · 2016-03-30T14:45:08Z

OK, I've added the tests to CMake, but they will not run unless luajit can be found on the path.

I'm afraid I don't have access to the buildbot to install luajit. Could anybody take care of that for me? :)

4ast · 2016-03-30T15:59:31Z

tests/lua/CMakeLists.txt

@@ -0,0 +1,6 @@
+find_program(LUAJIT luajit)
+
+if(LUAJIT)


@drzaeus77 can you install luajit on buildbots and re-trigger ?

installed on ubuntu:

Setting up libluajit-5.1-common (2.0.2+dfsg-1) ... Setting up luajit (2.0.2+dfsg-1) ...

installed on fedora:

luajit.x86_64 2.0.3-4.fc22

4ast · 2016-03-30T17:48:32Z

@vmg
20/20 Test #20: Unable to find executable: sudo
lua_test_clang ...................***Not Run 0.00 sec

drzaeus77 · 2016-03-30T17:50:12Z

FYI, sudo is definitely installed on the box, we use and depend on it in
other tests.

On Wed, Mar 30, 2016 at 10:48 AM, 4ast notifications@github.com wrote:

@vmg https://github.com/vmg
20/20 Test #20 #20: Unable to find
executable: sudo
lua_test_clang ...................***Not Run 0.00 sec

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#457 (comment)

vmg · 2016-03-30T18:44:45Z

💥 Green in Ubuntu and Fedora. I've also added some uprobe and dumping tests.

I'm working on a standalone build for the lua tools. It's coming along nicely. Anything else you'd like to see from this first PR before merging? :)

4ast · 2016-03-30T18:58:23Z

the last two commits could have been squashed, but ok for the history

vmg · 2016-03-30T19:52:34Z

🎉 🎉 🎉

Exciting! Thanks for the feedback and support!

vmg reviewed Mar 29, 2016
View reviewed changes

4ast reviewed Mar 30, 2016
View reviewed changes

src/lua/bcc/bpf.lua

n = n + 1

end

assert(n == kprobe_count, "wtf")

Copy link

Member

4ast Mar 30, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

meaningful error please

drzaeus77 reviewed Mar 30, 2016
View reviewed changes

vmg added 11 commits March 30, 2016 17:31

Lua Tools for BCC

39468d9

Add a few examples written in Lua

df11f32

bcc-probe: Allow loading absolute paths

74d2e6c

bcc-probe: Allow running the helper from any path

a978401

bpf.lua: Add support for CFLAGS and LLVM debug flags

e789360

bpf.lua: Implement load_funcs

e6b7c74

table.lua: Add support for decoding bitfield types

fb7d642

table.py: Implementscanf and printf helpers

d58fe06

test.lua: Add basic test library and helpers

c0e5ef8

test.lua: Port test_clang to Lua

e4e83ec

memleak.lua: Fix indentation

afd4303

vmg added 5 commits March 30, 2016 17:31

bpf.lua: Sanity check

abdbc36

cmake: Run the Lua tests too

28b4efe

bpf.lua: fix detaching uprobes and kprobes

b495d2c

ld.lua: fix parsing of base16 addresses

b2a64a7

test.lua: Port the test_uprobes suite

72d7aa3

vmg force-pushed the vmg/lua branch from 5b39fae to 72d7aa3 Compare March 30, 2016 15:31

bpf.lua: Implement dump_func

81990c3

4ast reviewed Mar 30, 2016
View reviewed changes

vmg added 2 commits March 30, 2016 20:17

cmake: Find sudo explicitly

1e543fa

cmake: Use the wrapper script for Lua tests too

ab32481

4ast merged commit a480304 into iovisor:master Mar 30, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lua Tools for BCC #457

Lua Tools for BCC #457

vmg commented Mar 29, 2016

vmg Mar 29, 2016

4ast Mar 30, 2016

drzaeus77 commented Mar 29, 2016

vmg commented Mar 29, 2016

brendangregg commented Mar 29, 2016

drzaeus77 commented Mar 29, 2016

vmg commented Mar 29, 2016

vmg commented Mar 29, 2016

vmg commented Mar 29, 2016

goldshtn commented Mar 29, 2016

4ast Mar 30, 2016

4ast commented Mar 30, 2016

drzaeus77 Mar 30, 2016

drzaeus77 Mar 30, 2016

drzaeus77 Mar 30, 2016

vmg commented Mar 30, 2016

vmg commented Mar 30, 2016

4ast Mar 30, 2016

drzaeus77 Mar 30, 2016

4ast commented Mar 30, 2016

drzaeus77 commented Mar 30, 2016

vmg commented Mar 30, 2016

4ast commented Mar 30, 2016

vmg commented Mar 30, 2016

Lua Tools for BCC #457

Lua Tools for BCC #457

Conversation

vmg commented Mar 29, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drzaeus77 commented Mar 29, 2016

vmg commented Mar 29, 2016

brendangregg commented Mar 29, 2016

drzaeus77 commented Mar 29, 2016

vmg commented Mar 29, 2016

vmg commented Mar 29, 2016

vmg commented Mar 29, 2016

goldshtn commented Mar 29, 2016

Choose a reason for hiding this comment

4ast commented Mar 30, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vmg commented Mar 30, 2016

vmg commented Mar 30, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

4ast commented Mar 30, 2016

drzaeus77 commented Mar 30, 2016

vmg commented Mar 30, 2016

4ast commented Mar 30, 2016

vmg commented Mar 30, 2016