generate perf event data structure in Python automatically

[boat0]

When ring buffers are opened to receive custom perf event, we have
to define the event data structure twice: once in BPF C and once
in Python. It is redundant and error-prone.

This patch implements the automatic generation of Python data structure
from the C declaration, thus making the redundant definition in Python
unnecessary. Example:

// define output data structure in C
struct data_t {
    u32 pid;
    u64 ts;
    char comm[TASK_COMM_LEN];
};
BPF_PERF_OUTPUT(events);
...

Old way:

# define output data structure in Python
TASK_COMM_LEN = 16    # linux/sched.h
class Data(ct.Structure):
    _fields_ = [("pid", ct.c_ulonglong),
                ("ts", ct.c_ulonglong),
                ("comm", ct.c_char * TASK_COMM_LEN)]

def print_event(cpu, data, size):
    event = ct.cast(data, ct.POINTER(Data)).contents
...

New way:

def print_event(cpu, data, size):
    event = b["events"].event(data)
...

[yonghong-song]

[buildbot, test this please]

[boat0]

It seems that buildbot has skipped 60073ef upon which 78ea22f is based.

[yonghong-song]

Maybe add s8, s16, s32, s64 as well?
How do we handle __int128 and unsigned __int128? I guess in such cases, the old way is needed where __int128 is coded as s64[2]?

[boat0]

Yes. The old way works great for __int128 here too:
'__int128' : (ct.c_longlong * 2),

s8/16/32/64 also added. Any other type that is not supported will stop
the script from running with message
"Type: '%s' not recognized. Please define the data with ctypes manually."

[yonghong-song]

Is it possible to have just one libbcc call like lib.bpf_perf_event_field to get field name and type in one shot?

[boat0]

Ok. We can concatenate type and name in libbcc and then split them in Python.

[yonghong-song]

@boat0 buildbot is okay. I see the error with llvm 3.8.0 and llvm 3.8.1 on ubuntu.

CMakeFiles/usdt-static.dir/usdt_args.cc.o -c /tmp/debuild.DTQhKj/bcc/src/cc/usdt/usdt_args.cc
/tmp/debuild.DTQhKj/bcc/src/cc/frontends/clang/b_frontend_action.cc: In member function ‘bool ebpf::BTypeVisitor::VisitCallExpr(clang::CallExpr*)’:
/tmp/debuild.DTQhKj/bcc/src/cc/frontends/clang/b_frontend_action.cc:818:46: error: ‘const class clang::Type’ has no member named ‘getPointeeOrArrayElementType’; did you mean ‘getPointeeType’?
                                            ->getPointeeOrArrayElementType()->getAsTagDecl();
                                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                              getPointeeType

Could you check whether you can find an alternative implementation? It would be good to find an implementation working for all llvm's. But if you cannot, using #ifdef is okay.

Also, could you add some tests to test some of your supported types including array types?
This patch can greatly simplify how to write bcc python scripts. Thanks!

[boat0]

@boat0 buildbot is okay. I see the error with llvm 3.8.0 and llvm 3.8.1 on ubuntu.

CMakeFiles/usdt-static.dir/usdt_args.cc.o -c /tmp/debuild.DTQhKj/bcc/src/cc/usdt/usdt_args.cc
/tmp/debuild.DTQhKj/bcc/src/cc/frontends/clang/b_frontend_action.cc: In member function ‘bool ebpf::BTypeVisitor::VisitCallExpr(clang::CallExpr*)’:
/tmp/debuild.DTQhKj/bcc/src/cc/frontends/clang/b_frontend_action.cc:818:46: error: ‘const class clang::Type’ has no member named ‘getPointeeOrArrayElementType’; did you mean ‘getPointeeType’?
                                            ->getPointeeOrArrayElementType()->getAsTagDecl();
                                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                              getPointeeType

Could you check whether you can find an alternative implementation? It would be good to find an implementation working for all llvm's. But if you cannot, using #ifdef is okay.

Now we use getPointeeType().getTypePtr() instead of getPointeeOrArrayElementType()

[boat0]

Also, could you add some tests to test some of your supported types including array types?

OK. I will do it in a subsequent PR.
I am now using tools/tcpconnect.py as a test and everything is OK.

tcpconnect.py has two events defined and features __int128 and array types:

// separate data structs for ipv4 and ipv6
struct ipv4_data_t {
    u64 ts_us;
    u32 pid;
    u32 uid;
    u32 saddr;
    u32 daddr;
    u64 ip;
    u16 dport;
    char task[TASK_COMM_LEN];
};
BPF_PERF_OUTPUT(ipv4_events);

struct ipv6_data_t {
    u64 ts_us;
    u32 pid;
    u32 uid;
    unsigned __int128 saddr;
    unsigned __int128 daddr;
    u64 ip;
    u16 dport;
    char task[TASK_COMM_LEN];
};
BPF_PERF_OUTPUT(ipv6_events);

[yonghong-song]

[buildbot, test this please]

[boat0]

Last build failed at test_usdt2.py in which event type is defined as a simple int type.
It is supposed to do nothing in this case, but the checking turns out to be not working.
In this update I make it working by

auto type_arg1 = Call->getArg(1)->IgnoreCasts()->getType().getTypePtr()->getPointeeType().getTypePtr();

if (type_arg1->isStructureType()) {...}

[yonghong-song]

[buildbot, test this please]

[yonghong-song]

The following __int128 should be the most common usage:

                      '__int128'          : (ct.c_longlong * 2),
                       'unsigned __int128' : (ct.c_ulonglong * 2),

If users try to map __int128 differently in python, they should try to explicit ct_type definitions in python instead.

[pchaigno]

This is great! Thanks a lot @boat0!

The developer tutorial would also need to be updated to take this pull request into account.

[boat0]

This is great! Thanks a lot @boat0!

The developer tutorial would also need to be updated to take this pull request into account.

pr submitted #2226. Thanks!