generate perf event data structure in Python automatically #2198
Conversation
boat0
requested review from
4ast,
drzaeus77,
goldshtn,
yonghong-song and
brendangregg
as code owners
|
[buildbot, test this please] |
src/python/bcc/table.py
Outdated
| 'long long': ct.c_longlong, | ||
| 'unsigned long long': ct.c_ulonglong, | ||
| 'u64': ct.c_ulonglong, | ||
| 'void *': ct.c_void_p } |
There was a problem hiding this comment.
Maybe add s8, s16, s32, s64 as well?
How do we handle __int128 and unsigned __int128? I guess in such cases, the old way is needed where __int128 is coded as s64[2]?
There was a problem hiding this comment.
Yes. The old way works great for __int128 here too:
'__int128' : (ct.c_longlong * 2),
s8/16/32/64 also added. Any other type that is not supported will stop
the script from running with message
"Type: '%s' not recognized. Please define the data with ctypes manually."
src/python/bcc/table.py
Outdated
| i = 0 | ||
| while i < num_fields: | ||
| field_name = lib.bpf_perf_event_field_name(self.bpf.module, i) | ||
| field_type = lib.bpf_perf_event_field_type(self.bpf.module, i) |
There was a problem hiding this comment.
Is it possible to have just one libbcc call like lib.bpf_perf_event_field to get field name and type in one shot?
There was a problem hiding this comment.
Ok. We can concatenate type and name in libbcc and then split them in Python.
|
@boat0 buildbot is okay. I see the error with llvm 3.8.0 and llvm 3.8.1 on ubuntu. Could you check whether you can find an alternative implementation? It would be good to find an implementation working for all llvm's. But if you cannot, using Also, could you add some tests to test some of your supported types including array types? |
Now we use getPointeeType().getTypePtr() instead of getPointeeOrArrayElementType() |
OK. I will do it in a subsequent PR. tcpconnect.py has two events defined and features __int128 and array types: |
boat0
force-pushed
the
gen-pydata
branch
2 times, most recently
from
d8dbbd5 to
b3868c5
Compare
|
[buildbot, test this please] |
When ring buffers are opened to receive custom perf event, we have
to define the event data structure twice: once in BPF C and once
in Python. It is tedious and error-prone.
This patch implements the automatic generation of Python data structure
from the C declaration, thus making the redundant definition in Python
unnecessary. Example:
// define output data structure in C
struct data_t {
u32 pid;
u64 ts;
char comm[TASK_COMM_LEN];
};
BPF_PERF_OUTPUT(events);
...
Old way:
# define output data structure in Python
TASK_COMM_LEN = 16 # linux/sched.h
class Data(ct.Structure):
_fields_ = [("pid", ct.c_ulonglong),
("ts", ct.c_ulonglong),
("comm", ct.c_char * TASK_COMM_LEN)]
def print_event(cpu, data, size):
event = ct.cast(data, ct.POINTER(Data)).contents
...
New way:
def print_event(cpu, data, size):
event = b["events"].event(data)
...
Take tcpconnect.py as an example to show the new, simpler way of outputing perf event. Other tools/examples can be simplified in a similar way.
|
Last build failed at test_usdt2.py in which event type is defined as a simple int type. |
|
[buildbot, test this please] |
|
The following If users try to map __int128 differently in python, they should try to explicit ct_type definitions in python instead. |
Simplify code following iovisor#2198 (iovisor#2198). Some tools are not touched: mountsnoop.py, trace.py, lib/*.py, old/*.py.
…e in Python Follow-up to iovisor#2198.
…e in Python Follow-up to iovisor#2198.
|
This is great! Thanks a lot @boat0! The developer tutorial would also need to be updated to take this pull request into account. |
pr submitted #2226. Thanks! |
) * generate perf event data structure in Python automatically When ring buffers are opened to receive custom perf event, we have to define the event data structure twice: once in BPF C and once in Python. It is tedious and error-prone. This patch implements the automatic generation of Python data structure from the C declaration, thus making the redundant definition in Python unnecessary. Example: // define output data structure in C struct data_t { u32 pid; u64 ts; char comm[TASK_COMM_LEN]; }; BPF_PERF_OUTPUT(events); ... Old way: # define output data structure in Python TASK_COMM_LEN = 16 # linux/sched.h class Data(ct.Structure): _fields_ = [("pid", ct.c_ulonglong), ("ts", ct.c_ulonglong), ("comm", ct.c_char * TASK_COMM_LEN)] def print_event(cpu, data, size): event = ct.cast(data, ct.POINTER(Data)).contents ... New way: def print_event(cpu, data, size): event = b["events"].event(data) ... * tools/tcpconnect.py: deduce perf event data structure automatically Take tcpconnect.py as an example to show the new, simpler way of outputing perf event. Other tools/examples can be simplified in a similar way.
…sor#2204) Simplify code following iovisor#2198 (iovisor#2198). Some tools are not touched: mountsnoop.py, trace.py, lib/*.py, old/*.py.
…e in Python (iovisor#2211) Follow-up to iovisor#2198.
) * generate perf event data structure in Python automatically When ring buffers are opened to receive custom perf event, we have to define the event data structure twice: once in BPF C and once in Python. It is tedious and error-prone. This patch implements the automatic generation of Python data structure from the C declaration, thus making the redundant definition in Python unnecessary. Example: // define output data structure in C struct data_t { u32 pid; u64 ts; char comm[TASK_COMM_LEN]; }; BPF_PERF_OUTPUT(events); ... Old way: # define output data structure in Python TASK_COMM_LEN = 16 # linux/sched.h class Data(ct.Structure): _fields_ = [("pid", ct.c_ulonglong), ("ts", ct.c_ulonglong), ("comm", ct.c_char * TASK_COMM_LEN)] def print_event(cpu, data, size): event = ct.cast(data, ct.POINTER(Data)).contents ... New way: def print_event(cpu, data, size): event = b["events"].event(data) ... * tools/tcpconnect.py: deduce perf event data structure automatically Take tcpconnect.py as an example to show the new, simpler way of outputing perf event. Other tools/examples can be simplified in a similar way.
…sor#2204) Simplify code following iovisor#2198 (iovisor#2198). Some tools are not touched: mountsnoop.py, trace.py, lib/*.py, old/*.py.
…e in Python (iovisor#2211) Follow-up to iovisor#2198.
When ring buffers are opened to receive custom perf event, we have
to define the event data structure twice: once in BPF C and once
in Python. It is redundant and error-prone.
This patch implements the automatic generation of Python data structure
from the C declaration, thus making the redundant definition in Python
unnecessary. Example:
Old way:
New way: