Reduce CPU overhead of per-event tools by allowing more buffering

[brendangregg]

Not a big problem yet, but at some point we should look into this...

The normal perf_output-consuming loop we've been using is:

while 1:
    b.kprobe_poll()

This wakes up as quickly as it can, and can consume noticeable CPU. I had to tune it for my cpuunclaimed tool:

while 1:
    # allow some buffering by calling sleep(), to reduce the context switch
    # rate and lower overhead.
    try:
        if not exiting:
            sleep(wakeup_s)
    except KeyboardInterrupt:
        exiting = 1
    b.kprobe_poll()
    slept += wakeup_s

This ticket is to explore doing this for more tools. Should b.kprobe_poll() have a built-in sleep, that can be tuned? Eg, 100ms by default?

This is more practical now we have #997, and can increase the buffer size.

[brendangregg]

A problem of a built-in sleep to kprobe_poll() would be Ctrl-C behavior. Maybe it's fine if it just lets it quit early, but the calling program would need to know Ctrl-C was hit so it can begin the exit code path.

[vitalyvch]

In my opinion, the best way is to implement it in kernel and to configure through ioctl(), like ttys and sockets do :-)

Should b.kprobe_poll() have a built-in sleep, that can be tuned?

Most likely no. I believe, much better is an example of this good practice in the documentation.

100ms by default?

I prefer something about 30 ms.

A problem of a built-in sleep to kprobe_poll() would be Ctrl-C behaviour.

I don't like the current handling of Ctrl-C at all. Probably it is a good idea to redevelop or to refactor it.

[brendangregg]

I'd allow a 30 ms sleep through an argument:

b.kprobe_poll(30)

It's about finding the balance: too short and the tool wakes up too often and consumes CPU. Too long and the tool may overflow its buffer. I'm guessing a 100 ms default for all tools would work, then we tune some as needed.

[yonghong-song]

Not sure whether b.kprobe_poll(30) would reduce the CPU usage or not as the poll will still wake up immediately if the file descriptor is ready. We may have to implement the sleep functionality in the library somehow to really enforce the waiting period between subsequent polls.

@brendangregg is this what you want?

[brendangregg]

I'm wondering about the status of this. So everything is using b.perf_buffer_poll() now? That has a timeout argument, but I don't think that's what we want. We want it to sleep and not wake up for a duration, to allow things to buffer. So it should have a sleep= argument as well. The bigger question is whether we make it the default, eg, 50ms for 20 wakeups per second. Fewer wakeups is good, but we don't want to overflow the buffer.

[pavlix]

Do I understand it correctly that the ideal behavior would roughly be to “wake up” when the buffer gets reasonably filled rather than when it's just non-empty?

[yonghong-song]

@pavlix that will be ideal. But the default value is very hard to choose. We do not want to overflow the buffer.

[hsane2001]

Has this change of a sleep value to perf_buffer_poll been included? This would be very useful for perf events.

[yonghong-song]

@hsane2001 No. The timeout value for perf_buffer_poll still used for poll timeout. Here, we want to sleep for sometime to wait for buffer to fill up before starting to collect data. The default sleep time is hard to choose though. You mentioned this would be very useful for perf events. Could you elaborate?

[hsane2001]

@yonghong-song - There seems to be a wide variation of the buffer's fill rate between Linux perf in counting mode or profiling (record) mode, with the latter easily leading to lost samples. I assumed the intention was to get some indication to wakeup when the buffer is full so we could drain it in user space and restart collection.

[yonghong-song]

I assumed the intention was to get some indication to wakeup when the buffer is full so we could drain it in user space and restart collection.
Yes, this is the intention. The primary goal is to save cpu time to avoid always busy waiting/polling. But we have not figured it out how to to do without polling :-(

[brendangregg]

The perf ring buffer in bcc should be reworked to make sense, to find a balance between buffering, dropping event,s and consuming CPU. We don't want tools burning too much CPU (as described above) or dropping events (e.g., #4219). We also would prefer to fix this in bcc rather than have people add custom fixes in each tool, although that may be unavoidable in some special cases.

For example, #4219 (adding "-b"/"--buffer-pages" to opensnoop) doesn't feel like a special case so hopefully that should be done via bcc for all tools instead (and hopefully at some point we can remove -b from opesnoop).