PGO build

[zamazan4ik]

Hi!

Did you evaluate PGO applicability to fluentbit? I have done some investigation for Vector, and I think the same method should be applicable to fluentbit as well.

Would be awesome to get some feedback about the idea.

[zamazan4ik]

I did some PGO benchmarks with Fluent-Bit and want to share my results.

Test environment

• Fedora 38
• Linux kernel 6.3.11
• AMD Ryzen 9 5900x
• 48 Gib RAM
• SSD Samsung 980 Pro 2 Tib
• Clang 16 (from the Fedora repositories). I use Clang just because I prefer LLVM-based tooling
• Fluent-Bit version: from master branch (commit ac3cb7c5d5e79cb19fc127c680341af7956d3337)

Benchmark

In general, the benchmark methodology is the same as described in the corresponding Vector issue (link). That's a near real-life log scenario from our production (just Elasticsearch in the end instead of /dev/null). As the source of logs, I used one file with logs in the required format with a size ~7.9 Gib.

Configuration files

fluent-bit.conf:

[SERVICE]
    Parsers_File /home/zamazan4ik/open_source/bench_fb/parsers.conf

[INPUT]
    name              tail
    path              /home/zamazan4ik/open_source/bench_fb/logs/*
    Parser            point
    read_from_head    true

[OUTPUT]
    Name  null
    Match *

parsers.conf:

[PARSER]
    Name   point
    Format regex
    Regex /<(?<level>[EWD]) (?<thread>.+?) (?<tag>[a-z.]+) (?<datetime>[\d.]+ [\d:]*) (?<function>[\S]+) (?<mess>.*) \(from (?<file>[\S.]*) \+(?<line>\d+)\)/
    Time_Key datetime
    Time_Format %d.%m %H:%M:%S

Tested configurations

I have tested the following Fluent-Bit configurations (with corresponding CFLAGS):

• Release: CC=clang CXX=clang++ cmake ../fluent-bit -DFLB_RELEASE=On -DFLB_DEBUG=Off -DFLB_IPO=Off
• Release + LTO: CC=clang CXX=clang++ cmake ../fluent-bit -DFLB_RELEASE=On -DFLB_DEBUG=Off -DFLB_IPO=On
• Release + LTO + PGO: CC=clang CXX=clang++ cmake ../fluent-bit -DFLB_RELEASE=On -DFLB_DEBUG=Off -DFLB_IPO=On + -fprofile-instr-generate/-fprofile-instr-use flags.

Results

Here are the results of running the benchmark on different configurations. All configurations are benchmarked on the same machine, with the same Fluent-Bit configuration, multiple times, etc. The results show how much time Fluent-Bit needs to process logs (the source log file is the same for all runs). I have rechecked - the results are consistent between runs.

Also, I shared results for Instrumentation mode so you would be able to estimate how slow Fluent-Bit is in Instrumentation mode for PGO.

• Release: ~2m2s
• Release + LTO: ~1m46s
• Release + LTO + PGO: ~1m34s
• Release + LTO with Instrumentation: ~2m19s

At least in this scenario, Fluent-Bit performs measurably better (=faster) with LTO and PGO.

More results with PGO on real-life applications you can find here.

[agup006]

pretty cool, on the benchmark I wonder if PGO would have an effect if on the in_tail you added threaded on

[zamazan4ik]

Didn't test it but I expect the same result because improvements in single-threaded mode should be valid for multi-threaded config as well.

[zamazan4ik]

@agup006 I think letting know the Fluent-Bit users about the possibility of optimizing Fluent-Bit further with PGO would be helpful in practice. What do you think about the idea of creating a dedicated page about PGO in the Fluent-Bit docs? Maybe we need to create an additional "Performance tuning" section?

Here are the examples of how it's done in other projects:

• ClickHouse: https://clickhouse.com/docs/en/operations/optimizing-performance/profile-guided-optimization
• Databend: https://databend.rs/doc/contributing/pgo
• Vector: https://vector.dev/docs/administration/tuning/pgo/
• Nebula: https://docs.nebula-graph.io/3.5.0/8.service-tuning/enable_autofdo_for_nebulagraph/

[patrick-stephens]

Worth submitting a PR I think