Add Fluent Bit source and sink operators #3461

tobim · 2023-08-15T14:24:12Z

This will add fluent-bit loaders and savers.

This PR is in a very early stage right now, the actual functionality is not implemented yet.

We include a custom fluent-bit-minimal.h from fluent/fluent-bit#7165 (comment) because the public API of fluent-bit is currently unusable. This will hopefully be included upstream in the near future.

Business Logic

Give feedback

Implement first scaffold to test fluent-bit library

Implement first scaffold to test fluent-bit library
Options
Successfully updated the issue's project

There was an error updating the issue's project
Write Fluent Bit input plugin

Write Fluent Bit input plugin
Options
Successfully updated the issue's project

There was an error updating the issue's project
Implement `fluentbit` sink operator

Implement fluentbit sink operator
Options
Successfully updated the issue's project

There was an error updating the issue's project
Implement `fluentbit` source operator

Implement fluentbit source operator
Options
Successfully updated the issue's project

There was an error updating the issue's project
Add logic that converts Fluent Bit JSON to table slices

Add logic that converts Fluent Bit JSON to table slices
Options
Successfully updated the issue's project

There was an error updating the issue's project
Apply decoding heuristic

Apply decoding heuristic
Options
Successfully updated the issue's project

There was an error updating the issue's project
Fix crash on invalid sink, e.g., `fluent-bit xxx`

Fix crash on invalid sink, e.g., fluent-bit xxx
Options
Successfully updated the issue's project

There was an error updating the issue's project
Add integration tests

Add integration tests
Options
Successfully updated the issue's project

There was an error updating the issue's project
Options

Scaffolding

Give feedback

Get initial scaffold compiling

Get initial scaffold compiling
Options
Successfully updated the issue's project

There was an error updating the issue's project
Reconsider rolling our own input plugin, as opposed to pushing JSON through `flb_lib_push`

Reconsider rolling our own input plugin, as opposed to pushing JSON through flb_lib_push
Options
Successfully updated the issue's project

There was an error updating the issue's project
Fix Docker build

Fix Docker build
Options
Successfully updated the issue's project

There was an error updating the issue's project
Static libraries are currently not installed and scattered across many archive files. We need to manually pass them to the linker command line.

Static libraries are currently not installed and scattered across many archive files. We need to manually pass them to the linker command line.
Options
Successfully updated the issue's project

There was an error updating the issue's project
The few dependencies that are not vendored need to be linked in manually for the static build.

The few dependencies that are not vendored need to be linked in manually for the static build.
Options
Successfully updated the issue's project

There was an error updating the issue's project
Options

mavam · 2023-08-17T07:17:30Z

That build system work is quite a feat!

I had started something local using crtp_operator. Is saver/loader the right abstractions? Aren't we going from void to events directly? I haven't studied the fluent-bit API in depth, perhaps there's a MsgPack parser/printer here as well.

tobim · 2023-08-17T14:08:01Z

I had started something local using crtp_operator. Is saver/loader the right abstractions? Aren't we going from void to events directly? I haven't studied the fluent-bit API in depth, perhaps there's a MsgPack parser/printer here as well.

I just assumed that we would go to bytes first, but looking at the included minimal API it seems to make more sense to go to directly to events.

This is an implementation choice that you should make as the main implementer or the feature.

mavam · 2023-08-17T18:01:38Z

This is an implementation choice that you should make as the main implementer or the feature.

Alrighty, can I take it from here?

mavam · 2023-08-18T07:42:08Z

@tobim I've split the outstanding todos in two tasklists for clearer responsibilities. In particular, please review da417f1, which I needed to get things to link on macOS.

I'm surprised that the Docker build fails, as the script that installs the dev deps runs through just fine, and the plain Debian CI build works. Here I have no intuition how to proceed and would probably not invest any time myself. Any help would be appreciated.

plugins/fluent-bit/src/plugin.cpp

mavam · 2023-09-02T09:45:09Z

I managed to get rid of any custom Fluent Bit plugins. This now works with a stock Fluent Bit package that you get on your system.

The consequence is that we now have to go through JSON when using Fluent Bit as sink (Fluent Bit input), because the library function flb_lib_push only supports JSON. We're also restricted to JSON when using Fluent Bit as source (= Fluent Bit output), although here Fluent Bit already provides us with the option of parsing the raw MsgPack.

Net effect: we have the potential to tune performance as source operator, but not as sink operator.

I've asked @edsiper in Slack:

[E]duardo, would you accept a patch that enhances in_lib and flb_lib_push to accept MsgPack in addition to JSON? It's already possible to consume MsgPack with out_lib, but not yet in_lib. This would make the library API symmetric.

I'm envisioning no API change, just making this work:
in_ffd = flb_input(ctx, "lib", NULL);
flb_input_set(ctx, in_ffd, "format", "msgpack", NULL);
flb_lib_push(ctx, in_ffd, msgpack_buf, msgpack_buf_len);

With a bit of luck, this will be a welcome future contribution as it doesn't change anything about the public API.

plugins/fluent-bit/cmake/Findfluentbit.cmake

libtenzir/src/tql/parser.cpp

tobim · 2023-09-15T19:46:41Z

I'm surprised that the Docker build fails, as the script that installs the dev deps runs through just fine, and the plain Debian CI build works. Here I have no intuition how to proceed and would probably not invest any time myself. Any help would be appreciated.

The regular debian CI job does not build any plugins except for web. Plugins are built in a dedicated job matrix to which I just added fluent-bit.

The issue in the docker build was quite hairy and took me longer to fix than the static build. Behold: https://github.com/tenzir/tenzir/pull/3461/files#diff-4f95d2e348b638a037c80301070df229e817aaba51d9cd3b868976c7ae96c695R37-R69

mavam · 2023-09-16T07:49:24Z

CI error:

CMake Error at CMakeLists.txt:29 (dependency_summary):
  Unknown CMake command "dependency_summary".

@tobim IIRC we don't have that function in plugins. @dominiklohmann once told me to just remove it.

tobim · 2023-09-16T08:13:03Z

@tobim IIRC we don't have that function in plugins. @dominiklohmann once told me to just remove it.

This has been bugging us often enough. I refactored the functionality so it can be reused for standalone plugins in the latest commit.

The plugin scaffold requires static plugins to have identical plugin and project names.

We need to work around the limitations of the upstream build and installation logic, which is done by collecting all necessary archives into the plugin linker command line. See the code comments for details.

The fluent bit shared library contains a private copy of jemalloc which can't be loaded with `dlopen()` at runtime. We work around the issue by preloading fluent bit with the Tenzir binary when possible. The proper fix would be to use the system jemalloc instead of vendoring a private copy in the fluent-bit package.

These are no longer necessary because there's not more shared state between our plugin and a third-party Fluent Bit plugin. Phew.

Co-authored-by: Daniel Kostuj <daniel.kostuj@tenzir.com>

The official fluent-bit package contains a bundeld copy of jemalloc that poses 2 problems: First, it is built in a way that makes it impossible to load `libfluent-bit.so` with `dlopen()`. And second, the way it initializes thread local variables creates problems with the jemalloc present in `libarrow.so`, specifically it causes a segfault when initializing the s3 filesystem. I'm trying to resolve this issue upstream at fluent/fluent-bit#8005, but until it is fixed we have to rely on this fallback. Signed-off-by: Tobias Mayer <tobim@fastmail.fm>

tobim added the connector Loader and saver label Aug 15, 2023

tobim changed the title ~~Topic/fluent bit~~ Add a fluent-bit connector Aug 15, 2023

jachris reviewed Aug 18, 2023

View reviewed changes

plugins/fluent-bit/src/plugin.cpp Outdated Show resolved Hide resolved

mavam reviewed Aug 27, 2023

View reviewed changes

plugins/fluent-bit/src/plugin.cpp Outdated Show resolved Hide resolved

mavam reviewed Aug 30, 2023

View reviewed changes

plugins/fluent-bit/src/plugin.cpp Outdated Show resolved Hide resolved

mavam force-pushed the topic/fluent-bit branch from dd77a10 to a44f8f7 Compare August 31, 2023 09:14

mavam changed the title ~~Add a fluent-bit connector~~ Add a Fluent Bit operator Aug 31, 2023

mavam added operator Source, transformation, and sink and removed connector Loader and saver labels Aug 31, 2023

mavam mentioned this pull request Aug 31, 2023

Public-consumable fluent-bit.h should not reference unnecessary dependencies fluent/fluent-bit#7165

Closed

mavam reviewed Sep 1, 2023

View reviewed changes

plugins/fluent-bit/src/plugin.cpp Outdated Show resolved Hide resolved

mavam linked an issue Sep 2, 2023 that may be closed by this pull request

FluentBit Connector tenzir/public-roadmap#47

Closed

mavam reviewed Sep 10, 2023

View reviewed changes

plugins/fluent-bit/cmake/Findfluentbit.cmake Outdated Show resolved Hide resolved

mavam force-pushed the topic/fluent-bit branch from 4405d39 to 41dfdf8 Compare September 10, 2023 07:50

jachris reviewed Sep 15, 2023

View reviewed changes

libtenzir/src/tql/parser.cpp Show resolved Hide resolved

tobim force-pushed the topic/fluent-bit branch from bd7a9b7 to 0c3ab14 Compare September 15, 2023 10:14

mavam added the feature New functionality label Sep 15, 2023

tobim force-pushed the topic/fluent-bit branch from 0c3ab14 to b6c67ed Compare September 15, 2023 19:35

tobim force-pushed the topic/fluent-bit branch from 52ca0c5 to 872e4cd Compare September 16, 2023 14:55

mavam force-pushed the topic/fluent-bit branch 2 times, most recently from 86a7c67 to 89a3c90 Compare September 16, 2023 18:51

mavam added the integration Integration with third-party tools label Sep 17, 2023

tobim and others added 25 commits October 6, 2023 09:32

Consistently use the spelling "fluent-bit"

7cf6906

The plugin scaffold requires static plugins to have identical plugin and project names.

Support the fluent-bit operator in static binary mode

75a47c5

We need to work around the limitations of the upstream build and installation logic, which is done by collecting all necessary archives into the plugin linker command line. See the code comments for details.

Also rename operator in docs

eb02212

Add changelog entry

0fa97d0

Install fluent-bit in the docker images

7f901e9

Build the fluent-bit plugin in CI

34d1973

Support the build summary for standalone plugins

4178745

Add plugin dependencies to the summary

0236e53

Don't dearmor the fluent-bit package signing key

438d0dc

Silence warnings

c0e8605

Get rid of C'isms

55473d6

These are no longer necessary because there's not more shared state between our plugin and a third-party Fluent Bit plugin. Phew.

Implement table slice production logic

1b6dd3e

Parse JSON with new series builder

63fe806

Add release blog post describing Fluent Bit

3cddcfa

Fix title

31b9c98

Perform poor-man's decoding

6e94d1d

Add guard rails to fix a Fluent Bit crash

031337e

Add integration tests

90d9af8

Fix typos

45e9f6e

Co-authored-by: Daniel Kostuj <daniel.kostuj@tenzir.com>

Remove unneeded include

0649e35

Add Fluent Bit exclusion

0702d64

Push error handling deeper

4936d5e

Return more meaningful error

68de96f

Build the fluent-bit plugin as part of tenzir

0f54c42

tobim force-pushed the topic/fluent-bit branch from 0dcdd6c to 75a26c0 Compare October 6, 2023 07:42

tobim force-pushed the topic/fluent-bit branch from 75a26c0 to 7919561 Compare October 6, 2023 08:00

Dakostu merged commit 3b4b4b4 into main Oct 6, 2023
38 checks passed

Dakostu deleted the topic/fluent-bit branch October 6, 2023 09:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Fluent Bit source and sink operators #3461

Add Fluent Bit source and sink operators #3461

tobim commented Aug 15, 2023 •

edited by mavam

Business Logic

Scaffolding

mavam commented Aug 17, 2023 •

edited

tobim commented Aug 17, 2023

mavam commented Aug 17, 2023

mavam commented Aug 18, 2023

mavam commented Sep 2, 2023

tobim commented Sep 15, 2023

mavam commented Sep 16, 2023

tobim commented Sep 16, 2023

Add Fluent Bit source and sink operators #3461

Add Fluent Bit source and sink operators #3461

Conversation

tobim commented Aug 15, 2023 • edited by mavam

Business Logic

Scaffolding

mavam commented Aug 17, 2023 • edited

tobim commented Aug 17, 2023

mavam commented Aug 17, 2023

mavam commented Aug 18, 2023

mavam commented Sep 2, 2023

tobim commented Sep 15, 2023

mavam commented Sep 16, 2023

tobim commented Sep 16, 2023

tobim commented Aug 15, 2023 •

edited by mavam

mavam commented Aug 17, 2023 •

edited