fluent-bit shared object go plugin

[cosmo0920]

What this PR does / why we need it:

The first step to merge https://github.com/cosmo0920/fluent-bit-go-loki.

Which issue(s) this PR fixes:
No fixes. This is the first task of #770.

Special notes for your reviewer:

I think that we would have to create fluent-bit/fluent-bit-go-loki folder instead of cmd/fluent-bit.
Because fluent-bit-go-loki Golang plugin uses shared object format.
It is not executable.

Building steps for fluent-bit-go-loki

fluent-bit-go-loki can build with:

$ (cd fluent-bit/fluent-bit-go-loki && go build -buildmode=c-shared -o out_loki.so .)

Or, make with added Makefile(building and testing):

$ (cd fluent-bit/fluent-bit-go-loki && make)

Testing for fluent-bit-go-loki

fluent-bit-go-loki test can execute with:

$ (cd fluent-bit/fluent-bit-go-loki && go test -cover -race -coverprofile=coverage.txt -covermode=atomic)

Or, make with added Makefile:

$ (cd fluent-bit/fluent-bit-go-loki && make test)

Checklist

• Documentation added
• Tests updated

[sh0rez]

Thanks for this, looks pretty good so far! Just some minor nits 😄

[sh0rez]

Regarding the placement of the fluentd / fluent-bit plugins ... Couldn't we place them in a common fluent folder? Something like

.
├── fluent
│   ├── bit
│   └── d

/cc @cyriltovena @slim-bean

[sh0rez]

Awesome! Thanks a lot 🎉

LGTM

[cyriltovena]

Can you confirm the new code in fluent-bit folder is linted by the linter ?

[cosmo0920]

golangci-lint complains the following warning:

% GOGC=20 golangci-lint run
WARN [runner/golint] Golint: can't lint 5 files: no file name for file &{Doc:<nil> Package:21595946 Name:main Decls:[0xc0287bd040 0xc0287bd180 0xc0287bd2c0 0xc0287bfb60] Scope:scope 0xc028786dc0 {
	func getLokiConfig
	type lokiConfig
	type labelSetJSON
}
 Imports:[0xc0287bf1a0 0xc0287bf1d0 0xc0287bf200 0xc0287bf230 0xc0287bf260 0xc0287bf290] Unresolved:[flagext time int model string string string string string string error flagext nil nil fmt strconv nil time time strconv nil json byte nil nil fmt make model model model nil] Comments:[]} 

But I have no idea what it means. 😓

[sh0rez]

Can you confirm the new code in fluent-bit folder is linted by the linter ?

It is

loki/Makefile

Lines 191 to 192 in a22be17

	lint:
	GOGC=20 golangci-lint run

This lints everything it can find

[cyriltovena]

It should be the import issue passed and I think there is more but the linter seems to be skipping.

Let me have quick look.

[cyriltovena]

You can remove the license file we have the same.

[cyriltovena]

Still not able to run the linter, Looking into a solution in the meantime there is still some issues I'd like you to take a look.

[cyriltovena]

@cosmo0920 I'm on vacation for 2 weeks, will review asap when I'm back.

[cosmo0920]

Yep, thank you for notifying it.
I'm also on vacation for a week and a half.

[cyriltovena]

I'm taking another look !

[cyriltovena]

Looks awesome just a last effort on the config to match fluent. This would ease transition between the two.

[JensErat]

When putting together a proof of concept towards Loki, we also had to extend the fluent-bit output plugin due to some issues. I will definitely extract and provide a new PR for some useful parts as soon as this PR is merged (especially added quite some testing code because debugging edge cases is a hassle with CGO shared objects).

Mostly, our issues boil down to fluent-bit records not necessarily being flat, but can be a tree-like structure. For example, the fluent-bit Kubernetes filter plugin will add meta information from Kubernetes (pod, namespace, ...) within a kubernetes "folder", and if you enable JSON parsing, whole nested JSON trees can be passed as record (think of Kubernetes audit logs that sometimes contain three or four times nested records).

This resulted in two issues for us:

• Extracting stream labels requires a non-trivial configuration. We went for JSON like this:

  {
    "TENANT": "tenant",
    "HOSTNAME": "hostname",
    "kubernetes": {
      "container_name": "container",
      "pod_name": "pod",
      "namespace_name": "namespace"
    },
    "BOOT_ID": "_drop",
    "CONTAINER_ID": "_drop",
    "CONTAINER_ID_FULL": "_drop",
    "CONTAINER_TAG": "_drop",
    "SYSTEMD_INVOCATION_ID": "_drop",
    "SELINUX_CONTEXT": "_drop",
    "CONTAINER_NAME": "_drop",
    "MACHINE_ID": "_drop"
  }

  The semantics of this is the key being a match on the fluent-bit record label, and the value either a nested sub-match (container_name from kubernetes dictionary), the target label (which is flat in Loki) or a specific drop annotation. This is easily implementable using a recursive function (we have partially test-backed prototype code). This is pretty much a tree-aware alternative to the fluentd-plugin remove_keys and label_keys configuration. The example above keeps TENANT and HOSTNAME in their lower-case variants, and renames kubernetes.pod_name etc. to simple pod labels (we see them in queries and write them in Grafana all the time, so rather keep them short).

  Surely this is also implementable as a more simple data structure, but going for a well-defined JSON-based configuration resolved lots of issues around escaping of special characters.

• The []byte/base64 conversion has also to be applied to keys and values beyond the outermost record layer. We wondered about lots of base64-encoded text until we realized this is based on nested labels.

I'm currently doing some minor cleanups and will push the code as-is to a fork. We would be glad if we can find some agreements on how to handle nested labels with both of the issues they bring -- and we're ready to provide code to get them running, we just have to align on how to do it properly.

[cyriltovena]

out of curiosity do you think the same issue also happen in fluentd plugin ? For promtail we have pipeline stage to parse json however we don't yet support drop.

This looks like a nice solution but we need to make sure we use the same config across the board unless there is a good argument against.

/cc @slim-bean @rfratto WDYT ?

[JensErat]

Thank you for the great work -- I didn't get round checking the code before now again. I really like the solution with separate relabel- and drop configuration though. At first I tried rebasing my changes against the current HEAD (as I somehow was not aware you reimplemented the small recursive function already) -- pretty much puzzled my because you came up with nearly 100% identical code but on the "wrong side" of the diffs until I realized what was going on... :)

I ported some of my edge case tests I wrote while drafting my code:

#1096