out_opentelemetry: add support for OTLP Logs

[Syn3rman]

Adds support for exporting Fluent Bit logs to OpenTelemetry collector using OTLP

Fixes #5559

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change

[INPUT]
	Name dummy
	Tag dummy.log
	Rate 3

[INPUT]
	name node_exporter_metrics
	tag node_metrics
	scrape_interval 2

[OUTPUT]
	name opentelemetry
	match *
	host 172.18.0.2
	port 4318
	logs_uri /v1/logs
	metrics_uri /v1/metrics

• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

[2022/07/19 09:45:04] [ info] [fluent bit] version=1.9.5, commit=238dbece1c, pid=79063
[2022/07/19 09:45:04] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/07/19 09:45:04] [ info] [cmetrics] version=0.3.1
[2022/07/19 09:45:04] [ info] [input:node_exporter_metrics:node_exporter_metrics.1] path.procfs = /proc
[2022/07/19 09:45:04] [ info] [input:node_exporter_metrics:node_exporter_metrics.1] path.sysfs  = /sys
[2022/07/19 09:45:04] [ info] [sp] stream processor started
[2022/07/19 09:45:06] [ info] [output:opentelemetry:opentelemetry.0] 172.18.0.2:4318, HTTP status=200
[2022/07/19 09:45:07] [ info] [output:opentelemetry:opentelemetry.0] 172.18.0.2:4318, HTTP status=200
[2022/07/19 09:45:07] [ info] [output:opentelemetry:opentelemetry.0] 172.18.0.2:4318, HTTP status=200
[2022/07/19 09:45:08] [ info] [output:opentelemetry:opentelemetry.0] 172.18.0.2:4318, HTTP status=200
[2022/07/19 09:45:09] [ info] [output:opentelemetry:opentelemetry.0] 172.18.0.2:4318, HTTP status=200
[2022/07/19 09:45:09] [ info] [output:opentelemetry:opentelemetry.0] 172.18.0.2:4318, HTTP status=200
[2022/07/19 09:45:10] [ info] [output:opentelemetry:opentelemetry.0] 172.18.0.2:4318, HTTP status=200
^C[2022/07/19 09:45:10] [engine] caught signal (SIGINT)
[2022/07/19 09:45:10] [ info] [input] pausing node_exporter_metrics.1
[2022/07/19 09:45:10] [ warn] [engine] service will shutdown in max 5 seconds
[2022/07/19 09:45:10] [ info] [output:opentelemetry:opentelemetry.0] 172.18.0.2:4318, HTTP status=200
[2022/07/19 09:45:10] [ info] [output:opentelemetry:opentelemetry.0] 172.18.0.2:4318, HTTP status=200
[2022/07/19 09:45:11] [ info] [engine] service has stopped (0 pending tasks)
==79063==
==79063== HEAP SUMMARY:
==79063==     in use at exit: 66,451 bytes in 2,263 blocks
==79063==   total heap usage: 56,284 allocs, 54,021 frees, 7,764,864 bytes allocated
==79063==
==79063== LEAK SUMMARY:
==79063==    definitely lost: 0 bytes in 0 blocks
==79063==    indirectly lost: 0 bytes in 0 blocks
==79063==      possibly lost: 0 bytes in 0 blocks
==79063==    still reachable: 66,451 bytes in 2,263 blocks
==79063==         suppressed: 0 bytes in 0 blocks
==79063== Rerun with --leak-check=full to see details of leaked memory
==79063==
==79063== For lists of detected and suppressed errors, rerun with: -s
==79063== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Attached local packaging test output showing all targets (including any new ones) build.

Documentation

• Documentation required for this feature

Documentation PR: fluent/fluent-bit-docs#852

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[edsiper]

@Syn3rman, some early feedback:

• please rebase on top of master
• do all code changes inside out_opentelemetry only (not cmetrics)

[edsiper]

@leonardo-albertovich is this PR approved ?

[edsiper]

we need a PR for the documentation

[Syn3rman]

@edsiper Added link to PR in description  ✅

[epsteina16]

Is support for traces for the opentelemetry output plugin currently under development?

[Syn3rman]

@epsteina16 yes

[svettwer]

Heyho! 👋

Thanks a lot @Syn3rman for adding this! 🙇
I've one question on testing this new feature:
With

docker run -p 127.0.0.1:24224:24224 ghcr.io/fluent/fluent-bit/master /fluent-bit/bin/fluent-bit -i forward -f 1 -o opentelemetry logs_uri="/v1/logs" -v

I still get

[2022/09/27 10:28:41] [debug] [router] data generated by forward.0 input are logs, but matching destination plugin opentelemetry.0 don't handle logs. Skipping destination.

Is there something missing to add this feature to the container build from master? Same goes for the staging container btw.
Thanks! 😃

[Syn3rman]

Hey @svettwer ,
Sorry for the late response 😓
My best guess would be that the url you are using defaults to master:latest which seems to have been published 5 months ago thus does not have this feature. Could you perhaps try using something like master:x86-64 instead which I have confirmed works with the following output:

[2022/10/05 06:28:29] [debug] [opentelemetry:opentelemetry.0] created event channels: read=24 write=25
[2022/10/05 06:28:29] [debug] [router] default match rule forward.0:opentelemetry.0

[svettwer]

Awesome! Thanks for your response! 😄

[svettwer]

Took me a while now to come back with feedback. master:x86-64 indeed contains the new logs implementation for the opentelemetry output plugin. Thanks again for your help! 😄