out_loki: add compress option for gzip #7949

ksauzz · 2023-09-20T09:51:55Z

Add compress option for gzip compression in out_loki

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

[N/A] Example configuration file for the change

Run loki

docker run --rm --net=host grafana/loki

[N/A] Debug log output from testing the change

I'm not sure what I should write here. Any example?

Attached Valgrind output that shows no leaks or memory corruption was found

with gzip

% valgrind  bin/fluent-bit -i dummy -p rate=60 -o loki -p match='*' -p labels="job=gzip" -p compress=gzip

==61990== Memcheck, a memory error detector
==61990== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==61990== Using Valgrind-3.21.0 and LibVEX; rerun with -h for copyright info
==61990== Command: bin/fluent-bit -i dummy -p rate=60 -o loki -p match=* -p labels=job=gzip -p compress=gzip
==61990== 
Fluent Bit v2.1.10
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/09/20 21:11:00] [ info] [fluent bit] version=2.1.10, commit=47b42dd4df, pid=61990
[2023/09/20 21:11:00] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/20 21:11:00] [ info] [cmetrics] version=0.6.3
[2023/09/20 21:11:00] [ info] [ctraces ] version=0.3.1
[2023/09/20 21:11:00] [ info] [input:dummy:dummy.0] initializing
[2023/09/20 21:11:00] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/09/20 21:11:00] [ info] [output:loki:loki.0] configured, hostname=127.0.0.1:3100
[2023/09/20 21:11:00] [ info] [sp] stream processor started
^C[2023/09/20 21:11:13] [engine] caught signal (SIGINT)
[2023/09/20 21:11:13] [ warn] [engine] service will shutdown in max 5 seconds
[2023/09/20 21:11:13] [ info] [input] pausing dummy.0
[2023/09/20 21:11:13] [ info] [engine] service has stopped (0 pending tasks)
[2023/09/20 21:11:13] [ info] [input] pausing dummy.0
==61990== 
==61990== HEAP SUMMARY:
==61990==     in use at exit: 0 bytes in 0 blocks
==61990==   total heap usage: 9,767 allocs, 9,767 frees, 23,202,347 bytes allocated
==61990== 
==61990== All heap blocks were freed -- no leaks are possible
==61990== 
==61990== For lists of detected and suppressed errors, rerun with: -s
==61990== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

without gzip

% valgrind  bin/fluent-bit -i dummy -p rate=60 -o loki -p match='*' -p labels="job=plain"                  

==64340== Memcheck, a memory error detector
==64340== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==64340== Using Valgrind-3.21.0 and LibVEX; rerun with -h for copyright info
==64340== Command: bin/fluent-bit -i dummy -p rate=60 -o loki -p match=* -p labels=job=plain
==64340== 
Fluent Bit v2.1.10
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/09/20 21:11:50] [ info] [fluent bit] version=2.1.10, commit=47b42dd4df, pid=64340
[2023/09/20 21:11:50] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/20 21:11:50] [ info] [cmetrics] version=0.6.3
[2023/09/20 21:11:50] [ info] [ctraces ] version=0.3.1
[2023/09/20 21:11:50] [ info] [input:dummy:dummy.0] initializing
[2023/09/20 21:11:50] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/09/20 21:11:50] [ info] [output:loki:loki.0] configured, hostname=127.0.0.1:3100
[2023/09/20 21:11:50] [ info] [sp] stream processor started
^C[2023/09/20 21:12:04] [engine] caught signal (SIGINT)
[2023/09/20 21:12:04] [ warn] [engine] service will shutdown in max 5 seconds
[2023/09/20 21:12:04] [ info] [input] pausing dummy.0
[2023/09/20 21:12:04] [ info] [engine] service has stopped (0 pending tasks)
[2023/09/20 21:12:04] [ info] [input] pausing dummy.0
==64340== 
==64340== HEAP SUMMARY:
==64340==     in use at exit: 0 bytes in 0 blocks
==64340==   total heap usage: 10,984 allocs, 10,984 frees, 21,295,881 bytes allocated
==64340== 
==64340== All heap blocks were freed -- no leaks are possible
==64340== 
==64340== For lists of detected and suppressed errors, rerun with: -s
==64340== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

how to query loki

with gzip

% curl -G -s  "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query={job="gzip"}' --data 'limit=2' | jq '.data.result'
[
  {
    "stream": {
      "job": "gzip"
    },
    "values": [
      [
        "1695211873416884326",
        "{\"message\":\"dummy\"}"
      ],
      [
        "1695211873400239362",
        "{\"message\":\"dummy\"}"
      ]
    ]
  }
]

without gzip

% curl -G -s  "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query={job="plain"}' --data 'limit=2' | jq '.data.result'
[
  {
    "stream": {
      "job": "plain"
    },
    "values": [
      [
        "1695211924850171543",
        "{\"message\":\"dummy\"}"
      ],
      [
        "1695211924833646667",
        "{\"message\":\"dummy\"}"
      ]
    ]
  }
]

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

[N/A] Run local packaging test showing all targets (including any new ones) build.
[N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

fluent/fluent-bit-docs#1199

Backporting

[N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

ksauzz · 2023-09-20T11:56:55Z

Removed unnecessary changes.

cosmo0920

I added comments for the first impressions.

plugins/out_loki/loki.c

ksauzz · 2023-09-20T13:39:40Z

Thank you for the review. I pushed the changes.

with gzip

% valgrind  bin/fluent-bit -i dummy -p rate=60 -o loki -p match='*' -p labels="job=gzip" -p compress=gzip

==90770== Memcheck, a memory error detector
==90770== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==90770== Using Valgrind-3.21.0 and LibVEX; rerun with -h for copyright info
==90770== Command: bin/fluent-bit -i dummy -p rate=60 -o loki -p match=* -p labels=job=gzip -p compress=gzip
==90770== 
Fluent Bit v2.1.10
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/09/20 22:36:56] [ info] [fluent bit] version=2.1.10, commit=47b42dd4df, pid=90770
[2023/09/20 22:36:56] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/20 22:36:56] [ info] [cmetrics] version=0.6.3
[2023/09/20 22:36:56] [ info] [ctraces ] version=0.3.1
[2023/09/20 22:36:56] [ info] [input:dummy:dummy.0] initializing
[2023/09/20 22:36:56] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/09/20 22:36:56] [ info] [output:loki:loki.0] configured, hostname=127.0.0.1:3100
[2023/09/20 22:36:56] [ info] [sp] stream processor started
^C[2023/09/20 22:37:10] [engine] caught signal (SIGINT)
[2023/09/20 22:37:10] [ warn] [engine] service will shutdown in max 5 seconds
[2023/09/20 22:37:10] [ info] [input] pausing dummy.0
[2023/09/20 22:37:11] [ info] [engine] service has stopped (0 pending tasks)
[2023/09/20 22:37:11] [ info] [input] pausing dummy.0
==90770== 
==90770== HEAP SUMMARY:
==90770==     in use at exit: 0 bytes in 0 blocks
==90770==   total heap usage: 10,385 allocs, 10,385 frees, 24,903,131 bytes allocated
==90770== 
==90770== All heap blocks were freed -- no leaks are possible
==90770== 
==90770== For lists of detected and suppressed errors, rerun with: -s
==90770== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

% curl -G -s  "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query={job="gzip"}' --data 'limit=2' | jq '.data.result' 
[
  {
    "stream": {
      "job": "gzip"
    },
    "values": [
      [
        "1695217030265459845",
        "{\"message\":\"dummy\"}"
      ],
      [
        "1695217030243066342",
        "{\"message\":\"dummy\"}"
      ]
    ]
  }
]

without gzip

% valgrind bin/fluent-bit -i dummy -p rate=60 -o loki -p match='*' -p labels="job=plain"                 

==89675== Memcheck, a memory error detector
==89675== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==89675== Using Valgrind-3.21.0 and LibVEX; rerun with -h for copyright info
==89675== Command: bin/fluent-bit -i dummy -p rate=60 -o loki -p match=* -p labels=job=plain
==89675== 
Fluent Bit v2.1.10
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/09/20 22:36:31] [ info] [fluent bit] version=2.1.10, commit=47b42dd4df, pid=89675
[2023/09/20 22:36:32] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/20 22:36:32] [ info] [cmetrics] version=0.6.3
[2023/09/20 22:36:32] [ info] [ctraces ] version=0.3.1
[2023/09/20 22:36:32] [ info] [input:dummy:dummy.0] initializing
[2023/09/20 22:36:32] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/09/20 22:36:32] [ info] [output:loki:loki.0] configured, hostname=127.0.0.1:3100
[2023/09/20 22:36:32] [ info] [sp] stream processor started
^C[2023/09/20 22:36:46] [engine] caught signal (SIGINT)
[2023/09/20 22:36:46] [ warn] [engine] service will shutdown in max 5 seconds
[2023/09/20 22:36:46] [ info] [input] pausing dummy.0
[2023/09/20 22:36:47] [ info] [engine] service has stopped (0 pending tasks)
[2023/09/20 22:36:47] [ info] [input] pausing dummy.0
==89675== 
==89675== HEAP SUMMARY:
==89675==     in use at exit: 0 bytes in 0 blocks
==89675==   total heap usage: 10,736 allocs, 10,736 frees, 21,130,607 bytes allocated
==89675== 
==89675== All heap blocks were freed -- no leaks are possible
==89675== 
==89675== For lists of detected and suppressed errors, rerun with: -s
==89675== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

% curl -G -s  "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query={job="plain"}' --data 'limit=2' | jq '.data.result'
[
  {
    "stream": {
      "job": "plain"
    },
    "values": [
      [
        "1695217006259634558",
        "{\"message\":\"dummy\"}"
      ],
      [
        "1695217006242857311",
        "{\"message\":\"dummy\"}"
      ]
    ]
  }
]

cosmo0920 · 2023-09-20T14:29:54Z

It looks great. Thanks for addressing my comments. I'll confirm this PR with the actual Loki and associated Grafana instances tomorrow.

cosmo0920 · 2023-09-21T01:57:20Z

I confirmed that your PR is working as expected:

Command

% bin/fluent-bit -i dummy -o loki 
Fluent Bit v2.1.10
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/09/21 10:54:47] [ info] [fluent bit] version=2.1.10, commit=398f4346ae, pid=2472415
[2023/09/21 10:54:47] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/21 10:54:47] [ info] [cmetrics] version=0.6.3
[2023/09/21 10:54:47] [ info] [ctraces ] version=0.3.1
[2023/09/21 10:54:47] [ info] [input:dummy:dummy.0] initializing
[2023/09/21 10:54:47] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/09/21 10:54:47] [ info] [output:loki:loki.0] configured, hostname=127.0.0.1:3100
[2023/09/21 10:54:47] [ info] [sp] stream processor started
^C[2023/09/21 10:54:50] [engine] caught signal (SIGINT)
[2023/09/21 10:54:50] [ warn] [engine] service will shutdown in max 5 seconds
[2023/09/21 10:54:50] [ info] [input] pausing dummy.0
[2023/09/21 10:54:51] [ info] [engine] service has stopped (0 pending tasks)
[2023/09/21 10:54:51] [ info] [input] pausing dummy.0
% bin/fluent-bit -i dummy -o loki -pcompress=gzip
Fluent Bit v2.1.10
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/09/21 10:54:52] [ info] [fluent bit] version=2.1.10, commit=398f4346ae, pid=2472520
[2023/09/21 10:54:52] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/21 10:54:52] [ info] [cmetrics] version=0.6.3
[2023/09/21 10:54:52] [ info] [ctraces ] version=0.3.1
[2023/09/21 10:54:52] [ info] [input:dummy:dummy.0] initializing
[2023/09/21 10:54:52] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/09/21 10:54:52] [ info] [output:loki:loki.0] configured, hostname=127.0.0.1:3100
[2023/09/21 10:54:52] [ info] [sp] stream processor started
^C[2023/09/21 10:54:55] [engine] caught signal (SIGINT)
[2023/09/21 10:54:55] [ warn] [engine] service will shutdown in max 5 seconds
[2023/09/21 10:54:55] [ info] [input] pausing dummy.0
[2023/09/21 10:54:55] [ info] [engine] service has stopped (0 pending tasks)
[2023/09/21 10:54:55] [ info] [input] pausing dummy.0

Result on Loki and associated Grafana instances

cosmo0920

Honestly speaking, I'm also writing for gzip compression patch. But, I love to use yours.
This approach looks better than mine. Thanks for your contribution. 👍

cosmo0920

Oh, one thing. This patch looks great but the commit message does not follow our guideline.
So, could you use out_loki: ... style instead of feat(out_loki): ...?

Signed-off-by: Kazuhiro Suzuki <ksauzzmsg@gmail.com>

cosmo0920 · 2023-09-21T02:11:26Z

Waiting for CI results... And thanks for the quick response. 💪

ksauzz · 2023-09-21T02:12:04Z

Sorry for the overlap in the task. I appreciate your help. 🙏

cosmo0920 · 2023-09-21T02:20:35Z

No worries. Encouraging for a new contributor is also important for us. 😄

edsiper · 2023-09-21T04:10:37Z

thank you!

--------- Signed-off-by: Kazuhiro Suzuki <ksauzzmsg@gmail.com>

github-actions bot added the docs-required label Sep 20, 2023

ksauzz force-pushed the feat/loki-gzip branch 2 times, most recently from a6acd89 to b69692c Compare September 20, 2023 10:25

ksauzz mentioned this pull request Sep 20, 2023

feat(out_loki): add compress option fluent/fluent-bit-docs#1199

Merged

ksauzz marked this pull request as ready for review September 20, 2023 10:43

ksauzz requested review from edsiper, leonardo-albertovich, fujimotos and koleini as code owners September 20, 2023 10:43

ksauzz force-pushed the feat/loki-gzip branch from b69692c to ef05cc4 Compare September 20, 2023 11:52

cosmo0920 requested changes Sep 20, 2023

View reviewed changes

ksauzz temporarily deployed to pr September 21, 2023 02:00 — with GitHub Actions Inactive

cosmo0920 approved these changes Sep 21, 2023

View reviewed changes

cosmo0920 requested changes Sep 21, 2023

View reviewed changes

ksauzz added 3 commits September 21, 2023 11:07

out_loki: add compress option for gzip

cf81573

Signed-off-by: Kazuhiro Suzuki <ksauzzmsg@gmail.com>

out_loki: fix per review

50b3329

Signed-off-by: Kazuhiro Suzuki <ksauzzmsg@gmail.com>

out_loki: extract function

c887734

Signed-off-by: Kazuhiro Suzuki <ksauzzmsg@gmail.com>

ksauzz force-pushed the feat/loki-gzip branch from 398f434 to c887734 Compare September 21, 2023 02:08

ksauzz temporarily deployed to pr September 21, 2023 02:10 — with GitHub Actions Inactive

cosmo0920 changed the title ~~feat(out_loki): add compress option for gzip~~ out_loki: add compress option for gzip Sep 21, 2023

ksauzz temporarily deployed to pr September 21, 2023 02:39 — with GitHub Actions Inactive

cosmo0920 approved these changes Sep 21, 2023

View reviewed changes

edsiper merged commit 38145ed into fluent:master Sep 21, 2023
41 of 43 checks passed

ksauzz deleted the feat/loki-gzip branch September 21, 2023 05:09

Porkepix mentioned this pull request Sep 28, 2023

fluent-bit 2.1.10 Homebrew/homebrew-core#146545

Merged

leonardo-albertovich pushed a commit that referenced this pull request Oct 5, 2023

out_loki: add compress option for gzip (#7949)

142e1d1

--------- Signed-off-by: Kazuhiro Suzuki <ksauzzmsg@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

out_loki: add compress option for gzip #7949

out_loki: add compress option for gzip #7949

ksauzz commented Sep 20, 2023 •

edited

ksauzz commented Sep 20, 2023

cosmo0920 left a comment

ksauzz commented Sep 20, 2023

cosmo0920 commented Sep 20, 2023

cosmo0920 commented Sep 21, 2023

cosmo0920 left a comment

cosmo0920 left a comment

cosmo0920 commented Sep 21, 2023

ksauzz commented Sep 21, 2023 •

edited

cosmo0920 commented Sep 21, 2023 •

edited

edsiper commented Sep 21, 2023

out_loki: add compress option for gzip #7949

out_loki: add compress option for gzip #7949

Conversation

ksauzz commented Sep 20, 2023 • edited

Run loki

how to query loki

ksauzz commented Sep 20, 2023

cosmo0920 left a comment

Choose a reason for hiding this comment

ksauzz commented Sep 20, 2023

cosmo0920 commented Sep 20, 2023

cosmo0920 commented Sep 21, 2023

Command

Result on Loki and associated Grafana instances

cosmo0920 left a comment

Choose a reason for hiding this comment

cosmo0920 left a comment

Choose a reason for hiding this comment

cosmo0920 commented Sep 21, 2023

ksauzz commented Sep 21, 2023 • edited

cosmo0920 commented Sep 21, 2023 • edited

edsiper commented Sep 21, 2023

ksauzz commented Sep 20, 2023 •

edited

ksauzz commented Sep 21, 2023 •

edited

cosmo0920 commented Sep 21, 2023 •

edited