Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

processor_labels: Remove a needless existence check for insert operation #8824

Merged
merged 1 commit into from
May 15, 2024
Merged

processor_labels: Remove a needless existence check for insert operation #8824

merged 1 commit into from
May 15, 2024
+17 −16

Conversation

cosmo0920
Copy link
Contributor

@cosmo0920 cosmo0920 commented May 14, 2024 •
edited

Insert implies that adding new key-value pairs of labels. However, we did put the existence check for the keys of labels. This shouldn't be effective for such operation

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
service:
  flush: 2
  daemon: off
  log_level:  debug

pipeline:
  inputs:
    - name: fluentbit_metrics
      tag: fluentbit.metrics
      scrape_interval: 2

      processors:
        metrics:
          - name: labels
            insert: agent fluentbit


  outputs:
    - name: stdout
      match: '*'
  • Debug log output from testing the change
Fluent Bit v3.0.4
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________  
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \ 
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  < 
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/ 

[2024/05/14 22:11:43] [ info] Configuration:
[2024/05/14 22:11:43] [ info]  flush time     | 2.000000 seconds
[2024/05/14 22:11:43] [ info]  grace          | 5 seconds
[2024/05/14 22:11:43] [ info]  daemon         | 0
[2024/05/14 22:11:43] [ info] ___________
[2024/05/14 22:11:43] [ info]  inputs:
[2024/05/14 22:11:43] [ info]      fluentbit_metrics
[2024/05/14 22:11:43] [ info] ___________
[2024/05/14 22:11:43] [ info]  filters:
[2024/05/14 22:11:43] [ info] ___________
[2024/05/14 22:11:43] [ info]  outputs:
[2024/05/14 22:11:43] [ info]      stdout.0
[2024/05/14 22:11:43] [ info] ___________
[2024/05/14 22:11:43] [ info]  collectors:
[2024/05/14 22:11:43] [ info] [fluent bit] version=3.0.4, commit=549e7fbfc4, pid=337215
[2024/05/14 22:11:43] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/05/14 22:11:43] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/05/14 22:11:43] [ info] [cmetrics] version=0.9.0
[2024/05/14 22:11:43] [ info] [ctraces ] version=0.5.1
[2024/05/14 22:11:43] [ info] [input:fluentbit_metrics:fluentbit_metrics.0] initializing
[2024/05/14 22:11:43] [ info] [input:fluentbit_metrics:fluentbit_metrics.0] storage_strategy='memory' (memory only)
[2024/05/14 22:11:43] [debug] [fluentbit_metrics:fluentbit_metrics.0] created event channels: read=21 write=22
[2024/05/14 22:11:43] [debug] [stdout:stdout.0] created event channels: read=23 write=24
[2024/05/14 22:11:43] [ info] [sp] stream processor started
[2024/05/14 22:11:43] [ info] [output:stdout:stdout.0] worker #0 started
[2024/05/14 22:11:46] [debug] [task] created task=0x5f454e0 id=0 OK
[2024/05/14 22:11:46] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
2024-05-14T13:11:44.402905181Z fluentbit_uptime{hostname="cosmo-desktop2",agent="fluentbit"} = 2
2024-05-14T13:11:43.119393661Z fluentbit_input_bytes_total{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_records_total{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:44.400500038Z fluentbit_input_metrics_scrapes_total{name="fluentbit_metrics.0",agent="fluentbit"} = 1
2024-05-14T13:11:43.148140351Z fluentbit_output_proc_records_total{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_proc_bytes_total{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_errors_total{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_retries_total{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_retries_failed_total{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_dropped_records_total{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_retried_records_total{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:44.402905181Z fluentbit_process_start_time_seconds{hostname="cosmo-desktop2",agent="fluentbit"} = 1715692302
2024-05-14T13:11:44.402905181Z fluentbit_build_info{hostname="cosmo-desktop2",version="3.0.4",os="linux",agent="fluentbit"} = 1715692302
2024-05-14T13:11:44.402905181Z fluentbit_hot_reloaded_times{hostname="cosmo-desktop2",agent="fluentbit"} = 0
2024-05-14T13:11:44.405137050Z fluentbit_storage_chunks{agent="fluentbit"} = 0
2024-05-14T13:11:44.405137050Z fluentbit_storage_mem_chunks{agent="fluentbit"} = 0
2024-05-14T13:11:44.405137050Z fluentbit_storage_fs_chunks{agent="fluentbit"} = 0
2024-05-14T13:11:44.405137050Z fluentbit_storage_fs_chunks_up{agent="fluentbit"} = 0
2024-05-14T13:11:44.405137050Z fluentbit_storage_fs_chunks_down{agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_ingestion_paused{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_storage_overlimit{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_storage_memory_bytes{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_storage_chunks{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_storage_chunks_up{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_storage_chunks_down{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_storage_chunks_busy{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_storage_chunks_busy_bytes{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_upstream_total_connections{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_upstream_busy_connections{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_chunk_available_capacity_percent{name="stdout.0",agent="fluentbit"} = 100
[2024/05/14 22:11:46] [debug] [out flush] cb_destroy coro_id=0
[2024/05/14 22:11:46] [debug] [task] destroy task=0x5f454e0 (task_id=0)
^C[2024/05/14 22:11:47] [engine] caught signal (SIGINT)
[2024/05/14 22:11:47] [debug] [task] created task=0x6068210 id=0 OK
[2024/05/14 22:11:47] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
2024-05-14T13:11:46.418079697Z fluentbit_uptime{hostname="cosmo-desktop2",agent="fluentbit"} = 4
2024-05-14T13:11:43.119393661Z fluentbit_input_bytes_total{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_records_total{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:46.417894880Z fluentbit_input_metrics_scrapes_total{name="fluentbit_metrics.0",agent="fluentbit"} = 2
2024-05-14T13:11:43.148140351Z fluentbit_output_proc_records_total{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_proc_bytes_total{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_errors_total{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_retries_total{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_retries_failed_total{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_dropped_records_total[2024/05/14 22:11:47] [ warn] [engine] service will shutdown in max 5 seconds
[2024/05/14 22:11:47] [ info] [input] pausing fluentbit_metrics.0
{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_retried_records_total{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:46.418079697Z fluentbit_process_start_time_seconds{hostname="cosmo-desktop2",agent="fluentbit"} = 1715692302
2024-05-14T13:11:46.418079697Z fluentbit_build_info{hostname="cosmo-desktop2",version="3.0.4",os="linux",agent="fluentbit"} = 1715692302
2024-05-14T13:11:46.418079697Z fluentbit_hot_reloaded_times{hostname="cosmo-desktop2",agent="fluentbit"} = 0
2024-05-14T13:11:46.418434940Z fluentbit_storage_chunks{agent="fluentbit"} = 1
2024-05-14T13:11:46.418434940Z fluentbit_storage_mem_chunks{agent="fluentbit"} = 1
2024-05-14T13:11:46.418434940Z fluentbit_storage_fs_chunks{agent="fluentbit"} = 0
2024-05-14T13:11:46.418434940Z fluentbit_storage_fs_chunks_up{agent="fluentbit"} = 0
2024-05-14T13:11:46.418434940Z fluentbit_storage_fs_chunks_down{agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_ingestion_paused{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_storage_overlimit{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_storage_memory_bytes{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_storage_chunks{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_storage_chunks_up{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_storage_chunks_down{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_storage_chunks_busy{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.119393661Z fluentbit_input_storage_chunks_busy_bytes{name="fluentbit_metrics.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_upstream_total_connections{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_upstream_busy_connections{name="stdout.0",agent="fluentbit"} = 0
2024-05-14T13:11:43.148140351Z fluentbit_output_chunk_available_capacity_percent{name="stdout.0",agent="fluentbit"} = 100
[2024/05/14 22:11:47] [debug] [out flush] cb_destroy coro_id=1
[2024/05/14 22:11:47] [debug] [task] destroy task=0x6068210 (task_id=0)
[2024/05/14 22:11:48] [ info] [engine] service has stopped (0 pending tasks)
[2024/05/14 22:11:48] [ info] [input] pausing fluentbit_metrics.0
[2024/05/14 22:11:48] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/05/14 22:11:48] [ info] [output:stdout:stdout.0] thread worker #0 stopped
  • Attached Valgrind output that shows no leaks or memory corruption was found
==337215== 
==337215== HEAP SUMMARY:
==337215==     in use at exit: 0 bytes in 0 blocks
==337215==   total heap usage: 8,210 allocs, 8,210 frees, 2,566,010 bytes allocated
==337215== 
==337215== All heap blocks were freed -- no leaks are possible
==337215== 
==337215== For lists of detected and suppressed errors, rerun with: -s
==337215== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

fluent/fluent-bit-docs#1371

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@cosmo0920
processor_labels: Remove a needless existence check for insert operation
227b1e9 
Insert implies that adding new key-value pairs of labels.
However, we did put the existence check for the keys of labels.
This shouldn't be effective for such operation.

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
@cosmo0920 cosmo0920 force-pushed the cosmo0920-remove-needless-existence-checker-for-insert-operation branch from 2f9053b to 227b1e9 Compare May 14, 2024 13:17
@endersonmaia
Copy link
Contributor

Just tested with the image ghcr.io/fluent/fluent-bit/pr-8824:221606893d4ac979baca430c4b48312bf198461a and I can attest it's working as expected.

@cosmo0920 cosmo0920 marked this pull request as ready for review May 15, 2024 01:21
@cosmo0920 cosmo0920 mentioned this pull request May 15, 2024
Open
@edsiper edsiper merged commit bf04226 into master May 15, 2024
59 of 64 checks passed
@edsiper edsiper deleted the cosmo0920-remove-needless-existence-checker-for-insert-operation branch May 15, 2024 14:33
@edsiper
Copy link
Member

edsiper commented May 15, 2024

thank you both

@BrewTestBot BrewTestBot mentioned this pull request May 21, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@edsiper edsiper Awaiting requested review from edsiper edsiper is a code owner

@leonardo-albertovich leonardo-albertovich Awaiting requested review from leonardo-albertovich leonardo-albertovich is a code owner

@fujimotos fujimotos Awaiting requested review from fujimotos fujimotos is a code owner

@koleini koleini Awaiting requested review from koleini koleini is a code owner

Assignees
No one assigned
Labels
ci/integration-docker-ok docs-required ok-to-test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@cosmo0920 @endersonmaia @edsiper @patrick-stephens