filter_watermark: The initial commit for filter watermark #3459

ginobiliwang · 2021-05-08T06:18:13Z

Signed-off-by: Fenggang ginobiliwang@gmail.com

This is initial commit for watermark filter feature.
The idea is from a paper (The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing) from Google.
From stream processing perspective current function is limited. It has a fixed watermark. It can deal with out of order data. It has a batch window. It has tolerance for data latency.
Document PR for this filter watermark feature would be submitted soon.
The little demo and valgrind output below are for your reference.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@localhost ~]# nc 127.0.0.1 5170
{"time":"2021-11-12 18:30:00"}
{"time":"2021-11-12 18:30:01"}
{"time":"2021-11-12 18:30:02"}
{"time":"2021-11-12 18:30:07"}
{"time":"2021-11-12 18:30:06"}
{"time":"2021-11-12 18:30:04"}
{"time":"2021-11-12 18:30:05"}
{"time":"2021-11-12 18:31:00"}

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@localhost build]# valgrind --leak-check=full bin/fluent-bit -i tcp -F watermark -p 'time_field=time' 'watermark=20' -m '*' -o stdout
==3805== Memcheck, a memory error detector
==3805== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==3805== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==3805== Command: bin/fluent-bit -i tcp -F watermark -p time_field=time watermark=20 -m * -o stdout
==3805==
Fluent Bit v1.7.4

Copyright (C) 2019-2021 The Fluent Bit Authors
Copyright (C) 2015-2018 Treasure Data
Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
https://fluentbit.io

[2021/05/08 22:12:14] [ info] [engine] started (pid=3805)
[2021/05/08 22:12:14] [ info] [storage] version=1.1.1, initializing...
[2021/05/08 22:12:14] [ info] [storage] in-memory
[2021/05/08 22:12:14] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/05/08 22:12:14] [ info] [input:tcp:tcp.0] listening on 0.0.0.0:5170
[2021/05/08 22:12:14] [ info] [sp] stream processor started
==3805== Warning: client switching stacks? SP change: 0x9bc2788 --> 0xa48a850
==3805== to suppress, use: --max-stackframe=9208008 or greater
==3805== Warning: client switching stacks? SP change: 0xa48a7b8 --> 0x9bc2788
==3805== to suppress, use: --max-stackframe=9207856 or greater
==3805== Warning: client switching stacks? SP change: 0x9bc2788 --> 0xa48a7b8
==3805== to suppress, use: --max-stackframe=9207856 or greater
==3805== further instances of this message will not be shown.
[0] tcp.0: [1620483146.702023972, {"time"=>"2021-11-12 18:30:00"}]
[1] tcp.0: [1620483150.354546260, {"time"=>"2021-11-12 18:30:01"}]
[2] tcp.0: [1620483153.771547131, {"time"=>"2021-11-12 18:30:02"}]
[3] tcp.0: [1620483176.174986006, {"time"=>"2021-11-12 18:30:04"}]
[4] tcp.0: [1620483180.376079757, {"time"=>"2021-11-12 18:30:05"}]
[5] tcp.0: [1620483169.903685948, {"time"=>"2021-11-12 18:30:06"}]
[6] tcp.0: [1620483163.781975807, {"time"=>"2021-11-12 18:30:07"}]
^C[2021/05/08 22:13:19] [engine] caught signal (SIGINT)
[2021/05/08 22:13:19] [ warn] [engine] service will stop in 5 seconds
[2021/05/08 22:13:24] [ info] [engine] service stopped
==3805==
==3805== HEAP SUMMARY:
==3805== in use at exit: 99,816 bytes in 3,380 blocks
==3805== total heap usage: 4,229 allocs, 849 frees, 5,092,444 bytes allocated
==3805==
==3805== LEAK SUMMARY:
==3805== definitely lost: 0 bytes in 0 blocks
==3805== indirectly lost: 0 bytes in 0 blocks
==3805== possibly lost: 0 bytes in 0 blocks
==3805== still reachable: 99,816 bytes in 3,380 blocks
==3805== suppressed: 0 bytes in 0 blocks
==3805== Reachable blocks (those to which a pointer was found) are not shown.
==3805== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==3805==
==3805== For lists of detected and suppressed errors, rerun with: -s
==3805== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
[root@localhost build]#

Signed-off-by: Fenggang <ginobiliwang@gmail.com>

ginobiliwang · 2021-05-08T06:21:01Z

Hi @koleini & @edsiper, this is the feature i have discussed with you before. Now it is ready to be reviewed.

ginobiliwang · 2021-05-13T03:11:37Z

fluent/fluent-bit-docs#530

nigels-com · 2021-05-28T06:28:09Z

plugins/CMakeLists.txt

 REGISTER_OUT_PLUGIN("out_cloudwatch_logs")
 REGISTER_OUT_PLUGIN("out_kinesis_firehose")
 REGISTER_OUT_PLUGIN("out_kinesis_streams")
-REGISTER_OUT_PLUGIN("out_prometheus_exporter")


Intentional change?

Could be some merge issue, let me fix it.

Still looks wrong in the diff, does this need rebasing?

ok, i will rebase it in a whole after Masoud's initial review.

nigels-com · 2021-05-28T06:33:27Z

Is there an well known and tested 3rd party heap implementation, as an alternative?

ginobiliwang · 2021-05-31T13:00:50Z

Is there an well known and tested 3rd party heap implementation, as an alternative?

Most of implementations are customized versions. For this plugin, i have done customization as well.

github-actions · 2021-07-01T01:48:50Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

ginobiliwang · 2021-07-11T00:48:49Z

@edsiper @koleini , could you please review this PR ?

github-actions · 2021-08-11T01:47:29Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

edsiper · 2021-12-12T23:21:34Z

@koleini ping

github-actions · 2022-03-15T02:00:50Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

ginobiliwang · 2022-07-21T00:02:32Z

@koleini ,can you help to review this PR ?

koleini · 2022-07-29T21:47:43Z

This pull request requires documentation and unit tests to be included. In addition, there are indentation problems in the code that needs to be fixed.

Please see https://github.com/fluent/fluent-bit/blob/master/CONTRIBUTING.md for the guidelines.

ginobiliwang · 2022-08-01T10:04:14Z

Hi @koleini , this is the doc PR fluent/fluent-bit-docs#530 .

Let me fix the format issue and fill up unit tests.

Signed-off-by: Fenggang <ginobiliwang@gmail.com>

leonardo-albertovich · 2023-05-03T11:15:19Z

The binary tree we use is this well known red/black binary tree implementation https://github.com/fluent/fluent-bit/tree/master/lib/rbtree

ginobiliwang · 2023-05-03T11:47:51Z

I did not notice we have rbtree in place to use. And definitely it is ok to use rbtree for current feature.

leonardo-albertovich · 2023-05-03T13:32:19Z

I'm glad you find it fitting, I agree and would really appreciate it if at some point you updated your code to use it.

Thanks a lot!

github-actions · 2023-08-03T01:52:09Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

ginobiliwang · 2023-08-06T06:25:32Z

will continue to finish this pr soon, tied up by some other affairs.

github-actions · 2023-11-06T01:50:12Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

ginobiliwang · 2023-11-06T03:06:00Z

sorry for the delay, will catch up ASAP.

ginobiliwang · 2023-12-19T01:52:38Z

HI @leonardo-albertovich , i have tried to use rbtree in fluent bit repo to failed to apply. The main issue is that rbtree could not handle duplicate key. In this PR, it would deal with records with duplicate keys. So i have to revert back to use heap data structure to implement it, i think heap is the best data structure to do the job.

BTW, i have accidently closed this PR, would you please help to reopen it.

ginobiliwang · 2024-02-23T04:08:45Z

Hi @leonardo-albertovich , @koleini , @edsiper any chance to reopen this PR ? i can not re-open it by myself.

lecaros · 2024-05-15T01:22:12Z

ping @ginobiliwang

ginobiliwang · 2024-05-23T06:37:36Z

ping @ginobiliwang

Thanks @lecaros for reopening the PR. The same question i have post last year, re-paste here.
=======>
i have tried to use rbtree in fluent bit repo , but failed to apply. The main issue is that rbtree could not handle duplicate key. In this PR, it would need to deal with records with duplicate keys. So i have to revert back to use heap data structure to implement it, i think heap is the best data structure to do the job.
=======>

Please review the above, thanks.

lecaros · 2024-05-23T15:56:36Z

pinging @edsiper @leonardo-albertovich and @cosmo0920 to help you out with your question

github-actions · 2024-11-26T02:07:35Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

ginobiliwang · 2024-11-26T02:39:57Z

pinging @edsiper @leonardo-albertovich and @cosmo0920

github-actions · 2025-02-27T02:04:46Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

github-actions · 2025-08-27T02:10:03Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

filter_watermark: The initial commit for filter watermark

935d558

Signed-off-by: Fenggang <ginobiliwang@gmail.com>

github-actions bot added the docs-required label May 8, 2021

ginobiliwang mentioned this pull request May 13, 2021

filter_watermark: The initial commit to add docs fluent/fluent-bit-docs#530

Closed

nigels-com reviewed May 28, 2021

View reviewed changes

github-actions bot added the Stale label Jul 1, 2021

github-actions bot removed the Stale label Jul 11, 2021

github-actions bot added the Stale label Aug 11, 2021

github-actions bot removed the Stale label Dec 15, 2021

github-actions bot added the Stale label Mar 15, 2022

github-actions bot removed the Stale label Jul 21, 2022

format correction and merged with master stream

7c55f3f

ginobiliwang requested review from edsiper, fujimotos, koleini, leonardo-albertovich and niedbalski as code owners August 30, 2022 06:59

ginobiliwang added 2 commits August 30, 2022 15:06

Merge master change

80709b5

this commit is to merge with master stream

88f9f48

Signed-off-by: Fenggang <ginobiliwang@gmail.com>

github-actions bot removed the Stale label May 4, 2023

github-actions bot added the Stale label Aug 3, 2023

github-actions bot removed the Stale label Aug 7, 2023

github-actions bot added the Stale label Nov 6, 2023

github-actions bot removed the Stale label Nov 7, 2023

ginobiliwang closed this Dec 19, 2023

lecaros reopened this May 15, 2024

lecaros requested review from celalettin1286 and patrick-stephens as code owners May 15, 2024 01:21

github-actions bot added the Stale label Nov 26, 2024

github-actions bot removed the Stale label Nov 28, 2024

github-actions bot added the Stale label Feb 27, 2025

github-actions bot removed the Stale label Mar 21, 2025

github-actions bot added the Stale label Aug 27, 2025

filter_watermark: The initial commit for filter watermark #3459

Are you sure you want to change the base?

filter_watermark: The initial commit for filter watermark #3459

Uh oh!

Conversation

ginobiliwang commented May 8, 2021

Uh oh!

ginobiliwang commented May 8, 2021

Uh oh!

ginobiliwang commented May 13, 2021

Uh oh!

nigels-com May 28, 2021

Choose a reason for hiding this comment

Uh oh!

ginobiliwang May 31, 2021

Choose a reason for hiding this comment

Uh oh!

nigels-com Aug 10, 2022

Choose a reason for hiding this comment

Uh oh!

ginobiliwang Aug 16, 2022

Choose a reason for hiding this comment

Uh oh!

nigels-com commented May 28, 2021

Uh oh!

ginobiliwang commented May 31, 2021

Uh oh!

github-actions bot commented Jul 1, 2021

Uh oh!

ginobiliwang commented Jul 11, 2021

Uh oh!

github-actions bot commented Aug 11, 2021

Uh oh!

edsiper commented Dec 12, 2021

Uh oh!

github-actions bot commented Mar 15, 2022

Uh oh!

ginobiliwang commented Jul 21, 2022

Uh oh!

koleini commented Jul 29, 2022

Uh oh!

ginobiliwang commented Aug 1, 2022

Uh oh!

leonardo-albertovich commented May 3, 2023

Uh oh!

ginobiliwang commented May 3, 2023

Uh oh!

leonardo-albertovich commented May 3, 2023

Uh oh!

github-actions bot commented Aug 3, 2023

Uh oh!

ginobiliwang commented Aug 6, 2023

Uh oh!

github-actions bot commented Nov 6, 2023

Uh oh!

ginobiliwang commented Nov 6, 2023

Uh oh!

ginobiliwang commented Dec 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ginobiliwang commented Feb 23, 2024

Uh oh!

lecaros commented May 15, 2024

Uh oh!

ginobiliwang commented May 23, 2024

Uh oh!

lecaros commented May 23, 2024

Uh oh!

github-actions bot commented Nov 26, 2024

Uh oh!

ginobiliwang commented Nov 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 27, 2025

Uh oh!

github-actions bot commented Aug 27, 2025

Uh oh!

Reviewers

Assignees

Labels

ginobiliwang commented Dec 19, 2023 •

edited

Loading

ginobiliwang commented Nov 26, 2024 •

edited

Loading