parser: add Time_System_Timezone #8164

braydonk · 2023-11-13T01:06:10Z

Added a new config option for parsers called time_system_timezone which will fall back to mktime for tm to time conversion which will force the usage of the system timezone.

Fixes #593

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change

a.txt

{"message": "test", "time": "2023-11-11:13:00:00"}
{"message": "test2", "time": "2023-10-17:00:00:00"}

parser.conf

[PARSER]
    Name        no_timezone
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%d:%H:%M:%S
    Time_System_Timezone true

config.conf

[SERVICE]
  Log_Level debug

[INPUT]
    Name           tail
    Tag            test
    Exit_On_Eof    true
    Read_From_Head true
    Path           a.txt
    Key            log

[FILTER]
    Name     parser
    Match    *
    Parser   no_timezone 
    Key_Name log

[OUTPUT]
    Name   stdout
    Match  *

Debug log output from testing the change

Linux

braydonk@braydonk:~/Git/fluent-bit$ ./build/bin/fluent-bit --conf ~/Documents/fluent-bit-configs/config.conf --parser ~/Documents/fluent-bit-configs/parser.conf 
Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/11/12 19:30:17] [ info] Configuration:
[2023/11/12 19:30:17] [ info]  flush time     | 1.000000 seconds
[2023/11/12 19:30:17] [ info]  grace          | 5 seconds
[2023/11/12 19:30:17] [ info]  daemon         | 0
[2023/11/12 19:30:17] [ info] ___________
[2023/11/12 19:30:17] [ info]  inputs:
[2023/11/12 19:30:17] [ info]      tail
[2023/11/12 19:30:17] [ info] ___________
[2023/11/12 19:30:17] [ info]  filters:
[2023/11/12 19:30:17] [ info]      parser.0
[2023/11/12 19:30:17] [ info] ___________
[2023/11/12 19:30:17] [ info]  outputs:
[2023/11/12 19:30:17] [ info]      stdout.0
[2023/11/12 19:30:17] [ info] ___________
[2023/11/12 19:30:17] [ info]  collectors:
[2023/11/12 19:30:17] [ info] [fluent bit] version=2.2.0, commit=1c4c9ec2e3, pid=385488
[2023/11/12 19:30:17] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/11/12 19:30:17] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/11/12 19:30:17] [ info] [cmetrics] version=0.6.3
[2023/11/12 19:30:17] [ info] [ctraces ] version=0.3.1
[2023/11/12 19:30:17] [ info] [input:tail:tail.0] initializing
[2023/11/12 19:30:17] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2023/11/12 19:30:17] [debug] [tail:tail.0] created event channels: read=21 write=22
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] flb_tail_fs_inotify_init() initializing inotify tail input
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] inotify watch fd=27
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] scanning path a.txt
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] inode=5906182 with offset=0 appended as a.txt
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] scan_glob add(): a.txt, inode 5906182
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] 1 new files found on path 'a.txt'
[2023/11/12 19:30:17] [debug] [stdout:stdout.0] created event channels: read=29 write=30
[2023/11/12 19:30:17] [ info] [sp] stream processor started
[2023/11/12 19:30:17] [ info] [output:stdout:stdout.0] worker #0 started
[2023/11/12 19:30:17] [debug] [input chunk] update output instances with new chunk size diff=71, records=2, input=tail.0
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] [static files] processed 103b
[2023/11/12 19:30:17] [ info] [input:tail:tail.0] inode=5906182 file=a.txt ended, stop
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] inode=5906182 file=a.txt promote to TAIL_EVENT
[2023/11/12 19:30:17] [ info] [input:tail:tail.0] inotify_fs_add(): inode=5906182 watch_fd=1 name=a.txt
[2023/11/12 19:30:17] [debug] [input:tail:tail.0] [static files] processed 0b, done
[2023/11/12 19:30:17] [debug] [task] created task=0x7f0664038ea0 id=0 OK
[2023/11/12 19:30:17] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2023/11/12 19:30:17] [ warn] [engine] service will shutdown in max 5 seconds
[2023/11/12 19:30:17] [ info] [input] pausing tail.0
[0] test: [[1699725600.000000000, {}], {"message"=>"test"}]
[1] test: [[1697515200.000000000, {}], {"message"=>"test2"}]
[2023/11/12 19:30:17] [debug] [out flush] cb_destroy coro_id=0
[2023/11/12 19:30:17] [debug] [task] destroy task=0x7f0664038ea0 (task_id=0)
[2023/11/12 19:30:18] [ info] [engine] service has stopped (0 pending tasks)
[2023/11/12 19:30:18] [ info] [input] pausing tail.0
[2023/11/12 19:30:18] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/11/12 19:30:18] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2023/11/12 19:30:18] [debug] [input:tail:tail.0] inode=5906182 removing file name a.txt
[2023/11/12 19:30:18] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=5906182 watch_fd=1
braydonk@braydonk:~/Git/fluent-bit$ date -d @1699725600
Sat Nov 11 01:00:00 PM EST 2023
braydonk@braydonk:~/Git/fluent-bit$ date -d @1697515200
Tue Oct 17 12:00:00 AM EDT 2023

Note that the timestamps printed from date -d can be compared with the log lines from a.txt above. They are both correctly interpreted as being the time in the timestamp at my system timezone, which is Eastern Time. Daylight savings is correctly determined for Oct 17, which occurs before the clocks go back on Nov 5.
If your system is in a different timezone, running TZ=EST5EDT ./fluent-bit ... etc should produce the same result as what I got.

Windows

PS C:\Users\braydonk\Documents> C:\Users\braydonk\Git\fluent-bit\build\bin\Debug\fluent-bit.exe --config config.conf --parser parser.conf
Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/11/12 19:55:28] [ info] Configuration:
[2023/11/12 19:55:28] [ info]  flush time     | 1.000000 seconds
[2023/11/12 19:55:28] [ info]  grace          | 5 seconds
[2023/11/12 19:55:28] [ info]  daemon         | 0
[2023/11/12 19:55:28] [ info] ___________
[2023/11/12 19:55:28] [ info]  inputs:
[2023/11/12 19:55:28] [ info]      tail
[2023/11/12 19:55:28] [ info] ___________
[2023/11/12 19:55:28] [ info]  filters:
[2023/11/12 19:55:28] [ info]      parser.0
[2023/11/12 19:55:28] [ info] ___________
[2023/11/12 19:55:28] [ info]  outputs:
[2023/11/12 19:55:28] [ info]      stdout.0
[2023/11/12 19:55:28] [ info] ___________
[2023/11/12 19:55:28] [ info]  collectors:
[2023/11/12 19:55:28] [ info] [fluent bit] version=2.2.0, commit=640762fb33, pid=4328
[2023/11/12 19:55:28] [debug] [engine] coroutine stack size: 98302 bytes (96.0K)
[2023/11/12 19:55:28] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/11/12 19:55:28] [ info] [cmetrics] version=0.6.3
[2023/11/12 19:55:28] [ info] [ctraces ] version=0.3.1
[2023/11/12 19:55:28] [ info] [input:tail:tail.0] initializing
[2023/11/12 19:55:28] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2023/11/12 19:55:28] [debug] [tail:tail.0] created event channels: read=820 write=824
[2023/11/12 19:55:28] [debug] [input:tail:tail.0] flb_tail_fs_stat_init() initializing stat tail input
[2023/11/12 19:55:28] [debug] [input:tail:tail.0] inode=53198770598527222 with offset=0 appended as C:\Users\braydonk\Documents\a.txt
[2023/11/12 19:55:28] [debug] [input:tail:tail.0] 1 new files found on path 'a.txt'
[2023/11/12 19:55:28] [debug] [stdout:stdout.0] created event channels: read=848 write=852
[2023/11/12 19:55:28] [ info] [sp] stream processor started
[2023/11/12 19:55:28] [debug] [input chunk] update output instances with new chunk size diff=71, records=2, input=tail.0
[2023/11/12 19:55:28] [ info] [output:stdout:stdout.0] worker #0 started
[2023/11/12 19:55:28] [debug] [input:tail:tail.0] [static files] processed 105b
[0] test: [[1699725600.000000000, [2023/11/12 19:55:28] [ info] [input:tail:tail.0] inode=53198770598527222 file=C:\Users\braydonk\Documents\a.txt ended, stop
{}], [2023/11/12 19:55:28] [debug] [input:tail:tail.0] inode=53198770598527222 file=C:\Users\braydonk\Documents\a.txt promote to TAIL_EVENT
{[2023/11/12 19:55:28] [debug] [input:tail:tail.0] [static files] processed 0b, done
"[2023/11/12 19:55:28] [debug] [task] created task=000002A3795D4AA0 id=0 OK
message[2023/11/12 19:55:28] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
"[2023/11/12 19:55:28] [ warn] [engine] service will shutdown in max 5 seconds
=>"[2023/11/12 19:55:28] [ info] [input] pausing tail.0
test"}]
[1] test: [[1697515200.000000000, {}], {"message"=>"test2"}]
[2023/11/12 19:55:28] [debug] [out flush] cb_destroy coro_id=0
[2023/11/12 19:55:28] [debug] [task] destroy task=000002A3795D4AA0 (task_id=0)
[2023/11/12 19:55:29] [ info] [engine] service has stopped (0 pending tasks)
[2023/11/12 19:55:29] [ info] [input] pausing tail.0
[2023/11/12 19:55:29] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/11/12 19:55:29] [debug] [input:tail:tail.0] inode=53198770598527222 removing file name C:\Users\braydonk\Documents\a.txt
[2023/11/12 19:55:29] [ info] [output:stdout:stdout.0] thread worker #0 stopped

With my Windows machine set to the same timezone as my Linux machine, I got the same timestamp results.
(I couldn't tell you why the output from stdout is mangled like that, but it's the same on master 🤷 )

Attached Valgrind output that shows no leaks or memory corruption was found

braydonk@braydonk:~/Git/fluent-bit$ valgrind --leak-check=full ./buildvalgrind/bin/fluent-bit --conf ~/Documents/fluent-bit-configs/config.conf --parser ~/Documents/fluent-bit-configs/parser.conf 
==399441== Memcheck, a memory error detector
==399441== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==399441== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==399441== Command: ./buildvalgrind/bin/fluent-bit --conf /home/braydonk/Documents/fluent-bit-configs/config.conf --parser /home/braydonk/Documents/fluent-bit-configs/parser.conf
==399441== 
Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/11/12 20:04:53] [ info] Configuration:
[2023/11/12 20:04:53] [ info]  flush time     | 1.000000 seconds
[2023/11/12 20:04:53] [ info]  grace          | 5 seconds
[2023/11/12 20:04:53] [ info]  daemon         | 0
[2023/11/12 20:04:53] [ info] ___________
[2023/11/12 20:04:53] [ info]  inputs:
[2023/11/12 20:04:53] [ info]      tail
[2023/11/12 20:04:53] [ info] ___________
[2023/11/12 20:04:53] [ info]  filters:
[2023/11/12 20:04:53] [ info]      parser.0
[2023/11/12 20:04:53] [ info] ___________
[2023/11/12 20:04:53] [ info]  outputs:
[2023/11/12 20:04:53] [ info]      stdout.0
[2023/11/12 20:04:53] [ info] ___________
[2023/11/12 20:04:53] [ info]  collectors:
[2023/11/12 20:04:53] [ info] [fluent bit] version=2.2.0, commit=6af61749af, pid=399441
[2023/11/12 20:04:53] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/11/12 20:04:53] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/11/12 20:04:53] [ info] [cmetrics] version=0.6.3
[2023/11/12 20:04:53] [ info] [ctraces ] version=0.3.1
[2023/11/12 20:04:53] [ info] [input:tail:tail.0] initializing
[2023/11/12 20:04:53] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2023/11/12 20:04:53] [debug] [tail:tail.0] created event channels: read=21 write=22
[2023/11/12 20:04:53] [ info] [output:stdout:stdout.0] worker #0 started
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] flb_tail_fs_inotify_init() initializing inotify tail input
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] inotify watch fd=27
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] scanning path a.txt
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] inode=5906182 with offset=0 appended as a.txt
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] scan_glob add(): a.txt, inode 5906182
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] 1 new files found on path 'a.txt'
[2023/11/12 20:04:53] [debug] [stdout:stdout.0] created event channels: read=29 write=30
[0] test: [[1699725600.000000000, {}], {"message"=>"test"}]
[1] test: [[1697515200.000000000, {}], {"message"=>"test2"}]
[2023/11/12 20:04:53] [ info] [sp] stream processor started
[2023/11/12 20:04:53] [debug] [out flush] cb_destroy coro_id=0
[2023/11/12 20:04:53] [debug] [input chunk] update output instances with new chunk size diff=71, records=2, input=tail.0
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] [static files] processed 103b
[2023/11/12 20:04:53] [ info] [input:tail:tail.0] inode=5906182 file=a.txt ended, stop
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] inode=5906182 file=a.txt promote to TAIL_EVENT
[2023/11/12 20:04:53] [ info] [input:tail:tail.0] inotify_fs_add(): inode=5906182 watch_fd=1 name=a.txt
[2023/11/12 20:04:53] [debug] [input:tail:tail.0] [static files] processed 0b, done
[2023/11/12 20:04:53] [debug] [task] created task=0x5fd5e30 id=0 OK
[2023/11/12 20:04:53] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2023/11/12 20:04:53] [ warn] [engine] service will shutdown in max 5 seconds
[2023/11/12 20:04:53] [ info] [input] pausing tail.0
[2023/11/12 20:04:53] [debug] [task] destroy task=0x5fd5e30 (task_id=0)
[2023/11/12 20:04:54] [ info] [engine] service has stopped (0 pending tasks)
[2023/11/12 20:04:54] [ info] [input] pausing tail.0
[2023/11/12 20:04:54] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/11/12 20:04:54] [debug] [input:tail:tail.0] inode=5906182 removing file name a.txt
[2023/11/12 20:04:54] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2023/11/12 20:04:54] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=5906182 watch_fd=1
==399441== 
==399441== HEAP SUMMARY:
==399441==     in use at exit: 0 bytes in 0 blocks
==399441==   total heap usage: 3,313 allocs, 3,313 frees, 983,448 bytes allocated
==399441== 
==399441== All heap blocks were freed -- no leaks are possible
==399441== 
==399441== For lists of detected and suppressed errors, rerun with: -s
==399441== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Documentation

Documentation required for this feature

fluent/fluent-bit-docs#1388

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

braydonk · 2023-11-13T19:33:04Z

The MacOS test failure looks like a flake to me.

Signed-off-by: braydonk <braydonk@google.com>

braydonk · 2024-04-29T18:32:20Z

The Windows unit test failure looks like a flake; I didn't change anything in that test with this PR.

tguenneguez · 2024-05-29T19:44:05Z

A good PR, can't wait for it to be validated and included in a release...

lecaros · 2024-06-04T14:06:37Z

Hi @braydonk,
can you please add the docs PR?

braydonk · 2024-06-04T14:46:09Z

@lecaros done: fluent/fluent-bit-docs#1388

s5unty · 2024-06-08T10:50:33Z

nginx error_log is already chomping at the bit

edsiper · 2024-06-13T03:39:28Z

@braydonk thanks for this contribution! we are ready to start the merge.

Would you please split the commits per interface ? we will need as a minimum commits for:

filter_kubernetes
parser
multiline
tests: internal: fuzzers

thank you

edsiper · 2024-06-24T19:40:44Z

I've pushed a new PR with clean commits on #9006

edsiper · 2024-06-25T00:52:30Z

merged in #9006

braydonk requested review from edsiper, leonardo-albertovich, fujimotos and koleini as code owners November 13, 2023 01:06

github-actions bot added the docs-required label Nov 13, 2023

braydonk force-pushed the fix_timezone branch from 6af6174 to 72ed839 Compare November 13, 2023 01:06

braydonk temporarily deployed to pr November 13, 2023 01:07 — with GitHub Actions Inactive

braydonk temporarily deployed to pr November 13, 2023 01:35 — with GitHub Actions Inactive

braydonk temporarily deployed to pr November 13, 2023 15:18 — with GitHub Actions Inactive

braydonk force-pushed the fix_timezone branch from 0fab094 to 7fa0fc8 Compare November 13, 2023 15:19

braydonk temporarily deployed to pr November 13, 2023 15:20 — with GitHub Actions Inactive

braydonk temporarily deployed to pr November 13, 2023 15:29 — with GitHub Actions Inactive

braydonk temporarily deployed to pr November 13, 2023 15:58 — with GitHub Actions Inactive

braydonk temporarily deployed to pr November 13, 2023 16:33 — with GitHub Actions Inactive

braydonk force-pushed the fix_timezone branch from 92ee69f to 2e1fcb2 Compare November 13, 2023 16:35

braydonk temporarily deployed to pr November 13, 2023 16:35 — with GitHub Actions Inactive

braydonk temporarily deployed to pr November 13, 2023 17:04 — with GitHub Actions Inactive

braydonk force-pushed the fix_timezone branch from 1beee34 to aafe617 Compare April 29, 2024 17:19

braydonk temporarily deployed to pr April 29, 2024 17:24 — with GitHub Actions Inactive

braydonk temporarily deployed to pr April 29, 2024 17:44 — with GitHub Actions Inactive

parser: add Time_System_Timezone

b4592c1

Signed-off-by: braydonk <braydonk@google.com>

braydonk force-pushed the fix_timezone branch from aafe617 to b4592c1 Compare April 29, 2024 17:49

braydonk temporarily deployed to pr April 29, 2024 17:49 — with GitHub Actions Inactive

braydonk temporarily deployed to pr April 29, 2024 18:10 — with GitHub Actions Inactive

braydonk had a problem deploying to pr April 29, 2024 18:10 — with GitHub Actions Failure

braydonk temporarily deployed to pr April 29, 2024 18:10 — with GitHub Actions Inactive

braydonk mentioned this pull request May 1, 2024

Use system time offset for local datetimes #593

Closed

edsiper self-assigned this May 30, 2024

edsiper added this to the Fluent Bit v3.1.0 milestone May 30, 2024

braydonk mentioned this pull request Jun 4, 2024

Add Time_System_Timezone parser option docs fluent/fluent-bit-docs#1388

Merged

edsiper mentioned this pull request Jun 24, 2024

parser: add new option time_system_timezone (copy of #8164) #9006

Merged

edsiper closed this Jun 25, 2024

franciscovalentecastro mentioned this pull request Oct 31, 2024

[Feature Request] Support time_system_timezone feature in Ops Agent GoogleCloudPlatform/ops-agent#1821

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

parser: add Time_System_Timezone #8164

parser: add Time_System_Timezone #8164

braydonk commented Nov 13, 2023 •

edited

Loading

braydonk commented Nov 13, 2023

braydonk commented Apr 29, 2024

tguenneguez commented May 29, 2024

lecaros commented Jun 4, 2024

braydonk commented Jun 4, 2024

s5unty commented Jun 8, 2024

edsiper commented Jun 13, 2024

edsiper commented Jun 24, 2024

edsiper commented Jun 25, 2024

parser: add Time_System_Timezone #8164

parser: add Time_System_Timezone #8164

Conversation

braydonk commented Nov 13, 2023 • edited Loading

a.txt

parser.conf

config.conf

Linux

Windows

braydonk commented Nov 13, 2023

braydonk commented Apr 29, 2024

tguenneguez commented May 29, 2024

lecaros commented Jun 4, 2024

braydonk commented Jun 4, 2024

s5unty commented Jun 8, 2024

edsiper commented Jun 13, 2024

edsiper commented Jun 24, 2024

edsiper commented Jun 25, 2024

braydonk commented Nov 13, 2023 •

edited

Loading