parser: Support 'System' timezone in Time_Offset #7994

SijmenHuizenga · 2023-09-30T18:21:41Z

This introduces the System choice for the Time_Offset parser option. This new option uses the system timezone to automatically compute the UTC offset. This resolves #593.

To test this change I had to fix a few bugs in the unit tests related to time-offset-parsing. These bug fixes are included in this pr. See individual commits for the individual bug fixes.

This implementation is inspired by #4849

Testing
Before we can approve your change; please submit the following in a comment:

[N/A] Example configuration file for the change
[N/A] Debug log output from testing the change
Attached Valgrind output that shows no leaks or memory corruption was found

~/projects/fluent-bit/build (system-timezone✓) $ valgrind ./bin/flb-it-parser
==870255== Memcheck, a memory error detector
==870255== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==870255== Using Valgrind-3.21.0 and LibVEX; rerun with -h for copyright info
==870255== Command: ./bin/flb-it-parser
==870255==
Test tzone_offset...                            [ OK ]
==870255== Warning: invalid file descriptor -1 in syscall close()
Test time_lookup...                             ==870255== Warning: invalid file descriptor -1 in syscall close()
[ OK ]
==870255== Warning: invalid file descriptor -1 in syscall close()
Test json_time_lookup...                        ==870255== Warning: invalid file descriptor -1 in syscall close()
[ OK ]
==870255== Warning: invalid file descriptor -1 in syscall close()
Test regex_time_lookup...                       ==870255== Warning: invalid file descriptor -1 in syscall close()
[ OK ]
==870255== Warning: invalid file descriptor -1 in syscall close()
Test mysql_unquoted...                          ==870255== Warning: invalid file descriptor -1 in syscall close()
[ OK ]
==870255== Warning: invalid file descriptor -1 in syscall close()
SUCCESS: All unit tests have passed.
==870255==
==870255== HEAP SUMMARY:
==870255==     in use at exit: 0 bytes in 0 blocks
==870255==   total heap usage: 14,269 allocs, 14,269 frees, 2,493,340 bytes allocated
==870255==
==870255== All heap blocks were freed -- no leaks are possible
==870255==
==870255== For lists of detected and suppressed errors, rerun with: -s
==870255== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Run local packaging test showing all targets (including any new ones) build.
Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

fluent/fluent-bit-docs#1215

Backporting
Not required.

Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Resolves a bug that caused 8 of test cases being skipped in test_parser_tzone_offset Signed-off-by: Sijmen Huizenga <sijmenhuizenga@gmail.com>

Signed-off-by: Sijmen Huizenga <sijmenhuizenga@gmail.com>

Add the 'System' option to Time_Offset that uses the system timezone to compute the UTC offset during startup. As proposed in fluent#593 Signed-off-by: Sijmen Huizenga <sijmenhuizenga@gmail.com>

braydonk

Thank you for doing this, I hope a maintainer accepts this change.

braydonk · 2023-10-17T21:02:14Z

src/flb_parser.c

@@ -1008,6 +1025,11 @@ int flb_parser_tzone_offset(const char *str, int len, int *tmdiff)
        return 0;
    }

+    /* Check system timezones */
+    if (*p == 'S') {


Probably worth doing a strncmp here instead of just checking the first character.

braydonk · 2023-10-17T21:02:54Z

src/flb_parser.c

@@ -1040,7 +1062,7 @@ int flb_parser_tzone_offset(const char *str, int len, int *tmdiff)
        min = ((p[2] - '0') * 10) + (p[3] - '0');
    }

-    if (hour < 0 || hour > 59 || min < 0 || min > 59) {
+    if (hour < 0 || hour > 23 || min < 0 || min > 59) {


braydonk · 2023-10-17T21:05:30Z

tests/internal/parser.c

@@ -123,6 +128,35 @@ int flb_parser_regex_do(struct flb_parser *parser,
                        void **out_buf, size_t *out_size,
                        struct flb_time *out_time);

+static char* mock_timezone(char *tz)


I think it would be nicer if mock_timezone returned an int with success, and took a pointer to where the original timezone should be returned. Then instead of doing perror and exit here, the test can see a -1 from this function and go through the existing acutest failure pattern.

braydonk · 2023-10-17T21:06:23Z

tests/internal/parser.c

+{
+    char *original_tz = getenv("TZ");
+    if (original_tz) {
+        original_tz = strdup(original_tz);


what does strduping the string into itself do?

braydonk · 2023-10-17T21:09:07Z

tests/internal/parser.c

+        unsetenv("TZ");
+    }
+    tzset();
+    free(original_tz);


It's up to you, but imo it's easier to keep track of frees if they happen in the same scope as the allocation. I think it would make more sense if undo_mock_timezone wasn't in charge of freeing original_tz, and instead the test did this after using it to undo the mock timezone.

quentinmit · 2023-10-17T21:26:15Z

src/flb_parser.c

+    *tmdiff = -_timezone;
+    return 0;
+#else
+    time_t currentTime = time(NULL);


This only works for timezones that don't have DST; you need to calculate the offset for the timestamp of each record, not just capture the current offset at process start time. Different log entries will have different offsets because of DST.

github-actions · 2024-01-16T01:50:44Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

github-actions · 2024-04-17T01:49:37Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

teicher · 2024-04-19T14:40:51Z

it's a pity this MR is not progressing

braydonk · 2024-04-19T16:24:27Z

I've opened an alternative PR: #8164
That one isn't moving either though. I'll ping maintainers to see if we can get some movement on it.

SijmenHuizenga added 2 commits September 30, 2023 19:07

parsre: fix tzone_offset test

1cef8e2

Resolves a bug that caused 8 of test cases being skipped in test_parser_tzone_offset Signed-off-by: Sijmen Huizenga <sijmenhuizenga@gmail.com>

parser: Fix hour validation in timezone offset parsing

b7ee1e8

Signed-off-by: Sijmen Huizenga <sijmenhuizenga@gmail.com>

SijmenHuizenga requested review from edsiper, leonardo-albertovich, fujimotos and koleini as code owners September 30, 2023 18:21

github-actions bot added the docs-required label Sep 30, 2023

SijmenHuizenga mentioned this pull request Sep 30, 2023

Parser: Add new 'System' timezone option in Time_Offset fluent/fluent-bit-docs#1215

Open

parser: Support system timezone in Time_Offset

33e624f

Add the 'System' option to Time_Offset that uses the system timezone to compute the UTC offset during startup. As proposed in fluent#593 Signed-off-by: Sijmen Huizenga <sijmenhuizenga@gmail.com>

SijmenHuizenga force-pushed the system-timezone branch from 8a0860a to 33e624f Compare September 30, 2023 18:51

braydonk reviewed Oct 17, 2023

View reviewed changes

braydonk mentioned this pull request Oct 17, 2023

Use system time offset for local datetimes #593

Open

quentinmit reviewed Oct 17, 2023

View reviewed changes

github-actions bot added the Stale label Jan 16, 2024

braydonk removed the Stale label Jan 16, 2024

github-actions bot added the Stale label Apr 17, 2024

github-actions bot removed the Stale label Apr 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

parser: Support 'System' timezone in Time_Offset #7994

parser: Support 'System' timezone in Time_Offset #7994

SijmenHuizenga commented Sep 30, 2023 •

edited

braydonk left a comment

braydonk Oct 17, 2023

braydonk Oct 17, 2023

braydonk Oct 17, 2023

braydonk Oct 17, 2023

braydonk Oct 17, 2023

quentinmit Oct 17, 2023

github-actions bot commented Jan 16, 2024

github-actions bot commented Apr 17, 2024

teicher commented Apr 19, 2024

braydonk commented Apr 19, 2024

parser: Support 'System' timezone in Time_Offset #7994

Are you sure you want to change the base?

parser: Support 'System' timezone in Time_Offset #7994

Conversation

SijmenHuizenga commented Sep 30, 2023 • edited

braydonk left a comment

Choose a reason for hiding this comment

braydonk Oct 17, 2023

Choose a reason for hiding this comment

braydonk Oct 17, 2023

Choose a reason for hiding this comment

braydonk Oct 17, 2023

Choose a reason for hiding this comment

braydonk Oct 17, 2023

Choose a reason for hiding this comment

braydonk Oct 17, 2023

Choose a reason for hiding this comment

quentinmit Oct 17, 2023

Choose a reason for hiding this comment

github-actions bot commented Jan 16, 2024

github-actions bot commented Apr 17, 2024

teicher commented Apr 19, 2024

braydonk commented Apr 19, 2024

SijmenHuizenga commented Sep 30, 2023 •

edited