Date related macros based on UTC time

[Wernfried]

Is there any way to have date related macros (i.e. S_HOUR, S_MIN, S_SEC, S_DAY etc.) based on UTC times?

In my syslog-ng I have systems which provide local times and others provide UTC times. Thus I have different date-parsers:

options { 
   time-zone("Europe/Zurich");
};

parser p_date_time_local {
   date-parser(
      format("%Y-%m-%d %H:%M:%S"),
      template("${.FN.date} ${.FN.time}"),
      time-zone("Europe/Zurich")
   );
};

parser p_date_time_utc {
   date-parser(
      format("%Y-%m-%d %H:%M:%S"),
      template("${.HW.date} ${.HW.time}"),
      time-zone("UTC")
   );
};

In order to avoid clashes of filenames while transition of daylight-saving times I like to get hour as UTC time. I managed to develop these rewrites, but certainly these are just workarounds:

rewrite r_set_hour_local {
   set("${S_HOUR}", value("HOUR_UTC"));
   subst("^0", "", value("HOUR_UTC"));
   set("$(- ${HOUR_UTC} 1)", value("HOUR_UTC"), condition("${S_TZOFFSET}" eq "+01:00"));
   set("$(- ${HOUR_UTC} 2)", value("HOUR_UTC"), condition("${S_TZOFFSET}" eq "+02:00"));
   set("23", value("HOUR_UTC"), condition("${HOUR_UTC}" eq "-1"));
   set("22", value("HOUR_UTC"), condition("${HOUR_UTC}" eq "-2"));
};

rewrite r_set_hour_utc {
   fix-time-zone("Europe/Zurich");
   set-time-zone("UTC");
   set("${S_HOUR}", value("HOUR_UTC"));
   subst("^0", "", value("HOUR_UTC"));
};

Note, in general I like to get local time when I use S_HOUR macro. But for some parts, UTC based hour is required.
Would you have any other proposal?

Kind Regards
Wernfried

[bazsi]

Most cases where you can expand template strings you can specify the timezone as well. So you would be using the same S_HOUR macro but you specify the timezone as UTC since for a file destination there are two distinct purposes for which you can use a timezone (the name of the file and the content), you can specify both. One is called send-time-zone() the other local-time-zone(). I am on a phone and can't double check this for exact syntax, but the above should be roughly right. The documentation should have the exact spelling. Syslog-ng uses this strategy with timezones, some templates are expanded in the local timezone (name of the file, name of an SQL table or mongodb collection), others are expanded as you are sending the data. By explicitly setting either, syslog-ng would automatically convert the timestamp from its original timezone to the one specified. If you don't specify anything it will stick to the original timezone it received messages in. And it can do other nifty things like guessing the timezone or fixing it up if the sender was incorrect. Bazsi

…

On Tue, Nov 1, 2022, 19:16 Wernfried ***@***.***> wrote: Is there any way to have date related macros (i.e. S_HOUR, S_MIN, S_SEC, S_DAY etc.) based on UTC times? In my syslog-ng I have systems which provide local times and others provide UTC times. Thus I have different date-parsers: options { time-zone("Europe/Zurich"); }; parser p_date_time_local { date-parser( format("%Y-%m-%d %H:%M:%S"), template("${.FN.date} ${.FN.time}"), time-zone("Europe/Zurich") ); }; parser p_date_time_utc { date-parser( format("%Y-%m-%d %H:%M:%S"), template("${.HW.date} ${.HW.time}"), time-zone("UTC") ); }; In order to avoid clashes of filenames while transition of daylight-saving times I like to get hour as UTC time. I managed to develop these rewrites, but certainly these are just workarounds: rewrite r_set_hour_local { set("${S_HOUR}", value("HOUR_UTC")); subst("^0", "", value("HOUR_UTC")); set("$(- ${HOUR_UTC} 1)", value("HOUR_UTC"), condition("${S_TZOFFSET}" eq "+01:00")); set("$(- ${HOUR_UTC} 2)", value("HOUR_UTC"), condition("${S_TZOFFSET}" eq "+02:00")); set("23", value("HOUR_UTC"), condition("${HOUR_UTC}" eq "-1")); set("22", value("HOUR_UTC"), condition("${HOUR_UTC}" eq "-2")); }; rewrite r_set_hour_utc { fix-time-zone("Europe/Zurich"); set-time-zone("UTC"); set("${S_HOUR}", value("HOUR_UTC")); subst("^0", "", value("HOUR_UTC")); }; Note, in general I like to get local time when I use S_HOUR macro. But for some parts, UTC based hour is required. Would you have any other proposal? Kind Regards Wernfried — Reply to this email directly, view it on GitHub <#4187>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFOK5Q3TLV2Q2MU6THWPXDWGFNAPANCNFSM6AAAAAARUJ37ZA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Wernfried]

I would need local times and UTC times within one message. I do some more stuff with the hour value:

 rewrite r_set_hour {
   set("${S_HOUR}", value("HOUR_UTC"));
   subst("^0", "", value("HOUR_UTC"));
   set("$(- ${HOUR_UTC} 1)", value("HOUR_UTC"), condition("${S_TZOFFSET}" eq "+01:00"));
   set("$(- ${HOUR_UTC} 2)", value("HOUR_UTC"), condition("${S_TZOFFSET}" eq "+02:00"));
   set("23", value("HOUR_UTC"), condition("${HOUR_UTC}" eq "-1"));
   set("22", value("HOUR_UTC"), condition("${HOUR_UTC}" eq "-2"));
   set("$(% ${HOUR_UTC} 2)", value("HOUR_UTC"));
};

Thus, setting time zone on source or destination level (or even global) is not sufficient.

[bazsi]

What about this:

#4202

This would let you do stuff like this $(format-time --time-zone UTC %H $S_UNIXTIME)

If you omit $S_UNIXTIME it takes the timestamp of the message by default (which is slightly faster, as we don't have to parse a string value). The function takes any strftime format string to format the timestamp.

[Wernfried]

What about this:

#4202

This would let you do stuff like this $(format-time --time-zone UTC %H $S_UNIXTIME)

If you omit $S_UNIXTIME it takes the timestamp of the message by default (which is slightly faster, as we don't have to parse a string value). The function takes any strftime format string to format the timestamp.

Looks very promising. Would provide full flexibility for every use case, you could even manipulate the time value, if needed.
Unix timestamps are always UTC, so I would consider --time-zone UTC as default.

[bazsi]

Note this PR improves this area further: #4319

[bazsi]

I think the previous PR should already solve this use-case, so closing this issue. Let me know if something is missing.