Proposal: Switch date and time handling to use CPAN DateTime modules

[gdevenyi]

It looks like based on #406 and #395 and maybe others that the time/date handling for snapshot naming has become a bit of a problem.

Perl offers a particularly useful module DateTime, for both handling dates and times, generating standard formatting, and doing math with these objects in a sensible and standards-compliant way. It also supports sorting and comparisons between DateTime objects in perl compatible ways for tests.

Short example, get the current time in ISO UTC format as a string, then re-parse that string back into a DateTime object and extract a bunch of sub-date details

use DateTime;
use DateTime::Format::ISO8601;

$mydatetime = DateTime->now()->iso8601();

print("$mydatetime \n");

$dt = DateTime::Format::ISO8601->parse_datetime( $mydatetime );

use DateTime;
use DateTime::Format::ISO8601;

$mydatetime = DateTime->now()->iso8601();

print("$mydatetime \n");

$dt = DateTime::Format::ISO8601->parse_datetime( $mydatetime );

$year   = $dt->year;
$month  = $dt->month;          # 1-12
 
$day    = $dt->day;            # 1-31
 
$dow    = $dt->day_of_week;    # 1-7 (Monday is 1)
 
$hour   = $dt->hour;           # 0-23
$minute = $dt->minute;         # 0-59
 
$second = $dt->second;         # 0-61 (leap seconds!)
 
$doy    = $dt->day_of_year;    # 1-366 (leap years)
 
$doq    = $dt->day_of_quarter; # 1..
 
$qtr    = $dt->quarter;        # 1-4

print("$year $month $day $dow $hour $minute $second $doy $doq $qtr\n");

What's the exact difference in time between now and ~ a year from now.

use DateTime;
use DateTime::Duration;

$today = DateTime->now();
$later = DateTime->new(
    year      => 2020,
    month     => 06,
    day       => 26,
    hour      => 1,
    minute    => 30,
    second    => 0,
    time_zone => 'UTC',
);

$dur = $later->subtract_datetime_absolute($today);
print($dur->in_units( 'seconds'));

[gdevenyi]

https://metacpan.org/pod/DateTime
https://github.com/houseabsolute/DateTime.pm/wiki

Most importantly, the most common questions:
https://github.com/houseabsolute/DateTime.pm/wiki/Basic-Usage

[phreaker0]

@gdevenyi do you intend to port the entire code? The hard part will be detection the DST crossover and handle the duplicate snapshot accordingly.

[gdevenyi]

@phreaker0 I don't yet use sanoid, only syncoid, so I'm not sure I can effectively work on dev of such a change, however I am interested in contributing that in the future.

Regarding the DST crossovers and such, I believe DateTime already intrinsically handles this if you use UTC (default), and use the built-in date/time comparisons available in DateTime (see https://metacpan.org/pod/DateTime#Ambiguous-Local-Times ).

This is one of the reasons I'm suggesting going this direction for the code.

[phreaker0]

I specially meant the handling of the duplicate hour in the DST crossover case, DateTime handles it but it needs to be detected nevertheless because a duplicate snapshot name is not possible :-) and the later should be suffixed so no hourly snapshot is lost.

[gdevenyi]

I'm not sure what you mean here, if we use DateTime in UTC there is never a DST crossover because UTC doesn't have DST events. This would simplify checking further.

There's still the issue of leap seconds but thats a pretty tiny corner case...

[phreaker0]

sanoid currently supports UTC (which is the default for packages and recommend install instruction) and local timezones. In the later case DST crossover is possible and needs to be handled.

Local timezones are handy if for example you have an network file server and you are exposing snapshots directly so users can restore older versions themselves. Having the snapshot name/time in the local timezone makes it a lot easier because most non technical users have a hard time with utc :-)

[gdevenyi]

Ah, okay, makes sense. This is also a pretty simple solution, we need to properly store the daylight savings status when storing localtime.

Take, for example, "eastern time" https://en.wikipedia.org/wiki/Eastern_Time_Zone

Standard Time (EST) when observing standard time (autumn/winter) are 5 hours behind Coordinated Universal Time (UTC−05:00).

Eastern Daylight Time (EDT), when observing daylight saving time (spring/summer), is 4 hours behind Coordinated Universal Time (UTC−04:00).

iso8601 mandiates a storage format for local time:
https://en.wikipedia.org/wiki/ISO_8601#Time_zone_designators

The offset from UTC is appended to the time in the same way that 'Z' was above, in the form ±[hh]:[mm], ±[hh][mm], or ±[hh].

So, as of right now, where I am, the localtime is:

use DateTime;
use DateTime::Format::ISO8601::Format;
 
my $format = DateTime::Format::ISO8601::Format->new();

my $dt = DateTime->now->set_time_zone( 'local' );
$mydatetime = $format->format_datetime($dt);

print("$mydatetime \n");

> perl localtime.pl
2019-06-18T16:55:15-04:00

Then internally, we do all our math in UTC, after using the DateTime parsers and conversion to UTC.

This will handle when daylight savings flips over, and guarantees that we never have identically named snapshots because we embed our timezone info.

[gdevenyi]

I'll update PR #410 to add a localtime option to demonstrate this further.

[gdevenyi]

I've updated PR #410 to demonstrate UTC vs localtime generation, here's the difference in how the datestamps are generated:

use DateTime;
use DateTime::Format::ISO8601::Format;
 
my $format = DateTime::Format::ISO8601::Format->new();

my $dt = DateTime->now->set_time_zone( 'local' );
$mydatetime = $format->format_datetime($dt);

$dt->set_time_zone('UTC');
$myutctime = $format->format_datetime($dt);

print("$mydatetime \n");
print("$myutctime \n");

> perl printtime.pl
2019-06-19T12:43:47-04:00 
2019-06-19T16:43:47Z

^As we can see here, when my timezone goes from EDT to EDT, the name of the snapshot will go from -04:00 to -05:00, preventing the double-naming. Both these formats are back-parsable from ISO8601 format using DateTime::Format::ISO8601->parse_datetime($string) so we can use DateTime to compare the difference between any two snapshots using reliable maths.

[phreaker0]

@gdevenyi I didn't think about changing the snapshot name format, this would make it really easy. Let's see if @jimsalterjrs is open for such a change.

[timkgh]

It's that time of the year when I'm reminded about this issue.

[putnam]

Rather than create a new ticket, I just wanted to point out the datetime strings used in sanoid snaps vs. syncoid ephemeral snaps are oddly very different:

xyz/xyz@syncoid_abc_2022-01-26:23:07:34-GMT-08:00
xyz/xyz@autosnap_2022-01-27_07:30:04_frequently   

- vs. _, timezone indicator vs. not, and why not just use the ISO standard with Z?