Datetime support in Lua #6244

tsafin · 2021-07-23T22:05:42Z

tsafin
Jul 23, 2021

Introduction

Here I try to present the current version of datetime type support,
which is being developed at the moment: the datetime lua module in Tarantool,
and its' MessagePack serialization schema.

Our current datetime design is heavily influenced by Sci-Lua lua-time
module implementation https://github.com/stepelu/lua-time
and you could find, for example, very similar approach sued for
handling of operations with years and months intervals
(which should be handled differently than usual seconds,
days periods).

But, internally, it actually uses for all ISO-8601 dates manipulations
Christian Hanson' c-dt module https://github.com/chansen/c-dt
(though this code has been modified slightly, to make their integration
into our cmake build process easier)

tsafin · 2021-07-23T22:06:06Z

tsafin
Jul 23, 2021
Author

Datetime Module API

NB! 2021-08-18: There were late time changes introduced to the API, some features
have been excluded.

This module allows to parse various date/timestamps formats,
to operate with them, and to generate their textual representation.
Parsing of date constants will handle all known ISO-8601 formats,
but also will deal with few non-standard extensions.

For internal date representation it uses cdata structures in a form:

struct datetime_t {
    int secs;
    int nsecs;
    int offset;
};

Where:

secs is the (signed) number of seconds since epoch
1970-01-01T00:00Z;
nsecs is number of nanoseconds since beginning of day;
offset is the timezone offset (in minutes);

`datetime()` – create datetime object

datetime() call is alias to the datetime constructor method
datetime.new(), which could create date/time object using
either string literals, or initialization object.

Given simple string literal, it behaves like simple wrapper over
datetime.parse() method.

If there is initialization object passed, then it could create
date/time object using these attributes below:


YMD part
year	Year in range [1..9999]
month	Month in range [1..12]
day	Day in month in range [1..31]
HMS part
hour	Hour in range [0..23]
minute	Minute in range [0..59]
second	Seconds in range [0..60]. It allows to have fraction part in which case it goes to nsec field
tz	Timezone offset (in minutes) for HMS part

Example:

datetime = require `datetime`

d = datetime(‘1970-01-01’)
d = datetime{year = 1970, month = 1, day = 1}

Load level constructor `datetime.new_raw()`

Similarly to initialization object used in the datetime.new()
above you could create datetime object using its' "low-level"
attributes:

secs;
nsec;
offset;

Low-level approach
secs	Seconds since epoch
nsec	Fraction part in nanoseconds
offset	Time-zone offset from UTC in minutes

Example:

datetime = require `datetime`

d = datetime.new_raw{secs = 0, nsec = 0, offset = 0}
d = datetime.new_raw(0, 0, 0)

`interval()` – create time interval object

Create empty interval which may be used in arithmetic operations.
Please see below "Date and interval arithmetic operations"

`parse()` – parse full ISO-8601 string

Parse full length date/time literal, which may be in ISO-8601 format of
any of extended formats supported by parse_date(), parse_time() or
parse_timezone()

It deals with date/time string in format

date ([Tt ] time ([ ] time_zone)? )?

Where time or time_zone parts may be omitted.

Example:

datetime = require `datetime`

d = datetime.parse(`1970-01-01`)
d = datetime.parse(`1970-01-01T00:00:00Z`)
d = datetime.parse(`1970-01-01T02:00:00+02:00`)

`parse_date()` – parse ISO-8601 date literal

Parse date string literal, return partial date object which has
precision of up-to date period of time.

A number of standard ISO-8601 formats supported, plus there are some
relaxed formats which are of frequently use:

Basic	Extended
20121224	2012-12-24	Calendar date (ISO 8601)
2012359	2012-359	Ordinal date (ISO 8601)
2012W521	2012-W52-1	Week date (ISO 8601)
2012Q485	2012-Q4-85	Quarter date

`parse_time()` – parse ISO-8601 time literal

Parse time string literal, return partial date/time object, which
defines time period inside of single date.

A number of standard ISO-8601 formats supported, plus there are some
relaxed formats which are of frequently use:

Basic	Extended
T12	N/A
T1230	T12:30
T123045	T12:30:45
T123045.123456789	T12:30:45.123456789
T123045,123456789	T12:30:45,123456789

The time designator T may be omitted.

`parse_zone()` – parse ISO-8601 time zone

Parse time-zone string literal, return partial date/time object, which
defines timezone offset in minutes sing GMT.

A number of standard ISO-8601 formats supported, plus there are some
relaxed formats which are of frequently use:

Basic	Extended
Z	N/A
±hh	N/A
±hhmm	±hh:mm

`tostring()` – convert datetime object to string

Return string representation (probably compact if there are some parts
missing) of a date-time objects passed

`now()` – return current date/time

now() returns local date and time object. It will use local time-zone
and nanosecond precision.

`strftime()` – convert date object to string using format

strftime() is the FFI wrapper around strftime() function in LIBC. It
supports all the same flags which supports strftime() from host OS.

See
https://pubs.opengroup.org/onlinepubs/000095399/functions/strftime.html
for more details.

`asctime()` – convert date object to string using asctime predefined format

asctime() is the FFI wrapper over asctime_r() from a host libc. NB,
asctime_r is the reentrant version of asctime().
It returns string in the form "Sun Sep 16 01:03:52 1973\\n\\0"

https://pubs.opengroup.org/onlinepubs/009695399/functions/asctime.html

`ctime()` – convert local time to string using ctime() predefined format

ctime() is the FFI wrapper over ctime_r() (the reentrant ctime())
in the host libc. It returns string in the form "Sun Sep 16 01:03:52 1973\\n\\0"

https://pubs.opengroup.org/onlinepubs/009695399/functions/ctime.html

The difference of ctime() and asctime() is that former is returning
local time zone formatted, while the latter deals with GMT.

Examples:

tarantool> date = require 'datetime'
---
...
tarantool> T = date.now()
---
...
tarantool> T
---
- 2021-07-14T01:36:48.554105+03:00
...
tarantool> date.asctime(T)
---
- 'Tue Jul 13 22:36:48 2021
'
...
tarantool> date.ctime(T)
---
- 'Wed Jul 14 01:36:48 2021
'
...

Date attribute accessors


`timestamp`, `ts`	Calculate timestamp with seconds and nanoseconds parts combined
`nanoseconds`, `ns`	Number of nanoseconds in time part
`microseconds`, `us`	Number of microseconds in time part
`milliseconds`, `ms`	Number of milliseconds in time part
`seconds`, `s`	Alias to timestamp
~~`minutes`, `min`, `m`~~	Number of minutes in time part
~~`hours`, `hr`~~	Number of hours in time part
~~`days`, `d`~~	Number of days in time part

Note that there are several aliases to the same attribute may be used equally. i.e.
T.nanoseconds and T.ns are freely interchanged and return the same value.

tarantool> d = date{year = 1970, month = 1, day = 1, hour = 0, minute = 10, second=10}
tarantool> d.secs
---
- 610
...
tarantool> d.nsec
---
- 0
...
tarantool> d.offset
---
- 0
...
tarantool> d.nanoseconds
---
- 610000000000
...
tarantool> d.milliseconds
---
- 610000
...
tarantool> d.hours
---
- 0.16944444444444
...
tarantool> d.minutes
---
- 10.166666666667
...

Most of attributes mentioned above are readonly attributes, but there are 2
attributes, which are read-write, to which you could actually assign values:

date/interval attribute	meaning
`unixtime`	get/set unix epoch time
`timestamp`	get/set unix timestamp (seconds with fraction part)

tarantool> T = date('2000-01-01')
---
...
tarantool> T.unixtime
---
- 946684800
...
tarantool> T.timestamp = 120.5
---
...
tarantool> T
---
- 1970-01-01T00:02:00.500Z
...
tarantool> T.unixtime
---
- 120
...
tarantool> T.epoch = os.time()
---
...
tarantool> T
---
- 2021-08-12T17:10:48Z

...

Time-zone manipulations

There are modifier methods, which may be used for manipulations of timezone in
a datetime object:


`to_utc()`	Reset datetime timezone to UTC
`to_tz(offset)`	Switch datetime timezone to the given offset

Example:

tarantool> T
---
- 2021-08-12T22:18:28.842380+03:00
...
tarantool> T:to_tz(120)
---
- 2021-08-12T21:18:28.842380+02:00
...
tarantool> T:to_utc()
---
- 2021-08-12T19:18:28.842380Z
...

Time intervals

In addition to date/time objects there are time interval objects, which
could be used for date/time arithmetic (see below).

Interval type	Meaning
`years(y)`	Create object which represents interval of `y` years
`months(m)`	Create object which represents interval of `m` months
`weeks(w)`	Create object which represents interval of `w` weeks
`days(d)`	Create object which represents interval of `d` days
`hours(h)`	Create object which represents interval of `h` hours
`minutes(m)`	Create object which represents interval of `m` minutes
`seconds(s)`	Create object which represents interval of `s` seconds
`interval()`	Create empty interval object (representing time interval of 0 seconds 0 nanoseconds length)

Usually time interval uses data storage similar to date/time cdata
structure shown above, i.e.

struct datetime_interval_t {
    int secs;
    int nsec;
};

But we used 2 additional interval structures, which enable
correct handling of years and month arithmetics:

struct t_interval_months {
    int m;
};

struct t_interval_years {
    int y;
};

Date and interval arithmetic operations

Datetime module overloads addition and subtraction operations for
date/time and time interval objects. All reasonable combinations of
those types supported, i.e. you could do

Operand types	Result type
Time + Interval	Time
Interval + Time	Time
Time – Time	Interval
Time – Interval	Time
Interval + Interval	Interval
Interval – Interval	Interval

Also we have these methods in datetime and interval objects which
may simplify complex interval artithmetic.

add{object}
sub{object}

Example,

tarantool> T = date('1970-01-01')
---
...

tarantool> T:add{months = 4, years = 5, hours = 1, minutes = 1}
---
- 1975-05-01T01:01Z
...

Here is the list of attributes supported by add/sub methods:


`years=y`	Add/sub `y` years. y should be in range [0..9999]
`months=M`	Add/sub `M` months. m should be in the range [0..12]
`weeks=w`	Add/sub `w` weeks. w should be in the range [0..52]
`days=d`	Add/sub `d` days. d should be in the range [0..31]
`hours=h`	Add/sub `h` hours. h should be in the range [0..23]
`minutes=m`	Add/sub `m` minutes. m should be in the range [0..59]
`seconds=s`	Add/sub `s` seconds. s should be in the range [0..61). s may have fraction part, in this case it will be stored to nsec field of interval data structure.

~~> NB! Methods :add{} and :sub{} modify object state!~~

Methods :add{} and :sub{} do not modify object state, but rather create
another copy of an object!

Note, that operations involving months and years over datetime values
representing leap dates may produce order dependent results.

Example, '1972-02-29' is leap date, and if we would add 2 months and 1 year
in different order then we could end up producing different final results.

tarantool> T = date('1972-02-29')
---
...

tarantool> M = date.months(2)
---
...

tarantool> Y = date.years(1)
---
...

tarantool> T + M
---  
- 1972-04-29T00:00Z
...

tarantool> T + Y
---
- 1973-03-01T00:00Z
...

So far – so good… But let us check different order of adding intervals:

tarantool> T + M + Y
---  
- 1973-04-30T00:00Z
...

tarantool> T + Y + M
---
- 1973-05-01T00:00Z
...

¯\_(ツ)_/¯

If you want deterministic results, then I'd recommend to use
:add{}/:sub{} datetime methods instead, because they always use
predefined order of calculations.

tarantool> T:add{years = 1, months = 2}
---
- 1973-04-30T00:00Z
...

0 replies

tsafin · 2021-07-23T22:06:33Z

tsafin
Jul 23, 2021
Author

Datetime storage schema

Not all date/time types created equal. At the moment we do not persist
interval values, only real datetime values. i.e. those with internal
data structure names struct datetime_t.

In this case we have at most 3 integer fields to be saved:

signed integer secs (seconds since unix epoch) is required;
[unsigned] integer nsec (nanoseconds) is optional and will be
present only in timestamp with fractional parts in seconds;
Integer offset is optional and will be present only in timestamp
with timezone defined, otherwise UTC time assumed;

To make serialized storage as much as compact as possible we employ following
serialization schema for datetime extension in MessagePack:

Datetime uses standard MP_EXT extension type (which would be using 0xC7
for 1 byte length) extension, which creates container of 1 to 3 integers.

  +----+-----------+---+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
  |0xC7|len (uint8)| 4 | seconds (int) | nanoseconds (uint) | offset (int)  |
  +----+-----------+---+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+

Here we see that MessagePack extension MP_EXT byte (0xC7), followed by
extension kind MP_DATETIME (4), then 1-byte length, followed by variable
length payload, which consist of:

signed integer seconds part (required). Depending on the value of
seconds it may be from 1 to 8 bytes positive or negative integer number;
[optional] fraction time in nanoseconds as unsigned integer.
If this value is 0 then it's not saved (unless there is offset field, as
below);
[optional] timezone offset in minutes as signed integer.
If this field is 0 then it's not saved.

9 replies

tsafin Jul 27, 2021
Author

I've made changes to use int64_t for seconds, and apparently now the biggest year is circa 5881546, and the lowest year is circa -11769003 (though stringization is limited with 0000..9999 years). Limitation here is c-dt implementation - which accepts dates (since Rata Die) as in32_t, but it's good enough for this rather huge range of years.

rybakit Jul 29, 2021

@tsafin Could you please clarify how the optional fields should be composed. I'm wondering if it is possible to omit nanoseconds but specify offset, like:

+----+---+-----------+====~~~~~~~====+---~~~~~~~~-----+
|0xC7| 4 |len (uint8)| seconds (int) |  offset (uint) |
+----+---+-----------+====~~~~~~~====+---~~~~~~~~-----+

If so, how would decoder know that seconds are followed by an offset?

rybakit Jul 29, 2021

Nvm, I didn't notice this sentence:

If this value is 0 then it's not saved (unless there is offset field, as
below)

:)

tsafin Jul 30, 2021
Author

If this value is 0 then it's not saved (unless there is offset field, as
below)

Yes, if there is timezone, but there is no fraction part in seconds then we have to save extra 0 (encoded as 1-octet 0) before timezone offset.

tsafin Jul 30, 2021
Author

(now having explained it I realized that offset may be negative, but I've documented it as unsigned. (Though in the implementation code there was proper signed/unsigned function used).

tsafin · 2021-07-29T14:37:19Z

tsafin
Jul 29, 2021
Author

Current Limitations

There is no integration to the Olson timezone database for converting of symbolic names i.e. 'Europe/Moscow' to their corresponding numeric offsets UTC+03:00. Current plan is to use icu, which is already integrated to Tarantool, for these services;
Albeit having current timezone information value returned by now() is actually normalized and is in UTC. This is design decision used by current datetime module implementation. And due to UTC values kelp already we have not found any reason to introduce any local timezone specific variant, e.g. now() vs now_utc() vs now_local(). If you have any objections and need to have timezone shift operations (which?) then please scream now or never;
There is no defined set of date access helpers which may be of some usage, like return Julian day, Rata Die day, Easter day for various calendars, quarter of a year, days in month, day of the year, etc. They all look not essential for 1st version. But please say if you think otherwise.

0 replies

tsafin · 2021-08-12T17:44:57Z

tsafin
Aug 12, 2021
Author

datetime.new vs os.date vs os.time

You could (carefully) use datetime functions with the data returned by os.date(), which could return date-time object, representing
local time, or with os.time() which returns epoch time.

We do not recommend to use os.date information here, because it provides datetime object without timezone
information, though time values will be using local timezone.
It's not a problem though if you live in London...

`datetime` vs `os.time`

tarantool> T = date.new()
---
...

tarantool> T
---
- 1970-01-01T00:00Z
...

tarantool> os.time()
---
- 1628788236
...

tarantool> T.epoch = os.time()
---
...

tarantool> T
---
- 2021-08-12T17:10:48Z
...

tarantool> T:to_tz('+03:00')
---
- 2021-08-12T20:10:48+03:00
...

`datetime` vs `os.date`

tarantool> T = date.new(os.date('*t'))
---
...

tarantool> T
---
- 2021-08-12T20:12:08Z
...

tarantool> date.now()
---
- 2021-08-12T20:12:22.452146+03:00
…

tarantool> t = os.date '*t'
---
...

tarantool> t
---
- sec: 27
  min: 13
  day: 12
  isdst: false
  wday: 5
  yday: 224
  year: 2021
  month: 8
  hour: 20
...

tarantool> date.new(t)
---
- 2021-08-12T20:13:27Z
...

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tarantool

Datetime support in Lua #6244

{{title}}

Replies: 4 comments 9 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Tarantool

Datetime support in Lua #6244

tsafin Jul 23, 2021

Introduction

Replies: 4 comments · 9 replies

tsafin Jul 23, 2021 Author

Datetime Module API

datetime() – create datetime object

Load level constructor datetime.new_raw()

interval() – create time interval object

parse() – parse full ISO-8601 string

parse_date() – parse ISO-8601 date literal

parse_time() – parse ISO-8601 time literal

parse_zone() – parse ISO-8601 time zone

tostring() – convert datetime object to string

now() – return current date/time

strftime() – convert date object to string using format

asctime() – convert date object to string using asctime predefined format

ctime() – convert local time to string using ctime() predefined format

Date attribute accessors

Time-zone manipulations

Time intervals

Date and interval arithmetic operations

tsafin Jul 23, 2021 Author

Datetime storage schema

tsafin Jul 27, 2021 Author

rybakit Jul 29, 2021

rybakit Jul 29, 2021

tsafin Jul 30, 2021 Author

tsafin Jul 30, 2021 Author

tsafin Jul 29, 2021 Author

Current Limitations

tsafin Aug 12, 2021 Author

datetime.new vs os.date vs os.time

datetime vs os.time

datetime vs os.date

tsafin
Jul 23, 2021

Replies: 4 comments 9 replies

tsafin
Jul 23, 2021
Author

`datetime()` – create datetime object

Load level constructor `datetime.new_raw()`

`interval()` – create time interval object

`parse()` – parse full ISO-8601 string

`parse_date()` – parse ISO-8601 date literal

`parse_time()` – parse ISO-8601 time literal

`parse_zone()` – parse ISO-8601 time zone

`tostring()` – convert datetime object to string

`now()` – return current date/time

`strftime()` – convert date object to string using format

`asctime()` – convert date object to string using asctime predefined format

`ctime()` – convert local time to string using ctime() predefined format

tsafin
Jul 23, 2021
Author

tsafin Jul 27, 2021
Author

tsafin Jul 30, 2021
Author

tsafin Jul 30, 2021
Author

tsafin
Jul 29, 2021
Author

tsafin
Aug 12, 2021
Author

`datetime` vs `os.time`

`datetime` vs `os.date`