Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MCOL-265 Add support for TIMESTAMP data type #739

Merged
merged 2 commits into from Jun 10, 2019

Conversation

tntnatbry
Copy link
Contributor

This patch adds support for TIMESTAMP data type in columnstore.

Design decisions:

  1. Timestamp is stored on disk as 64bit int. 44 bits for the seconds (since unix epoch) and 20 bits for microsecond are used. Since the server enforces a maximum value of 0x7FFFFFFF (the 2038 year problem) for the seconds, we therefore enforce the same restriction in columnstore. This is however easily extensible in case the server supports higher range for seconds in future, since we are allocating 44 bits for seconds.
  2. Timezone in use by the server (available through time_zone system variable) is passed from the UM to the PMs to perform the conversion from broken-down representation (yyyy-mm-dd HH:MM:SS) to on disk representaion and vice versa. Currently, support is added for "SYSTEM" and offset in the form +/-HH:MM (https://mariadb.com/kb/en/library/time-zones/)
  3. For cpimport, I have added a new option, -T, in command line args for timezone. Default value is SYSTEM. User can specify the timezone to use while performing a cpimport through this option. This is also required for bulk insert (insert into select *) for timestamp.

Automatic default timestamp properties:

  • As documented here: https://mariadb.com/kb/en/library/timestamp/, support for automatic properties is also added to columnstore. I am using the default value in the system catalog (defaultvalue field in calpontsys.syscolumn) to store "current_timestamp() ON UPDATE current_timestamp()" for the first timestamp column in the table.

Current limitations:

  1. Default current_timestamp() and on update current_timestamp() are currently not supported directly in the DDL statement.
  2. Setting constraint to NULL for timestamp does not give default NULL since columnstore currently does not track whether the user supplied NULL constraint or not (it assumes NULL by default). However, for timestamp column, NOT NULL is the default in server. "show create table" therefore gives incorrect output as the timestamp column is actually set with the default automatic properties. I.e. "create table t1 (a timestamp null)engine=columnstore" is actually setting "a" with default timestamp properties"

I wrote a simple test suite to demonstrate some of this new functionality. See the gist (timestamp_test.c): https://gist.github.com/tntnatbry/bd51700ce0869db82516b035e519b7f6.

Notes on this test suite:

  1. This requires the Mariadb C connector: https://mariadb.com/kb/en/library/mariadb-connector-c/ . Once the connector is installed, instructions to compile and execute the test suite are at the top of the file. You can run the test with either "innodb" or "columnstore" as storage engines for the tables.
  2. This will drop "timestamp_test" database if it already exists.
  3. You might need to change the connection settings in main() based on your installation.

Pending:
Support in utils/udfsdk :)

I look forward to your feedback on this patch.

I am submitting this code under the BSD-new license.

@drrtuy
Copy link
Collaborator

drrtuy commented Apr 23, 2019

Greetings, that is a highly valued contribution indeed. We appreciate this so much and will review it as fast as possible. Could you post the incantation on donating the patch under BSD?

@drrtuy
Copy link
Collaborator

drrtuy commented Apr 23, 2019

It seems I was too fast and the incantation is in place.

Copy link
Contributor

@LinuxJedi LinuxJedi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fantastic work!

@LinuxJedi LinuxJedi merged commit d6cdef3 into mariadb-corporation:develop Jun 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants