Skip to content

Commit

Permalink
feat!: Store timestamps as timestamp_tz (#17843)
Browse files Browse the repository at this point in the history
#### Summary

Currently we are storing all timestamps as `timestamp_ntz` this is "wall clock" time meaning that it has no timezone associated with it. the `arrow.TimestampType` is:

```
TimestampType is encoded as a 64-bit signed integer since the UNIX epoch (2017-01-01T00:00:00Z). The zero-value is a second and time zone neutral. Time zone neutral can be considered UTC without having "UTC" as a time zone
```

Which makes it a better fit for users to be able to utilize the data in their queries.
  • Loading branch information
bbernays committed May 7, 2024
1 parent 2453f93 commit 6eb8711
Show file tree
Hide file tree
Showing 3 changed files with 3 additions and 3 deletions.
2 changes: 1 addition & 1 deletion plugins/destination/snowflake/client/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ func New(_ context.Context, logger zerolog.Logger, spec []byte, _ plugin.NewClie
if err != nil {
return nil, err
}
db, err := sql.Open("snowflake", dsn+"&BINARY_INPUT_FORMAT=BASE64&BINARY_OUTPUT_FORMAT=BASE64")
db, err := sql.Open("snowflake", dsn+"&BINARY_INPUT_FORMAT=BASE64&BINARY_OUTPUT_FORMAT=BASE64&timezone=UTC")
if err != nil {
return nil, err
}
Expand Down
2 changes: 1 addition & 1 deletion plugins/destination/snowflake/client/deletestale.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ func (c *Client) DeleteStale(ctx context.Context, msgs message.WriteDeleteStales
sb.WriteString(`"` + strings.ToUpper(schema.CqSourceNameColumn.Name) + `"`)
sb.WriteString(" = ? and \"")
sb.WriteString(strings.ToUpper(schema.CqSyncTimeColumn.Name))
sb.WriteString("\"::timestamp_ntz < ?::timestamp_ntz")
sb.WriteString("\" < CONVERT_TIMEZONE('UTC','UTC', TO_TIMESTAMP(?))")
sql := sb.String()
if _, err := c.db.ExecContext(ctx, sql, msg.SourceName, msg.SyncTime); err != nil {
return err
Expand Down
2 changes: 1 addition & 1 deletion plugins/destination/snowflake/client/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ func (*Client) SchemaTypeToSnowflake(t arrow.DataType) string {
case *arrow.BinaryType, *arrow.LargeBinaryType:
return "binary"
case *arrow.TimestampType:
return "timestamp_ntz"
return "timestamp_tz"
case *types.JSONType, *arrow.StructType:
return "variant"
default:
Expand Down

0 comments on commit 6eb8711

Please sign in to comment.