Native Support for Complex Types

[expanded-for-real]

Support more types, as described here - imprint-serde/imprint#5

Proposal

DECIMAL(0xB) - BigDecimal
UUID(0xC) - 16 byte UUID String
DATE(0xD) - LocalDate
TIME(0xE) - LocalTime
DURATION(0xF) - Duration (seconds + micro/nanoseconds maybe?)

Java doesn't really have an Interval type (although we could make a custom type to support it?) so I'm thinking we can leave that out for now, ar at least just include it, stub it out, but no-op it

[expanded-for-real]

@agavra let me know if this looks OK

[agavra]

@expanded-for-real this looks good to me, do you mind issuing a PR to the main rust project just reserving those in the README spec?

Also regarding Java Types those make sense to me, we should confirm they line up with Avro as I think that's likely to be the most inter-operated format with Imprint.

[expanded-for-real]

@agavra regarding the rust project - yes, will do!

I've been looking into the Avro type match up here and I think I can have a simple PR ready here shortly. Pretty much everything is 1-1 except it looks like Avro has both micro and millisecond time precision options. We could try to directly support both in Imprint as direct conversion from Avro (two TypeCodes) or create one TypeCode for like nanoseconds or something specific to Imprint and just convert.

We can match up Decimal, UUID, and Date already so we're good there

[expanded-for-real]

@agavra I can't branch the main rust project, but I can submit something in a fork? Let me know

Payload Encoding

type_code	Type	Encoding details
0x0	null	No payload; length = 0
0x1	bool	1 byte 0x00 / 0x01
0x2	int32	4-byte signed int32
0x3	int64	8-byte signed int64
0x4	float32	IEEE‑754 little‑endian bytes
0x5	float64	IEEE‑754 little‑endian bytes
0x6	bytes	length + payload
0x7	string	UTF‑8, length + payload
0x8	array	size + type_code + payload
0x9	map	size + key_type_code + value_type_code + payload
0xA	row	Nested Imprint row (recursive joins)
0xB	date	4-byte int32 days since Unix epoch
0xC	time	8-byte int64 microseconds since midnight
0xD	uuid	16-byte binary representation
0xE	decimal	varint scale + varint length + unscaled bytes
15–127	reserved	Future primitives / logical types

[agavra]

re: submitting to the main rust project -- yeah just create a fork and then when you push to your fork it will ask if you want to open a PR against it. After it's approved I can merge the PR into the main fork/branch.

Also for the types this makes sense to me, though we probably also want to have a timestamp type for completion (let's just do millis for now for both timestamp and for time, I think micro is pretty niche and we can always add it later if necessary)

[expanded-for-real]

@agavra will do! One final question re: timestamps - should we consider time zones on this at all? That will ultimately affect what kind of native type I map it to (Instance, Zoned, Offset, etc). Could also just save as UTC and let the server determine the timezone if we want? I'm open to anything here

[agavra]

Could also just save as UTC and let the server determine the timezone if we want?

I've seen it done this way 90% of the time where the timestamp is always considered "UTC" and then the client can choose to preprocess or postprocess timestamps to map it to different time zones. Eventually if we want to add a zoned timestamp type if demand comes up we can do that.

Postgres is typically a good system to take inspiration from and they always store values as UTC: https://www.postgresql.org/docs/current/datatype-datetime.html#DATATYPE-DATETIME-INPUT-TIME-STAMPS

For timestamp with time zone values, an input string that includes an explicit time zone will be converted to UTC (Universal Coordinated Time) using the appropriate offset for that time zone. If no time zone is stated in the input string, then it is assumed to be in the time zone indicated by the system's TimeZone parameter, and is converted to UTC using the offset for the timezone zone. In either case, the value is stored internally as UTC, and the originally stated or assumed time zone is not retained.

[expanded-for-real]

Agreed - will have this ready shortly