add support for TIMESTAMPTZ

[djouallah]

What happened?

DUNIT = DUNIT.cast({"SETTLEMENTDATE": "TIMESTAMPTZ"})
generate an error

What version of ibis are you using?

10.0.0.dev71

What backend(s) are you using, if any?

DuckDB and pyspark

Relevant log output

SignatureValidationError: Schema({'SETTLEMENTDATE': 'TIMESTAMPTZ'}) has failed due to the following errors:
  `fields`: {'SETTLEMENTDATE': 'TIMESTAMPTZ'} is not matching GenericMappingOf(key=InstanceOf(type=<class 'str'>), value=CoercedTo(type=<class 'ibis.expr.datatypes.core.DataType'>, func=<bound method DataType.__coerce__ of <class 'ibis.expr.datatypes.core.DataType'>>), type=CoercedTo(type=<class 'ibis.common.collections.FrozenOrderedDict'>, func=<class 'ibis.common.collections.FrozenOrderedDict'>))

Expected signature: Schema(fields: FrozenOrderedDict[str, DataType])

Code of Conduct

• I agree to follow this project's Code of Conduct

[ncclementi]

Hi @djouallah thanks for opening this issue.

Can you give us more context on what are you trying to achieve? If possible a minimal reproducible example?

What's the original type of the SETLEMENTDATE column, and what are you trying to cast it too?

You can create an example using a memtable, this way we can reproduce it and help you out.

import ibis
t = ibis.memtable([{"a": 1}, {"a": 2}])

[djouallah]

alright, I am trying ibis for the first time, my approach, load some data, do the transformation and write back to delta table, which works fine except that there is a subtle difference how arrow write timestamp and pyspark, i endup with two data type one with time zone the other not

see example here : AnalysisException: [DELTA_FAILED_TO_MERGE_FIELDS] Failed to merge fields 'SETTLEMENTDATE' and 'SETTLEMENTDATE'

https://colab.research.google.com/drive/18JGPcj3VaJ2tcxDZx1XB7UufTnEPjRfu#scrollTo=51jIBhgTXzrS

[ncclementi]

@djouallah I see there are two different issues here.

The one initially reported, about "TIMESTAMPTZ". To proved a timezone you need to use the Ibis timestamp dtype.

import ibis.expr.datatypes as dt
#then you can do something like
DUNIT = DUNIT.cast({"SETTLEMENTDATE": dt.Timestamp(timezone="UTC")})

The other issue, you are reporting on the second comment might be a bug. What seems to be happening is that at the moment the disk type that we are writing with is backend specific, and there seems to be backend inconsistencies when writing to disk. But the code provided, is not a minimal reproducible example as as it requires downloading some data from the internet. Would you be able to provide a
minimal reproducer (ideally with ibis.memtable()) so we can track this down.

[djouallah]

@ncclementi thanks a lot, it was just me not reading the documentation, problem solved all good thanks for your help