Crash When Loading timestamptz Values from Postgres Adapter

[ssanderson]

Describe the bug

Harlequin crashes when loading timestamptz values from postgres with ArrowInvalid. This only happens for timestamptz, not the timezone-less timestamp.

To Reproduce

Create a table containing a timestamptz column and insert a row:

In psql:

ssanderson=# create table repro (id serial primary key, x timestamptz);
CREATE TABLE
ssanderson=# insert into repro (id, x) values (1, now());
INSERT 0 1
ssanderson=# select * from repro;
 id |               x
----+-------------------------------
  1 | 2024-01-03 17:09:51.708286-05
(1 row)

Then run harlequin and attempt to select a row from the column.

Expected behavior

I expected harlequin to display the contents of the table.

Actual behavior

Harlequin crashes with ArrowInvalid: Cannot locate timezone '-05:00': -05:00 not found in timezone database.

Click to see traceback

(base) ctx=prod[~]$ harlequin -a postgres
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ /home/ssanderson/.local/pipx/venvs/harlequin/lib/python3.10/site-packages/harlequin/app.py:330 in load_tables                                                                                                                                                                │
│                                                                                                                                                                                                                                                                              │
│   327 │   @on(ResultsFetched)                                                                  ╭──────────────────────────────────────────────────────────────── locals ─────────────────────────────────────────────────────────────────╮                                   │
│   328 │   async def load_tables(self, message: ResultsFetched) -> None:                        │    cols = [('id', '#'), ('x', 'ts')]                                                                                                    │                                   │
│   329 │   │   for id_, (cols, data) in message.data.items():                                   │    data = [(1, datetime.datetime(2024, 1, 3, 17, 9, 51, 708286, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=68400))))] │                                   │
│ ❱ 330 │   │   │   await self.results_viewer.push_table(                                        │     id_ = 't8750968719400'                                                                                                              │                                   │
│   331 │   │   │   │   table_id=id_,                                                            │ message = ResultsFetched()                                                                                                              │                                   │
│   332 │   │   │   │   column_labels=cols,                                                      │    self = Harlequin(title='Harlequin', classes={'-dark-mode'})                                                                          │                                   │
│   333 │   │   │   │   data=data,  # type: ignore                                               ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                                   │
│                                                                                                                                                                                                                                                                              │
│ /home/ssanderson/.local/pipx/venvs/harlequin/lib/python3.10/site-packages/harlequin/components/results_viewer.py:80 in push_table                                                                                                                                            │
│                                                                                                                                                                                                                                                                              │
│    77 │   │   │   self._format_column_label(col_name, col_type)                                ╭───────────────────────────────────────────────────────────────────── locals ─────────────────────────────────────────────────────────────────────╮                          │
│    78 │   │   │   for col_name, col_type in column_labels                                      │    column_labels = [('id', '#'), ('x', 'ts')]                                                                                                    │                          │
│    79 │   │   ]                                                                                │             data = [(1, datetime.datetime(2024, 1, 3, 17, 9, 51, 708286, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=68400))))] │                          │
│ ❱  80 │   │   table = ResultsTable(                                                            │ formatted_labels = ['id [#777777]#[/]', 'x [#777777]ts[/]']                                                                                      │                          │
│    81 │   │   │   id=table_id,                                                                 │             self = ResultsViewer()                                                                                                               │                          │
│    82 │   │   │   column_labels=formatted_labels,  # type: ignore                              │         table_id = 't8750968719400'                                                                                                              │                          │
│    83 │   │   │   data=data,                                                                   ╰──────────A clear and concise description of what happened; please include screenshots or animated gifs of Harlequin if possible.
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                          │
│                                                                                                                                                                                                                                                                              │
│ /home/ssanderson/.local/pipx/venvs/harlequin/lib/python3.10/site-packages/textual_fastdatatable/data_table.py:675 in __init__                                                                                                                                                │
│                                                                                                                                                                                                                                                                              │
│    672 │   │   """Should we prioritize the cursor component class CSS background or the         ╭────────────────────────────────────────────────────────────────────────── locals ──────────────────────────────────────────────────────────────────────────╮               │
│    673 │   │   renderable background in the event where a cell contains a renderable with a     │                    backend = None                                                                                                                          │               │
│    674 │   │   background color."""                                                             │                    classes = None                                                                                                                          │               │
│ ❱  675 │   │   self.cursor_type = cursor_type                                                   │              column_labels = ['id [#777777]#[/]', 'x [#777777]ts[/]']                                                                                      │               │
│    676 │   │   """The type of cursor of the `DataTable`."""                                     │              column_widths = None                                                                                                                          │               │
│    677 │   │   self.null_rep = null_rep                                                         │ cursor_background_priority = 'renderable'                                                                                                                  │               │
│    678 │   │   """The string used to represent missing data (None or null)"""                   │ cursor_foreground_priority = 'css'                                                                                                                         │               │
│                                                                                                 │                cursor_type = 'range'                                                                                                                       │               │
│                                                                                                 │                       data = [(1, datetime.datetime(2024, 1, 3, 17, 9, 51, 708286, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=68400))))] │               │
│                                                                                                 │                   disabled = False                                                                                                                         │               │
│                                                                                                 │              fixed_columns = 0                                                                                                                             │               │
│                                                                                                 │                 fixed_rows = 0                                                                                                                             │               │
│                                                                                                 │              header_height = 1                                                                                                                             │               │
│                                                                                                 │                         id = 't8750968719400'                                                                                                              │               │
│                                                                                                 │   max_column_content_width = 100                                                                                                                           │               │
│                                                                                                 │                   max_rows = 100000                                                                                                                        │               │
│                                                                                                 │                       name = None                                                                                                                          │               │
│                                                                                                 │                   null_rep = ''                                                                                                                            │               │
│                                                                                                 │                       self = ResultsTable(id='t8750968719400')                                                                                             │               │
│                                                                                                 │                show_cursor = True                                                                                                                          │               │
│                                                                                                 │                show_header = True                                                                                                                          │               │
│                                                                                                 │            show_row_labels = True                                                                                                                          │               │
│                                                                                                 │              zebra_stripes = False                                                                                                                         │               │
│                                                                                                 ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯               │
│                                                                                                                                        - Shell:
- Terminal:
- OS of the shell:
- OS of the terminal (if different from the shell):

For example, for my system, these are:
- Bash
- Windows Terminal
- Ubuntu 22.04 / WSL2
- Windows 11                                                                                                                                      │
│ /home/ssanderson/.local/pipx/venvs/harlequin/lib/python3.10/site-packages/textual_fastdatatable/data_table.py:1113 in watch_cursor_type                                                                                                                                      │
│                                                                                                                                                                                                                                                                              │
│   1110 │   │   return Coordinate(row, column)                                                   ╭───────────────── locals ─────────────────╮                                                                                                                                 │
│   1111 │                                                                                        │  new = 'range'                           │                                                                                                                                 │
│   1112 │   def watch_cursor_type(self, old: str, new: str) -> None:                             │  old = 'cell'                            │                                                                                                                                 │
│ ❱ 1113 │   │   self._set_hover_cursor(False)                                                    │ self = ResultsTable(id='t8750968719400') │                                                                                                                                 │
│   1114 │   │   if self.show_cursor:                                                             ╰──────────────────────────────────────────╯                                                                                                                                 │
│   1115 │   │   │   self._highlight_cursor()                                                                                                                                                                                                                                  │
│   1116                                                                                                                                                                                                                                                                       │
│                                                                                                                                                                                                                                                                              │
│ /home/ssanderson/.local/pipx/venvs/harlequin/lib/python3.10/site-packages/textual_fastdatatable/data_table.py:2456 in _set_hover_cursor                                                                                                                                      │
│                                                                                                                                                                                                                                                                              │
│   2453 │   │   elif cursor_type == "row":                                                       ╭──────────────────── locals ─────────────────────╮                                                                                                                          │
│   2454 │   │   │   self.refresh_row(self.hover_row)                                             │      active = False                             │                                                                                                                          │
│   2455 │   │   elif cursor_type in ("cell", "range"):                                           │ cursor_type = 'range'                           │                                                                                                                          │
│ ❱ 2456 │   │   │   self.refresh_coordinate(self.hover_coordinate)                               │        self = ResultsTable(id='t8750968719400') │                                                                                                                          │
│   2457 │                                                                                        ╰─────────────────────────────────────────────────╯                                                                                                                          │
│   2458 │   @on(events.MouseDown)                                                                                                                                                                                                                                             │
│   2459 │   async def move_cursor_to_mouse_down(self, event: events.MouseDown) -> None:                                                                                                                                                                                       │
│                                                                                                                                                                                                                                                                              │
│ /home/ssanderson/.local/pipx/venvs/harlequin/lib/python3.10/site-packages/textual_fastdatatable/data_table.py:1642 in refresh_coordinate                                                                                                                                     │
│                                                                                                                                                                                                                                                                              │
│   1639 │   │   """                                                                              ╭──────────────────── locals ────────────────────╮                                                                                                                           │
│   1640 │   │   if not self.is_valid_coordinate(coordinate):                                     │ coordinate = Coordinate(row=0, column=0)       │                                                                                                                           │
│   1641 │   │   │   return self                                                                  │       self = ResultsTable(id='t8750968719400') │                                                                                                                           │
│ ❱ 1642 │   │   region = self._get_cell_region(coordinate)                                       ╰────────────────────────────────────────────────╯                                                                                                                           │
│   1643 │   │   self._refresh_region(region)                                                                                                                                                                                                                                  │
│   1644 │   │   return self                                                                                                                                                                                                                                                   │
│   1645                                                                                                                                                                                                                                                                       │
│                                                                                                                                                                                                                                                                              │
│ /home/ssanderson/.local/pipx/venvs/harlequin/lib/python3.10/site-packages/textual_fastdatatable/data_table.py:1300 in _get_cell_region                                                                                                                                       │
│                                                                                                                                                                                                                                                                              │
│   1297 │   │   # The x-coordinate of a cell is the sum of widths of the data cells to the left  ╭───────────────────── locals ─────────────────────╮                                                                                                                         │
│   1298 │   │   # plus the width of the render width of the longest row label.                   │ column_index = 0                                 │                                                                                                                         │
│   1299 │   │   x = (                                                                            │   coordinate = Coordinate(row=0, column=0)       │                                                                                                                         │
│ ❱ 1300 │   │   │   sum(column.render_width for column in self.ordered_columns[:column_index])   │    row_index = 0                                 │                                                                                                                         │
│   1301 │   │   │   # TODO: support row labels                                                   │         self = ResultsTable(id='t8750968719400') │                                                                                                                         │
│   1302 │   │   │   # + self._row_label_column_width                                             ╰──────────────────────────────────────────────────╯                                                                                                                         │
│   1303 │   │   )                                                                                                                                                                                                                                                             │
│                                                                                                                                                                                                                                                                              │
│ /home/ssanderson/.local/pipx/venvs/harlequin/lib/python3.10/site-packages/textual_fastdatatable/data_table.py:1761 in ordered_columns                                                                                                                                        │
│                                                                                                                                                                                                                                                                              │
│   1758 │   │   │   widths = [0] * len(labels)                                                   ╭───────────────────── locals ──────────────────────╮                                                                                                                        │
│   1759 │   │                                                                                    │ labels = ['id [#777777]#[/]', 'x [#777777]ts[/]'] │                                                                                                                        │
│   1760 │   │   if self.backend is not None:                                                     │   self = ResultsTable(id='t8750968719400')        │                                                                                                                        │
│ ❱ 1761 │   │   │   column_content_widths = self.backend.column_content_widths                   │ widths = [0, 0]                                   │                                                                                                                        │
│   1762 │   │   else:                                                                            ╰───────────────────────────────────────────────────╯                                                                                                                        │
│   1763 │   │   │   column_content_widths = [0] * len(labels)                                                                                                                                                                                                                 │
│   1764                                                                                                                                                                                                                                                                       │
│                                                                                                                                                                                                                                                                              │
│ /home/ssanderson/.local/pipx/venvs/harlequin/lib/python3.10/site-packages/textual_fastdatatable/backend.py:235 in column_content_widths                                                                                                                                      │
│                                                                                                                                                                                                                                                                              │
│   232 │   │   if not self._column_content_widths:                                              ╭─────────────────────────────────── locals ───────────────────────────────────╮                                                                                              │
│   233 │   │   │   if self._string_data is None:                                                │ self = <textual_fastdatatable.backend.ArrowBackend object at 0x7f57e51db310> │                                                                                              │
│   234 │   │   │   │   self._string_data = pa.Table.from_arrays(                                ╰──────────────────────────────────────────────────────────────────────────────╯                                                                                              │
│ ❱ 235 │   │   │   │   │   arrays=[                                                                                                                                                                                                                                           │
│   236 │   │   │   │   │   │   self._safe_cast_arr_to_str(arr) for arr in self.data.columns                                                                                                                                                                                   │
│   237 │   │   │   │   │   ],                                                                                                                                                                                                                                                 │
│   238 │   │   │   │   │   names=self.data.column_names,                                                                                                                                                                                                                      │
│                                                                                                                                                                                                                                                                              │
│ /home/ssanderson/.local/pipx/venvs/harlequin/lib/python3.10/site-packages/textual_fastdatatable/backend.py:236 in <listcomp>                                                                                                                                                 │
│                                                                                                                                                                                                                                                                              │
│   233 │   │   │   if self._string_data is None:                                                ╭─────────────────────────────────── locals ───────────────────────────────────╮                                                                                              │
│   234 │   │   │   │   self._string_data = pa.Table.from_arrays(                                │   .0 = <list_iterator object at 0x7f57e5152b60>                              │                                                                                              │
│   235 │   │   │   │   │   arrays=[                                                             │  arr = <pyarrow.lib.ChunkedArray object at 0x7f57e611ca90>                   │                                                                                              │
│ ❱ 236 │   │   │   │   │   │   self._safe_cast_arr_to_str(arr) for arr in self.data.columns     │        [                                                                     │                                                                                              │
│   237 │   │   │   │   │   ],                                                                   │          [                                                                   │                                                                                              │
│   238 │   │   │   │   │   names=self.data.column_names,                                        │        │   2024-01-03 22:09:51.708286                                        │                                                                                              │
│   239 │   │   │   │   )                                                                        │          ]                                                                   │                                                                                              │
│                                                                                                │        ]                                                                     │                                                                                              │
│                                                                                                │ self = <textual_fastdatatable.backend.ArrowBackend object at 0x7f57e51db310> │                                                                                              │
│                                                                                                ╰──────────────────────────────────────────────────────────────────────────────╯                                                                                              │
│                                                                                                                                                                                                                                                                              │
│ /home/ssanderson/.local/pipx/venvs/harlequin/lib/python3.10/site-packages/textual_fastdatatable/backend.py:346 in _safe_cast_arr_to_str                                                                                                                                      │
│                                                                                                                                                                                                                                                                              │
│   343 │   │   and other nested types), we fall back to Python.                                 ╭───────────────────────── locals ──────────────────────────╮                                                                                                                 │
│   344 │   │   """                                                                              │ arr = <pyarrow.lib.ChunkedArray object at 0x7f57e611ca90> │                                                                                                                 │
│   345 │   │   try:                                                                             │       [                                                   │                                                                                                                 │
│ ❱ 346 │   │   │   arr = arr.cast(                                                              │         [                                                 │                                                                                                                 │
│   347 │   │   │   │   pa.string(),                                                             │       │   2024-01-03 22:09:51.708286                      │                                                                                                                 │
│   348 │   │   │   │   safe=False,                                                              │         ]                                                 │                                                                                                                 │
│   349 │   │   │   )                                                                            │       ]                                                   │                                                                                                                 │
│                                                                                                ╰───────────────────────────────────────────────────────────╯                                                                                                                 │
│                                                                                                                                                                                                                                                                              │
│ in pyarrow.lib.ChunkedArray.cast:565                                                                                                                                                                                                                                         │
│                                                                                                                                                                                                                                                                              │
│ /home/ssanderson/.local/pipx/venvs/harlequin/lib/python3.10/site-packages/pyarrow/compute.py:404 in cast                                                                                                                                                                     │
│                                                                                                                                                                                                                                                                              │
│   401 │   │   │   options = CastOptions.unsafe(target_type)                                                                                                                                                                                                                  │
│   402 │   │   else:                                                                                                                                                                                                                                                          │
│   403 │   │   │   options = CastOptions.safe(target_type)                                                                                                                                                                                                                    │
│ ❱ 404 │   return call_function("cast", [arr], options, memory_pool)                                                                                                                                                                                                          │
│   405                                                                                                                                                                                                                                                                        │
│   406                                                                                                                                                                                                                                                                        │
│   407 def index(data, value, start=None, end=None, *, memory_pool=None):                                                                                                                                                                                                     │
│                                                                                                                                                                                                                                                                              │
│ ╭─────────────────────────────────────────────────────────────────────────────────────────────────── locals ───────────────────────────────────────────────────────────────────────────────────────────────────╮                                                             │
│ │              arr = <pyarrow.lib.ChunkedArray object at 0x7f57e611ca90>                                                                                                                                       │                                                             │
│ │                    [                                                                                                                                                                                         │                                                             │
│ │                      [                                                                                                                                                                                       │                                                             │
│ │                    │   2024-01-03 22:09:51.708286                                                                                                                                                            │                                                             │
│ │                      ]                                                                                                                                                                                       │                                                             │
│ │                    ]                                                                                                                                                                                         │                                                             │
│ │      memory_pool = None                                                                                                                                                                                      │                                                             │
│ │          options = CastOptions(to_type=string, allow_int_overflow=true, allow_time_truncate=true, allow_time_overflow=true, allow_decimal_truncate=true, allow_float_truncate=true, allow_invalid_utf8=true) │                                                             │
│ │             safe = False                                                                                                                                                                                     │                                                             │
│ │ safe_vars_passed = True                                                                                                                                                                                      │                                                             │
│ │      target_type = DataType(string)                                                                                                                                                                          │                                                             │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                                                             │
│                                                                                                                                                                                                                                                                              │
│ in pyarrow._compute.call_function:590                                                                                                                                                                                                                                        │
│                                                                                                                                                                                                                                                                              │
│ in pyarrow._compute.Function.call:385                                                                                                                                                                                                                                        │
│                                                                                                                                                                                                                                                                              │
│ in pyarrow.lib.pyarrow_internal_check_status:154                                                                                                                                                                                                                             │
│                                                                                                                                                                                                                                                                              │
│ in pyarrow.lib.check_status:91                                                                                                                                                                                                                                               │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ArrowInvalid: Cannot locate timezone '-05:00': -05:00 not found in timezone database

Additional context
What is the output of harlequin --version?

$ harlequin --version
harlequin, version 1.8.0

Installed Adapters:
  - postgres, version 0.2.1
  - duckdb, version 1.8.0
  - sqlite, version 1.8.0

What database adapter are you using with Harlequin?

PostgreSQL

Can you tell us more about your system?

I'm running in bash on Ubuntu 22.04. Postgres version is 14.10.

[tconbeer]

Thanks for the detailed issue and the repro. Will address in the next release.

[tconbeer]

This has been patched in v1.9.0