fix(deps): bump sqlglot to pick up duckdb array fixes #8682

cpcloud · 2024-03-17T11:43:28Z

Closes #8641. The fix was made upstream in sqlglot.

This PR ballooned into some other fixes:

We're segfaulting a lot because of BUG: Seg Fault when st_aswkb() to arrow duckdb/duckdb_spatial#236. I removed geospatial tests from the duckdb 0.10 CI job and added a 0.9.2 job that should mitigate this until the next duckdb geospatial release.
The sqlglot version bump introduced some changed SQL, so I updated the snapshots for that
The sqlglot verison bump changed mssql limit output to use fetch instead of top, which breaks the case of limit 0. fetch first 0 rows only raises a DB error, while top 0 is allowed. I elected to xfail this for now but perhaps we should raise an issue on the sqlglot side?
Because of the sqlglot bump, log function calls are now generated using log(x, y) (base first or not depends on the dialect) without explicitly constructing an Anonymous function using self.f.anon. This exposed a missing log implementation in the sqlite backend which I added.
I addressed the failing test_zip_null test that was failing for the cloud backends. BigQuery xfails with an AssertionError because it turns NULL arrays into empty arrays and the fix for snowflake was to check for any NULL input arrays and return NULL accordingly.

cpcloud · 2024-03-18T10:58:42Z

I've removed geospatial from the primary duckdb ci job and added another job that runs geospatial tests while constraining the install of duckdb to be <0.10. This should avoid segfaulting in CI, and once duckdb/duckdb_spatial ships a new release we can go back to testing just one version of duckdb.

ibis/backends/tests/snapshots/test_sql/test_isin_bug/clickhouse/out.sql

ibis/backends/__init__.py

ibis/expr/sql.py

cpcloud · 2024-03-18T18:46:26Z

Running the cloud tests locally since BigQuery CI takes over an hour to run.

cpcloud · 2024-03-18T20:31:44Z

Cloud tests are passing:

…/ibis on  bump-sqlglot is 📦 v8.0.0 via 🐍 v3.12.2 via ❄️   impure (ibis-3.12.2-env)
❯ pytest -m 'bigquery or snowflake' -n auto --dist loadgroup --snapshot-update -q
bringing up nodes...
..............................................................x....................................x...x.....xx...x........x.x......x.x...........x...x..xx..xxx.....x..........x.....x...x... [  5%]
..........x.xx..x......................................x....x..x........xx..........xx...xx.......xx.x..xx............x....x...x..x...x.............xxx..............................xx..x...x [ 11%]
...x....x.......x..x.......x..........................xx..x.....................x......x...x.xx..x..x.xx.......................................x.............................................. [ 16%]
.......x..........................................s........x.................x....x...............xx.........x...x...x.......x...xx..x...............s.........................xx.x..........x [ 22%]
................x........x...........................x..............x...xx......x.......x.....x......x.........................................x.x......x.......x................x...x.......x [ 28%]
......................xxx..................x......x......x..............xx..........x.x.....................xx..x...xxxxx.xxxxx.sx.......xx...xxxx.xxx..x.......xx......x...........x..x...x.. [ 33%]
........................x.....xx................................x......x...x...xxxx.........x..............x......x...x.............x.......x.xx......x...xx......xx..........x...x.....x..... [ 39%]
.............x..............x..................x...........x.....x....ssssssssssssssssssssssx...x..........xx...x...x......xxx...x..x.............x...........xx.............................x [ 45%]
........x.x.x.......x...x......x.........x...xxx.x.x..x.x.......x.....x....x.........x.x........x....x..........................xx....x....x.....x.x..................x...x..............x.... [ 50%]
....x..x..x...............................................x.xx.....x..x.x..x...........x.........x.....x...x..x..........x.x.x...x.....x..x.....x........x.x.........xx.x..x..x..x.x..xx....xx [ 56%]
................................xx....x...................................................................s...........s......x..x..s.....s...x...x.x...........xx..x.x...x...xx............... [ 62%]
.x..............x....x...x...................x..s.....................x...x......xx.x..x...........xx............xx....x........xxxx.....x........x...........x....x......x.......x..........x [ 67%]
.xxx.x.x.x..xx.x........x...........x.......................................................x.x..x..............................................................................xxxxxxxxx.xx.. [ 73%]
....xxx.xxxxx...x.x...x.xxx..x.xxxx.xxxxxx.xxxx...xxx.xx..x..x.x...x.x.x.x.......x.x...xxx..xx.xxx.x.xxxx.xxxxxxxxxx.xxx...x.x...xxx........x...x...........x......x......x..x...x............ [ 79%]
..xx.................x........x........x...x....x...x......x...xx...x.x.........x...................x......x.........x............x........................x.................................. [ 84%]
...................................x.......................................................................................................................................................... [ 90%]
........................................x........................................................................................................................................x............ [ 96%]
....................................................................................................sException ignored in: <socket.socket fd=16, family=2, type=1, proto=6, laddr=('192.168.50.72', 51904), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=16, family=2, type=1, proto=6, laddr=('192.168.50.72', 51904), raddr=('34.102.189.10', 443)>
..Exception ignored in: <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 51936), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 51936), raddr=('34.102.189.10', 443)>
....Exception ignored in: <socket.socket fd=16, family=2, type=1, proto=6, laddr=('192.168.50.72', 51980), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=16, family=2, type=1, proto=6, laddr=('192.168.50.72', 51980), raddr=('34.102.189.10', 443)>
....Exception ignored in: <socket.socket fd=17, family=2, type=1, proto=6, laddr=('192.168.50.72', 52026), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=17, family=2, type=1, proto=6, laddr=('192.168.50.72', 52026), raddr=('34.102.189.10', 443)>
....Exception ignored in: <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 35072), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 35072), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=17, family=2, type=1, proto=6, laddr=('192.168.50.72', 35088), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=17, family=2, type=1, proto=6, laddr=('192.168.50.72', 35088), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 35106), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 35106), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=17, family=2, type=1, proto=6, laddr=('192.168.50.72', 35122), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=17, family=2, type=1, proto=6, laddr=('192.168.50.72', 35122), raddr=('34.102.189.10', 443)>
.Exception ignored in: <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 35126), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 35126), raddr=('34.102.189.10', 443)>
.Exception ignored in: <socket.socket fd=19, family=2, type=1, proto=6, laddr=('192.168.50.72', 35144), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=19, family=2, type=1, proto=6, laddr=('192.168.50.72', 35144), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 35142), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 35142), raddr=('34.102.189.10', 443)>
.....Exception ignored in: <socket.socket fd=32, family=2, type=1, proto=6, laddr=('192.168.50.72', 35198), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=32, family=2, type=1, proto=6, laddr=('192.168.50.72', 35198), raddr=('34.102.189.10', 443)>
.Exception ignored in: <socket.socket fd=32, family=2, type=1, proto=6, laddr=('192.168.50.72', 35204), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=32, family=2, type=1, proto=6, laddr=('192.168.50.72', 35204), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 35226), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 35226), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 35234), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 35234), raddr=('34.102.189.10', 443)>
.Exception ignored in: <socket.socket fd=27, family=2, type=1, proto=6, laddr=('192.168.50.72', 35242), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=27, family=2, type=1, proto=6, laddr=('192.168.50.72', 35242), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=27, family=2, type=1, proto=6, laddr=('192.168.50.72', 35246), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=27, family=2, type=1, proto=6, laddr=('192.168.50.72', 35246), raddr=('34.102.189.10', 443)>
.Exception ignored in: <socket.socket fd=27, family=2, type=1, proto=6, laddr=('192.168.50.72', 35256), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=27, family=2, type=1, proto=6, laddr=('192.168.50.72', 35256), raddr=('34.102.189.10', 443)>
...Exception ignored in: <socket.socket fd=17, family=2, type=1, proto=6, laddr=('192.168.50.72', 44690), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=17, family=2, type=1, proto=6, laddr=('192.168.50.72', 44690), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=32, family=2, type=1, proto=6, laddr=('192.168.50.72', 44700), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=32, family=2, type=1, proto=6, laddr=('192.168.50.72', 44700), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=32, family=2, type=1, proto=6, laddr=('192.168.50.72', 44714), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=32, family=2, type=1, proto=6, laddr=('192.168.50.72', 44714), raddr=('34.102.189.10', 443)>
.Exception ignored in: <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 44730), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=14, family=2, type=1, proto=6, laddr=('192.168.50.72', 44730), raddr=('34.102.189.10', 443)>
.Exception ignored in: <socket.socket fd=27, family=2, type=1, proto=6, laddr=('192.168.50.72', 44736), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=27, family=2, type=1, proto=6, laddr=('192.168.50.72', 44736), raddr=('34.102.189.10', 443)>
Exception ignored in: <socket.socket fd=27, family=2, type=1, proto=6, laddr=('192.168.50.72', 44750), raddr=('34.102.189.10', 443)>
Traceback (most recent call last):
  File "/nix/store/lw9wd3vvhipidiik52px4q7my8463nin-python3-3.12.2-env/lib/python3.12/site-packages/snowflake/connector/vendored/urllib3/poolmanager.py", line 223, in clear
    self.pools.clear()
ResourceWarning: unclosed <socket.socket fd=27, family=2, type=1, proto=6, laddr=('192.168.50.72', 44750), raddr=('34.102.189.10', 443)>
..                                                           [100%]
2907 passed, 31 skipped, 424 xfailed in 472.68s (0:07:52)

cpcloud added this to the 9.0 milestone Mar 17, 2024

cpcloud added the sql Backends that generate SQL label Mar 17, 2024

cpcloud force-pushed the bump-sqlglot branch from cc1b17e to cff2051 Compare March 17, 2024 12:47

cpcloud mentioned this pull request Mar 17, 2024

fix(snowflake): handle null inputs to array zip #8683

Closed

cpcloud force-pushed the bump-sqlglot branch from 559fdda to 1fccab1 Compare March 17, 2024 14:46

cpcloud added tests Issues or PRs related to tests sqlite The SQLite backend duckdb The DuckDB backend labels Mar 18, 2024

cpcloud added the ci-run-cloud Add this label to trigger a run of Bigquery and Snowflake in CI label Mar 18, 2024

ibis-docs-bot bot removed the ci-run-cloud Add this label to trigger a run of Bigquery and Snowflake in CI label Mar 18, 2024

kszucs reviewed Mar 18, 2024

View reviewed changes

ibis/backends/tests/snapshots/test_sql/test_isin_bug/clickhouse/out.sql Outdated Show resolved Hide resolved

kszucs approved these changes Mar 18, 2024

View reviewed changes

kszucs reviewed Mar 18, 2024

View reviewed changes

ibis/backends/__init__.py Show resolved Hide resolved

kszucs reviewed Mar 18, 2024

View reviewed changes

ibis/expr/sql.py Show resolved Hide resolved

cpcloud added 15 commits March 18, 2024 14:12

fix(deps): bump sqlglot to pick up duckdb array fixes

f6e4234

fix(sql): avoid transpiling if read and write dialects are the same

a0e8802

fix(sqlite): fix log2 and log10

2a8ec08

chore(sql): impala

f19840c

chore(sql): fix druid log10

211bc86

chore: remove log10/log2 from simple ops

44d45cb

chore(impala): regen sql

71eeff3

test(mssql): xfail on zero-row fetch

81d1f88

test(math): add log2 test

7abdf90

feat(sqlite): add default log(...) function

0e9c198

test(flink): regen snapshot

ae4ee87

test(bigquery/snowflake): fix zip null test

4e83ea4

feat(sqlite): fixup log

7190377

test(duckdb): only test geospatial on duckdb<0.10

8c1dac1

chore: try installing with duckdb 0.9.2 explicitly

f4456b2

cpcloud added 8 commits March 18, 2024 14:12

test(duckdb): allow testing on 0.9.2

3a449e0

test(duckdb): skip geospatial example test if pins is not installed

96106eb

chore: pass query=expr to preserve subquery formatting

5b88296

chore: revert snapshots

f488a1d

chore: remove last uses of .to_sql

7569a83

chore: remove definitions of _to_sql

0b6d7b4

chore(clickhouse): snapshot gen

3dc6153

chore(tpch): revert snapshots

7cc3e2e

cpcloud force-pushed the bump-sqlglot branch from 94d6726 to 7cc3e2e Compare March 18, 2024 18:13

cpcloud added 2 commits March 18, 2024 14:13

chore(tpch): revert snapshots

e71b35b

chore(impala): revert snapshots

ce095a9

cpcloud merged commit a3bd853 into ibis-project:main Mar 18, 2024
83 checks passed

cpcloud deleted the bump-sqlglot branch March 18, 2024 20:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(deps): bump sqlglot to pick up duckdb array fixes #8682

fix(deps): bump sqlglot to pick up duckdb array fixes #8682

cpcloud commented Mar 17, 2024 •

edited

Loading

cpcloud commented Mar 18, 2024 •

edited

Loading

cpcloud commented Mar 18, 2024

cpcloud commented Mar 18, 2024

fix(deps): bump sqlglot to pick up duckdb array fixes #8682

fix(deps): bump sqlglot to pick up duckdb array fixes #8682

Conversation

cpcloud commented Mar 17, 2024 • edited Loading

cpcloud commented Mar 18, 2024 • edited Loading

cpcloud commented Mar 18, 2024

cpcloud commented Mar 18, 2024

cpcloud commented Mar 17, 2024 •

edited

Loading

cpcloud commented Mar 18, 2024 •

edited

Loading