Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: incorrect SQL generated on main (but not TES) for generating random numbers #8013

Closed
1 task done
lostmygithubaccount opened this issue Jan 18, 2024 · 0 comments · Fixed by #8014
Closed
1 task done
Assignees
Labels
bug Incorrect behavior inside of ibis
Milestone

Comments

@lostmygithubaccount
Copy link
Member

What happened?

I have this old code for generating a billion random numbers w/ an ugly SQL string:

import ibis

con = ibis.connect("duckdb://data.ddb")

ROWS = 1_000_000_000

sql_str = ""
sql_str += "select\n"
for c in list(map(chr, range(ord("a"), ord("z") + 1))):
    sql_str += f"  random() as {c},\n"
sql_str += f"from generate_series(1, {ROWS})"

t = con.sql(sql_str)
con.create_table("billion", t, overwrite=True)

I tried to translate it to Ibis expressions as:

t = (
    ibis.range(10)
    .unnest()
    .name("index")
    .as_table()
    .mutate({c: ibis.random() for c in list(map(chr, range(ord("a"), ord("z") + 1)))})
)

resulting in:

┏━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃ index ┃ 'a'    ┃ 'b'    ┃ 'c'    ┃ 'd'    ┃ 'e'    ┃ 'f'    ┃ 'g'    ┃ 'h'    ┃ 'i'    ┃ 'j'    ┃ 'k'    ┃ 'l'    ┃ 'm'    ┃ 'n'    ┃ 'o'    ┃ 'p'    ┃ 'q'    ┃ 'r'    ┃ 's'    ┃ 't'    ┃ 'u'    ┃ 'v'    ┃ 'w'    ┃ 'x'    ┃ 'y'    ┃ 'z'    ┃
┡━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│ int8  │ string │ string │ string │ string │ string │ string │ string │ string │ string │ string │ string │ string │ string │ string │ string │ string │ string │ string │ string │ string │ string │ string │ string │ string │ string │ string │
├───────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
│     0 │ a      │ b      │ c      │ d      │ e      │ f      │ g      │ h      │ i      │ j      │ k      │ l      │ m      │ n      │ o      │ p      │ q      │ r      │ s      │ t      │ u      │ v      │ w      │ x      │ y      │ z      │
│     1 │ a      │ b      │ c      │ d      │ e      │ f      │ g      │ h      │ i      │ j      │ k      │ l      │ m      │ n      │ o      │ p      │ q      │ r      │ s      │ t      │ u      │ v      │ w      │ x      │ y      │ z      │
│     2 │ a      │ b      │ c      │ d      │ e      │ f      │ g      │ h      │ i      │ j      │ k      │ l      │ m      │ n      │ o      │ p      │ q      │ r      │ s      │ t      │ u      │ v      │ w      │ x      │ y      │ z      │
│     3 │ a      │ b      │ c      │ d      │ e      │ f      │ g      │ h      │ i      │ j      │ k      │ l      │ m      │ n      │ o      │ p      │ q      │ r      │ s      │ t      │ u      │ v      │ w      │ x      │ y      │ z      │
│     4 │ a      │ b      │ c      │ d      │ e      │ f      │ g      │ h      │ i      │ j      │ k      │ l      │ m      │ n      │ o      │ p      │ q      │ r      │ s      │ t      │ u      │ v      │ w      │ x      │ y      │ z      │
│     5 │ a      │ b      │ c      │ d      │ e      │ f      │ g      │ h      │ i      │ j      │ k      │ l      │ m      │ n      │ o      │ p      │ q      │ r      │ s      │ t      │ u      │ v      │ w      │ x      │ y      │ z      │
│     6 │ a      │ b      │ c      │ d      │ e      │ f      │ g      │ h      │ i      │ j      │ k      │ l      │ m      │ n      │ o      │ p      │ q      │ r      │ s      │ t      │ u      │ v      │ w      │ x      │ y      │ z      │
│     7 │ a      │ b      │ c      │ d      │ e      │ f      │ g      │ h      │ i      │ j      │ k      │ l      │ m      │ n      │ o      │ p      │ q      │ r      │ s      │ t      │ u      │ v      │ w      │ x      │ y      │ z      │
│     8 │ a      │ b      │ c      │ d      │ e      │ f      │ g      │ h      │ i      │ j      │ k      │ l      │ m      │ n      │ o      │ p      │ q      │ r      │ s      │ t      │ u      │ v      │ w      │ x      │ y      │ z      │
│     9 │ a      │ b      │ c      │ d      │ e      │ f      │ g      │ h      │ i      │ j      │ k      │ l      │ m      │ n      │ o      │ p      │ q      │ r      │ s      │ t      │ u      │ v      │ w      │ x      │ y      │ z      │
└───────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘

this is the SQL it's generating:

sql
SELECT
  t0.index,
  'a' AS "'a'",
  'b' AS "'b'",
  'c' AS "'c'",
  'd' AS "'d'",
  'e' AS "'e'",
  'f' AS "'f'",
  'g' AS "'g'",
  'h' AS "'h'",
  'i' AS "'i'",
  'j' AS "'j'",
  'k' AS "'k'",
  'l' AS "'l'",
  'm' AS "'m'",
  'n' AS "'n'",
  'o' AS "'o'",
  'p' AS "'p'",
  'q' AS "'q'",
  'r' AS "'r'",
  's' AS "'s'",
  't' AS "'t'",
  'u' AS "'u'",
  'v' AS "'v'",
  'w' AS "'w'",
  'x' AS "'x'",
  'y' AS "'y'",
  'z' AS "'z'"
FROM (
  SELECT
    UNNEST(RANGE(CAST(0 AS TINYINT), CAST(10 AS TINYINT), CAST(1 AS TINYINT))) AS index
) AS t0

confirmed by Phillip the same is working on the-epic-split branch. planning to show my old vs new code in #8004

What version of ibis are you using?

main

What backend(s) are you using, if any?

DuckDB

Relevant log output

n/a

Code of Conduct

  • I agree to follow this project's Code of Conduct
@lostmygithubaccount lostmygithubaccount added the bug Incorrect behavior inside of ibis label Jan 18, 2024
@cpcloud cpcloud self-assigned this Jan 18, 2024
@cpcloud cpcloud added this to the 8.0 milestone Jan 18, 2024
cpcloud added a commit that referenced this issue Jan 18, 2024
…te`and`select` APIs (#8014)

We were not handling unsplatted dicts in mutate/select, this PR fixes
that. Fixes #8013.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Incorrect behavior inside of ibis
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants