-
Notifications
You must be signed in to change notification settings - Fork 590
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bug: impossible to pass dateformat and columns parameters to duckdb.read_csv from ibis.read_csv #6190
Comments
|
Related question: assuming I would first load the CSV using the automatically inferred dtype, I could not find in the documentation how to cast an Are there generic tools to handle date/datetime parsing and formatting in Ibis? If no, I can open a feature request. Otherwise, I can open an issue to improve the documentation to make that easier to find (e.g. by cross linking the API/docstring to the user guide/tutorial/howtos). |
|
Finally, the I tried both but got similar error messages in Ibis. |
|
I can reproduce this error even using duckdb parameters: table = ibis.read_csv(
"example.csv",
header=True,
columns={'id': 'BIGINT', 'date_col': 'DATE'},
dateformat="%Y%m%d",
)the issue is the function basically def _format_kwargs(kwargs: Mapping[str, Any]):
bindparams, pieces = [], []
for name, value in kwargs.items():
if name != "columns":
bindparams.append(sa.bindparam(name, value))
pieces.append(f"{name} = :{name}")
if "columns" in kwargs:
pieces.append(f"columns = {kwargs['columns']}")
return sa.text(", ".join(pieces)).bindparams(*bindparams)This makes the code: table = ibis.read_csv(
"example.csv",
header=True,
columns={'id': 'BIGINT', 'date_col': 'DATE'},
dateformat="%Y%m%d",
)
print(table.head().execute())produce: |
What happened?
I am trying to parse a custom date format directly when loading CSV data from ibis using the default duckdb backend.
I use the raw duckdb API, I can pass
dtypeanddate_formatto this efficiently:and it works as expected:
However, I if want to do this from ibis I get the following exception:
Full-traceback:
--------------------------------------------------------------------------- CompileError Traceback (most recent call last) Cell In[10], line 3 1 import ibis ----> 3 ibis.read_csv( 4 "example.csv", 5 table_name="members", 6 dtype={"date_col": "date"}, 7 date_format="%Y%m%d", 8 ) File [~/code/ibis/ibis/expr/api.py:907](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/code/ibis/ibis/expr/api.py:907), in read_csv(sources, **kwargs) 904 from ibis.config import _default_backend 906 con = _default_backend() --> 907 return con.read_csv(sources, **kwargs) File [~/code/ibis/ibis/backends/duckdb/__init__.py:321](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/code/ibis/ibis/backends/duckdb/__init__.py:321), in Backend.read_csv(self, source_list, table_name, **kwargs) 316 kwargs["auto_detect"] = kwargs.pop("auto_detect", "columns" not in kwargs) 317 source = sa.select(sa.literal_column("*")).select_from( 318 sa.func.read_csv(sa.func.list_value(*source_list), _format_kwargs(kwargs)) 319 ) --> 321 view = self._compile_temp_view(table_name, source) 322 with self.begin() as con: 323 con.exec_driver_sql(view) File [~/code/ibis/ibis/backends/duckdb/__init__.py:230](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/code/ibis/ibis/backends/duckdb/__init__.py:230), in Backend._compile_temp_view(self, table_name, source) 229 def _compile_temp_view(self, table_name, source): --> 230 raw_source = source.compile( 231 dialect=self.con.dialect, compile_kwargs=dict(literal_binds=True) 232 ) 233 return f'CREATE OR REPLACE TEMPORARY VIEW "{table_name}" AS {raw_source}' File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/elements.py:503](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/elements.py:503), in ClauseElement.compile(self, bind, dialect, **kw) 498 url = util.preloaded.engine_url 499 dialect = url.URL.create( 500 self.stringify_dialect 501 ).get_dialect()() --> 503 return self._compiler(dialect, **kw) File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/elements.py:567](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/elements.py:567), in ClauseElement._compiler(self, dialect, **kw) 563 def _compiler(self, dialect, **kw): 564 """Return a compiler appropriate for this ClauseElement, given a 565 Dialect.""" --> 567 return dialect.statement_compiler(dialect, self, **kw) File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:809](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:809), in SQLCompiler.__init__(self, dialect, statement, cache_key, column_keys, for_executemany, linting, **kwargs) 805 # a map which tracks "truncated" names based on 806 # dialect.label_length or dialect.max_identifier_length 807 self.truncated_names = {} --> 809 Compiled.__init__(self, dialect, statement, **kwargs) 811 if self.isinsert or self.isupdate or self.isdelete: 812 if statement._returning: File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:464](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:464), in Compiled.__init__(self, dialect, statement, schema_translate_map, render_schema_translate, compile_kwargs) 462 if self.can_execute: 463 self.execution_options = statement._execution_options --> 464 self.string = self.process(self.statement, **compile_kwargs) 466 if render_schema_translate: 467 self.string = self.preparer._render_schema_translates( 468 self.string, schema_translate_map 469 ) File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:499](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:499), in Compiled.process(self, obj, **kwargs) 498 def process(self, obj, **kwargs): --> 499 return obj._compiler_dispatch(self, **kwargs) File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/visitors.py:82](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/visitors.py:82), in _generate_compiler_dispatch.._compiler_dispatch(self, visitor, **kw) 79 return visitor.visit_unsupported_compilation(self, err, **kw) 81 else: ---> 82 return meth(self, **kw) File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:3545](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:3545), in SQLCompiler.visit_select(self, select_stmt, asfrom, insert_into, fromhints, compound_index, select_wraps_for, lateral, from_linter, **kwargs) 3515 translate = dict( 3516 zip( 3517 [ (...) 3537 ) 3538 ) 3540 self._result_columns = [ 3541 (key, name, tuple(translate.get(o, o) for o in obj), type_) 3542 for key, name, obj, type_ in self._result_columns 3543 ] -> 3545 text = self._compose_select_body( 3546 text, 3547 select_stmt, 3548 compile_state, 3549 inner_columns, 3550 froms, 3551 byfrom, 3552 toplevel, 3553 kwargs, 3554 ) 3556 if select_stmt._statement_hints: 3557 per_dialect = [ 3558 ht 3559 for (dialect_name, ht) in select_stmt._statement_hints 3560 if dialect_name in ("*", self.dialect.name) 3561 ] File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:3692](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:3692), in SQLCompiler._compose_select_body(self, text, select, compile_state, inner_columns, froms, byfrom, toplevel, kwargs) 3678 text += ", ".join( 3679 [ 3680 f._compiler_dispatch( (...) 3688 ] 3689 ) 3690 else: 3691 text += ", ".join( -> 3692 [ 3693 f._compiler_dispatch( 3694 self, 3695 asfrom=True, 3696 from_linter=from_linter, 3697 **kwargs 3698 ) 3699 for f in froms 3700 ] 3701 ) 3702 else: 3703 text += self.default_from() File ~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:3693, in (.0) 3678 text += ", ".join( 3679 [ 3680 f._compiler_dispatch( (...) 3688 ] 3689 ) 3690 else: 3691 text += ", ".join( 3692 [ -> 3693 f._compiler_dispatch( 3694 self, 3695 asfrom=True, 3696 from_linter=from_linter, 3697 **kwargs 3698 ) 3699 for f in froms 3700 ] 3701 ) 3702 else: 3703 text += self.default_from() File ~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/visitors.py:82, in _generate_compiler_dispatch.._compiler_dispatch(self, visitor, **kw) 79 return visitor.visit_unsupported_compilation(self, err, **kw) 81 else: ---> 82 return meth(self, **kw) File ~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:1905, in SQLCompiler.visit_function(self, func, add_to_result_map, **kwargs) 1887 name = ( 1888 self.preparer.quote(name) 1889 if self.preparer._requires_quotes_illegal_chars(name) 1890 or isinstance(name, elements.quoted_name) 1891 else name 1892 ) 1893 name = name + "%(expr)s" 1894 text = ".".join( 1895 [ 1896 ( 1897 self.preparer.quote(tok) 1898 if self.preparer._requires_quotes_illegal_chars(tok) 1899 or isinstance(name, elements.quoted_name) 1900 else tok 1901 ) 1902 for tok in func.packagenames 1903 ] 1904 + [name] -> 1905 ) % {"expr": self.function_argspec(func, **kwargs)} 1907 if func._with_ordinality: 1908 text += " WITH ORDINALITY" File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:1921](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:1921), in SQLCompiler.function_argspec(self, func, **kwargs) 1920 def function_argspec(self, func, **kwargs): -> 1921 return func.clause_expr._compiler_dispatch(self, **kwargs) File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/visitors.py:82](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/visitors.py:82), in _generate_compiler_dispatch.._compiler_dispatch(self, visitor, **kw) 79 return visitor.visit_unsupported_compilation(self, err, **kw) 81 else: ---> 82 return meth(self, **kw) File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:1430](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:1430), in SQLCompiler.visit_grouping(self, grouping, asfrom, **kwargs) 1429 def visit_grouping(self, grouping, asfrom=False, **kwargs): -> 1430 return "(" + grouping.element._compiler_dispatch(self, **kwargs) + ")" File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/visitors.py:82](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/visitors.py:82), in _generate_compiler_dispatch.._compiler_dispatch(self, visitor, **kw) 79 return visitor.visit_unsupported_compilation(self, err, **kw) 81 else: ---> 82 return meth(self, **kw) File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:1767](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:1767), in SQLCompiler.visit_clauselist(self, clauselist, **kw) 1764 else: 1765 sep = OPERATORS[clauselist.operator] -> 1767 return self._generate_delimited_list(clauselist.clauses, sep, **kw) File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:1733](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:1733), in SQLCompiler._generate_delimited_list(self, elements, separator, **kw) 1732 def _generate_delimited_list(self, elements, separator, **kw): -> 1733 return separator.join( 1734 s 1735 for s in (c._compiler_dispatch(self, **kw) for c in elements) 1736 if s 1737 ) File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:1733](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:1733), in (.0) 1732 def _generate_delimited_list(self, elements, separator, **kw): -> 1733 return separator.join( 1734 s 1735 for s in (c._compiler_dispatch(self, **kw) for c in elements) 1736 if s 1737 ) File ~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:1735, in (.0) 1732 def _generate_delimited_list(self, elements, separator, **kw): 1733 return separator.join( 1734 s -> 1735 for s in (c._compiler_dispatch(self, **kw) for c in elements) 1736 if s 1737 ) File ~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/visitors.py:82, in _generate_compiler_dispatch.._compiler_dispatch(self, visitor, **kw) 79 return visitor.visit_unsupported_compilation(self, err, **kw) 81 else: ---> 82 return meth(self, **kw) File ~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:1652, in SQLCompiler.visit_textclause(self, textclause, add_to_result_map, **kw) 1647 add_to_result_map(None, None, (textclause,), sqltypes.NULLTYPE) 1649 # un-escape any \:params 1650 return BIND_PARAMS_ESC.sub( 1651 lambda m: m.group(1), -> 1652 BIND_PARAMS.sub( 1653 do_bindparam, self.post_process_text(textclause.text) 1654 ), 1655 ) File ~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:1636, in SQLCompiler.visit_textclause..do_bindparam(m) 1634 name = m.group(1) 1635 if name in textclause._bindparams: -> 1636 return self.process(textclause._bindparams[name], **kw) 1637 else: 1638 return self.bindparam_string(name, **kw) File ~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:499, in Compiled.process(self, obj, **kwargs) 498 def process(self, obj, **kwargs): --> 499 return obj._compiler_dispatch(self, **kwargs) File ~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/visitors.py:82, in _generate_compiler_dispatch.._compiler_dispatch(self, visitor, **kw) 79 return visitor.visit_unsupported_compilation(self, err, **kw) 81 else: ---> 82 return meth(self, **kw) File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:2528](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:2528), in SQLCompiler.visit_bindparam(self, bindparam, within_columns_clause, literal_binds, skip_bind_expression, literal_execute, render_postcompile, **kwargs) 2525 post_compile = False 2527 if literal_binds: -> 2528 ret = self.render_literal_bindparam( 2529 bindparam, within_columns_clause=True, **kwargs 2530 ) 2531 if bindparam.expanding: 2532 ret = "(%s)" % ret File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:2644](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:2644), in SQLCompiler.render_literal_bindparam(self, bindparam, render_literal_value, bind_expression_template, **kw) 2642 return replacement_expr 2643 else: -> 2644 return self.render_literal_value(value, bindparam.type) File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/dialects/postgresql/base.py:2445](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/dialects/postgresql/base.py:2445), in PGCompiler.render_literal_value(self, value, type_) 2444 def render_literal_value(self, value, type_): -> 2445 value = super(PGCompiler, self).render_literal_value(value, type_) 2447 if self.dialect._backslash_escapes: 2448 value = value.replace("\\", "\\\\") File [~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:2678](https://file+.vscode-resource.vscode-cdn.net/Users/ogrisel/code/survival-analysis-benchmark/datasets/kkbox_churn/~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/sqlalchemy/sql/compiler.py:2678), in SQLCompiler.render_literal_value(self, value, type_) 2662 util.raise_( 2663 exc.CompileError( 2664 "Could not render literal value " (...) 2674 from_=e, 2675 ) 2677 else: -> 2678 raise exc.CompileError( 2679 "No literal value renderer is available for literal value " 2680 '"%s" with datatype %s' 2681 % (sql_util._repr_single_value(value), type_) 2682 ) CompileError: No literal value renderer is available for literal value "{'date_col': 'date'}" with datatype NULLEDIT: as discussed below the correct backend parameter names are
dateformatandcolumnsbut the problem stays the same.What version of ibis are you using?
ibis 5.0.0
duckdb 0.7.1
What backend(s) are you using, if any?
DuckDB
Code of Conduct
The text was updated successfully, but these errors were encountered: