Skip to content

bug: union fails if string lengths do not match in 10.4.0 #11287

Open
@rwhitten577

Description

@rwhitten577

What happened?

After upgrading existing Ibis code that uses Table.union to 10.4.0, I started receiving error messages about string lengths not matching, e.g.:

 Traceback (most recent call last):
    |   File "/Users/.../.venv/lib/python3.11/site-packages/ibis/expr/operations/relations.py", line 343, in __init__
    |     missing_from_left = right.schema - left.schema
    |                         ~~~~~~~~~~~~~^~~~~~~~~~~~~
    |   File "/Users/.../.venv/lib/python3.11/site-packages/ibis/common/collections.py", line 247, in __sub__
    |     common_keys = self._check_conflict(other)
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/.../.venv/lib/python3.11/site-packages/ibis/common/collections.py", line 212, in _check_conflict
    |     raise ConflictingValuesError(conflicts)
    | ibis.common.exceptions.ConflictingValuesError: Conflicting values for keys:
    |   `customer_id`: string(50) != string(4000)

In this scenario I am trying to union tables created via backend.sql(query_str) where each source table has a different length for the column in Snowflake, e.g.
query 1:

select customer_id from some_table1

query 2:

select customer_id from  some_table2

Is this a bug, or expected behavior after the implementation of #9382?

I can understand preference of the error to force explicit casting, but wondering if there is a way to keep the old upscaling behavior for the cases when it doesn't matter if the smaller types are kept. To work around this I'd have to always cast a few hundred columns to strings of a fixed length.

What version of ibis are you using?

10.4.0

What backend(s) are you using, if any?

Snowflake

Relevant log output

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugIncorrect behavior inside of ibis

    Type

    No type

    Projects

    Status

    backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions