You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The PySpark backend used to subclass the legacy Spark backend. Including the DDL functionality. When the old Spark backend was deleted in #2869, the functionality that PySpark used to inherit, was moved to the PySpark test. In the case of DDL, when the tests were moved from the Spark backend to the PySpark one, looks like some of the functionality inherited by the PySpark backend was not working. Since the DDL functionality was tested only for the Spark backend, it wasn't known until tests were moved that some things were broken in the PySpark backend.
The tests that are failing are (all them in ibis/backends/pyspark/tests/test_ddl.py):
test_truncate_table
test_truncate_table_expression
test_insert_table
test_create_table_reserved_identifier
test_create_table_or_temp_view_from_csv
All them have been xfailed for now, but should be eventually fixed (or the functionality should be removed if it doesn't make sense to have it in the PySpark backend).
The text was updated successfully, but these errors were encountered:
cpcloud
changed the title
TST: Fix pyspark ddl tests inherited from the old spark backend
test: fix pyspark ddl tests inherited from the old spark backend
Dec 28, 2021
cpcloud
added a commit
to cpcloud/ibis
that referenced
this issue
Jan 27, 2022
The PySpark backend used to subclass the legacy Spark backend. Including the DDL functionality. When the old Spark backend was deleted in #2869, the functionality that PySpark used to inherit, was moved to the PySpark test. In the case of DDL, when the tests were moved from the Spark backend to the PySpark one, looks like some of the functionality inherited by the PySpark backend was not working. Since the DDL functionality was tested only for the Spark backend, it wasn't known until tests were moved that some things were broken in the PySpark backend.
The tests that are failing are (all them in
ibis/backends/pyspark/tests/test_ddl.py):test_truncate_tabletest_truncate_table_expressiontest_insert_tabletest_create_table_reserved_identifiertest_create_table_or_temp_view_from_csvAll them have been xfailed for now, but should be eventually fixed (or the functionality should be removed if it doesn't make sense to have it in the PySpark backend).
The text was updated successfully, but these errors were encountered: