Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: fix pyspark ddl tests inherited from the old spark backend #2879

Closed
datapythonista opened this issue Jul 26, 2021 · 0 comments · Fixed by #3365
Closed

test: fix pyspark ddl tests inherited from the old spark backend #2879

datapythonista opened this issue Jul 26, 2021 · 0 comments · Fixed by #3365
Labels
pyspark The Apache PySpark backend tests Issues or PRs related to tests

Comments

@datapythonista
Copy link
Contributor

The PySpark backend used to subclass the legacy Spark backend. Including the DDL functionality. When the old Spark backend was deleted in #2869, the functionality that PySpark used to inherit, was moved to the PySpark test. In the case of DDL, when the tests were moved from the Spark backend to the PySpark one, looks like some of the functionality inherited by the PySpark backend was not working. Since the DDL functionality was tested only for the Spark backend, it wasn't known until tests were moved that some things were broken in the PySpark backend.

The tests that are failing are (all them in ibis/backends/pyspark/tests/test_ddl.py):

  • test_truncate_table
  • test_truncate_table_expression
  • test_insert_table
  • test_create_table_reserved_identifier
  • test_create_table_or_temp_view_from_csv

All them have been xfailed for now, but should be eventually fixed (or the functionality should be removed if it doesn't make sense to have it in the PySpark backend).

@datapythonista datapythonista added tests Issues or PRs related to tests pyspark The Apache PySpark backend labels Jul 26, 2021
@cpcloud cpcloud changed the title TST: Fix pyspark ddl tests inherited from the old spark backend test: fix pyspark ddl tests inherited from the old spark backend Dec 28, 2021
cpcloud added a commit to cpcloud/ibis that referenced this issue Jan 27, 2022
cpcloud added a commit to cpcloud/ibis that referenced this issue Feb 2, 2022
cpcloud added a commit that referenced this issue Feb 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pyspark The Apache PySpark backend tests Issues or PRs related to tests
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant