Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Add support for np.array literals #2603

Merged

Conversation

timothydijamco
Copy link
Contributor

This PR allows np.array literals in the PySpark backend. This was already supported on the Pandas backend.

import numpy as np

from ibis import backends
from ibis.expr.types import literal

spark = ...

ibis_pyspark = backends.pyspark.connect(spark)

ibis_pyspark.execute(literal(np.array([1, 2])))

Before:

AttributeError: 'numpy.int64' object has no attribute '_get_object_id'

After:

array([1, 2], dtype=int32)

Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a release note as well. ping on green.

ibis/backends/tests/test_array.py Show resolved Hide resolved
@jreback jreback added the pyspark The Apache PySpark backend label Jan 29, 2021
@jreback jreback added this to the Next release milestone Jan 29, 2021
@timothydijamco timothydijamco changed the title BUG: Add PySpark support for np.array literals ENH: Add PySpark support for np.array literals Feb 1, 2021
@timothydijamco
Copy link
Contributor Author

#2605 will take care of one of the failing tests (SQLite is supposed to skip test_np_array). I'll have to make a few more changes to fix the rest

@timothydijamco
Copy link
Contributor Author

timothydijamco commented Feb 2, 2021

All tests are passing now. I increased the scope slightly—this PR adds support for (1-d) np.array as literals for backends that support array/list literals in general (basically, all backends excluding MySQL and SQLite)

@timothydijamco timothydijamco changed the title ENH: Add PySpark support for np.array literals ENH: Add support for np.array literals Feb 2, 2021
@jreback jreback merged commit dccaec8 into ibis-project:master Feb 2, 2021
@jreback
Copy link
Contributor

jreback commented Feb 2, 2021

thanks @timothydijamco

@cpcloud cpcloud removed this from the Next release milestone Jan 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pyspark The Apache PySpark backend
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants