Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FEAT: Spark UDFs #1885

Merged
merged 51 commits into from
Aug 2, 2019
Merged

FEAT: Spark UDFs #1885

merged 51 commits into from
Aug 2, 2019

Conversation

amroid
Copy link
Contributor

@amroid amroid commented Jul 16, 2019 •
edited by cpcloud
Loading

Closes #1889

amroid added 30 commits June 5, 2019 15:57
@amroid
added spark client, compiler, some unit tests
bbb7244
@amroid
added kwargs for SparkClient and SparkContext
66067d9
@amroid
added Spark subclass of Backend
78f8fc1
@amroid
fixed PR changes
dc3608b
@amroid
Merge branch 'spark-client' into spark-tests
d55c64a
@amroid
fixed SparkUnion bug
0d7bb67
@amroid
test_string now works with Spark backend, added xfails and xpasses to…
54422ef 
… regex tests in test_string
@amroid
test_aggregation now works with Spark backend, changed base compiler …
bc9fff7 
…rewrite for any, notany, all, notall to use max and min instead of sum
@amroid
test_array now works with Spark backend, fixed Spark table creation a…
0560ada 
…nd tests
@amroid
added pyspark>=2.4.3 to requirements-3.x-dev.yml files
3b30726
@amroid
test_client now works with Spark backend, fixed mistake in test_sql i…
c96cb58 
…n test_client.py
@amroid
test_client now works with Spark backend, fixed mistake in test_sql i…
f934e40 
…n test_client.py
@amroid
test_column, test_generic now work with Spark backend (test_column re…
a8ecb0e 
…quired no changes)
@amroid
test_numeric now works with Spark backend, changed impala compiler im…
2ea6ac8 
…plementation of _number_literal_format
@amroid
test_param now works with Spark backend
45e80c6
@amroid
test_temporal now mostly works with Spark backend with some known iss…
2aa6d7a 
…ues, added floor option in convert_unit in util.py
@amroid
added string concat
c1b122f
@amroid
changed imports to not fail CI
87a2ece
@amroid
incorporated suggestions for spark-client
8c398ca
@amroid
Merge branch 'master' of https://github.com/ibis-project/ibis into sp…
1083348 
…ark-tests
@amroid
Basic elementwise UDFs
eb060ac
@amroid
Clean up basic udf functionality
c2bf387
@amroid
Break up spark to ibis type conversion into separate dt.dtype registe…
ebd8f08 
…red functions
@amroid
Break up spark to ibis type conversion into separate dt.dtype registe…
c3def6b 
…red functions
@amroid
Correct nullable behavior for spark to ibis type translation, fix tes…
8cb80ae 
…ts for Spark backend
@amroid
Correct nullable behavior for spark to ibis type translation, fix tes…
c8850cc 
…ts for Spark backend
@amroid
Add changes from PR review
6bae62c
@amroid
Add ibis to spark dtype conversion for UDFs
5c5fc57
@amroid
Fix TypeSignature validate method so that it uses inspect and has bet…
af9ceb3 
…ter argument checking with the signature (now errors if too many arguments are passed to a UDF)
@amroid
Tests now pass
54f948c
@cpcloud cpcloud self-requested a review July 18, 2019 21:11
@cpcloud cpcloud added this to the Next Feature Release milestone Jul 18, 2019
@cpcloud cpcloud added the spark label Jul 18, 2019
@cpcloud cpcloud changed the title Spark udf FEAT: Spark UDFs Jul 18, 2019
@cpcloud
Copy link
Member

cpcloud commented Jul 25, 2019

@amroid Looks like a few merge conflicts.

@pep8speaks
Copy link

pep8speaks commented Jul 26, 2019 •
edited
Loading

Hello @amroid! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-07-31 20:39:41 UTC

cpcloud
cpcloud requested changes Jul 29, 2019
View reviewed changes
Copy link
Member

@cpcloud cpcloud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job on abstracting out the details of the various spark UDFs! One or two more rounds of review and this should be good to go.

ibis/spark/tests/test_udf.py Outdated Show resolved Hide resolved
ibis/spark/tests/test_udf.py Outdated Show resolved Hide resolved
ibis/spark/tests/test_udf.py Outdated Show resolved Hide resolved
ibis/spark/tests/test_udf.py Outdated Show resolved Hide resolved
ibis/spark/tests/test_udf.py Outdated Show resolved Hide resolved
ibis/spark/udf.py Outdated Show resolved Hide resolved
ibis/spark/udf.py Outdated Show resolved Hide resolved
ibis/spark/udf.py Outdated Show resolved Hide resolved
ibis/spark/udf.py Outdated Show resolved Hide resolved
ibis/spark/client.py Outdated Show resolved Hide resolved
@amroid
Refactor spark udf code so elementwise_pandas is now elementwise.pand…
7dd5541 
…as, some changes to test_udf, fix bug in spark compiler SparkContext class
cpcloud
cpcloud requested changes Jul 31, 2019
View reviewed changes
Copy link
Member

@cpcloud cpcloud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some changes needed as a result of the most recent set of updates.

ibis/spark/tests/test_udf.py Outdated Show resolved Hide resolved
ibis/spark/tests/test_udf.py Outdated Show resolved Hide resolved
ibis/spark/tests/test_udf.py Outdated Show resolved Hide resolved
ibis/spark/tests/test_udf.py Outdated Show resolved Hide resolved
ibis/spark/tests/test_udf.py Outdated Show resolved Hide resolved
@cpcloud cpcloud merged commit 96f4ea4 into ibis-project:master Aug 2, 2019
@cpcloud
Copy link
Member

cpcloud commented Aug 2, 2019

Thanks @amroid! Great work!

@amroid amroid deleted the spark-udf branch August 2, 2019 18:24
@cpcloud cpcloud modified the milestones: Next Feature Release, Next Major Release Aug 26, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cpcloud cpcloud cpcloud requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

FEAT: Add UDF support to spark
3 participants
@amroid @cpcloud @pep8speaks