Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: pass scope to adjust_context in PySpark backend compiler #3108

Closed
timothydijamco opened this issue Nov 3, 2021 · 0 comments · Fixed by #3307
Closed

bug: pass scope to adjust_context in PySpark backend compiler #3108

timothydijamco opened this issue Nov 3, 2021 · 0 comments · Fixed by #3307
Labels
bug Incorrect behavior inside of ibis pyspark The Apache PySpark backend

Comments

@timothydijamco
Copy link
Contributor

#3037 was working towards making sure scope is available in all context-adjustment functions (see that PR for justification)

There is one place in the PySpark backend that directly calls a adjust_context function, but doesn't pass scope.
https://github.com/ibis-project/ibis/blob/master/ibis/backends/pyspark/compiler.py#L125-L129
We should pass scope here.
@cpcloud cpcloud changed the title Pass scope to adjust_context in PySpark backend compiler bug: pass scope to adjust_context in PySpark backend compiler Dec 28, 2021
@cpcloud cpcloud added bug Incorrect behavior inside of ibis pyspark The Apache PySpark backend labels Dec 28, 2021
timothydijamco pushed a commit to timothydijamco/ibis that referenced this issue Jan 13, 2022
fix(context-adjustment): pass scope when calling adjust_context in py…
6e0ed6f 
…spark backend

The Pandas and Dask backends always pass scope to adjust_context, so that adjust_context can potentially make use of extra information in scope. The PySpark backend was not passing scope to adjust_context, which can lead to behavior differences or errors across backends.

fixes ibis-project#3108
@timothydijamco timothydijamco mentioned this issue Jan 13, 2022
Merged
timothydijamco pushed a commit to timothydijamco/ibis that referenced this issue Jan 13, 2022
fix(context-adjustment): pass scope when calling adjust_context in py…
d10dfb1 
…spark backend

The Pandas and Dask backends always pass scope to adjust_context, so that adjust_context can potentially make use of extra information in scope. The PySpark backend was not passing scope to adjust_context, which can lead to behavior differences or errors across backends.

fixes ibis-project#3108
cpcloud pushed a commit to timothydijamco/ibis that referenced this issue Jan 15, 2022
@cpcloud
fix(context-adjustment): pass scope when calling adjust_context in py…
3ee1127 
…spark backend

The Pandas and Dask backends always pass scope to adjust_context, so
that adjust_context can potentially make use of extra information in
scope. The PySpark backend was not passing scope to adjust_context,
which can lead to behavior differences or errors across backends.

fixes ibis-project#3108
cpcloud pushed a commit that referenced this issue Jan 19, 2022
@cpcloud
fix(context-adjustment): pass scope when calling adjust_context in py…
33aad7b 
…spark backend

The Pandas and Dask backends always pass scope to adjust_context, so
that adjust_context can potentially make use of extra information in
scope. The PySpark backend was not passing scope to adjust_context,
which can lead to behavior differences or errors across backends.

fixes #3108
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Incorrect behavior inside of ibis pyspark The Apache PySpark backend
Projects
None yet
Milestone
No milestone
2 participants
@cpcloud @timothydijamco