[SPARK-47816][CONNECT][DOCS] Document the lazy evaluation of views in spark.{sql, table}
#46007
Conversation
| In Classic Spark, a referenced temporary view is resolved immediately, while in Spark | ||
| Connect it is lazy evaluated. | ||
| So in Spark Connect if a view is dropped, modified or replaced after `spark.sql`, the | ||
| execution may fail or generate different results. |
There was a problem hiding this comment.
Out of cusiority, in which cases the execution may fail?
There was a problem hiding this comment.
drop the view, for example
df = ...
df.createTempView("some_view")
df2 = spark.sql("SELECT * FROM some_view")
spark.catalog.dropTempView("some_view") <- drop the view
df2.show() <- should fail in Spark Connect
|
LGTM thanks! |
zhengruifeng
force-pushed
the
doc_connect_sql_table
branch
from
31032cb
to
e3fbdae
Compare
|
thanks @HyukjinKwon and @xinrong-meng merged to master |
| @@ -1630,6 +1630,13 @@ def sql( | |||
| ------- | |||
| :class:`DataFrame` | |||
|
|
|||
| Notes | |||
| ----- | |||
| In Spark Classic, a temporary view referenced in `spark.sql` is resolved immediately, | |||
There was a problem hiding this comment.
How about temp functions?
| Notes | ||
| ----- | ||
| In Spark Classic, a temporary view referenced in `spark.sql` is resolved immediately, | ||
| while in Spark Connect it is lazily evaluated. |
There was a problem hiding this comment.
I think this note might be very confusing to users, as data frames in Spark are all lazily evaluated, right? Maybe we can say "it is lazily analyzed".
We should probably document this as a behavior change for Spark Connect. I am pretty sure there are other behavior changes. Also does this lazy analysis apply to persistent tables and views as well?
There was a problem hiding this comment.
sounds good, let me update with it is lazily analyzed.
Besides temp views, this lazy analysis apply to temp functions / configurations / persistent tables.
If the functions/configurations/tables are changed after spark.table/sql, the results may be different from Spark Classic.
There was a problem hiding this comment.
other dataframe APIs may also have the same behavior change, we probably need to document it somewhere like docs/spark-connect-overview.md
### What changes were proposed in this pull request? `lazily evaluated` -> `lazily analyzed` ### Why are the changes needed? to address #46007 (comment) Closes #46118 from zhengruifeng/doc_nit. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
What changes were proposed in this pull request?
Document the lazy evaluation of views in
spark.{sql, table}Why are the changes needed?
it is by design in Spark Connect, so we need to document it
Does this PR introduce any user-facing change?
doc change
How was this patch tested?
ci
Was this patch authored or co-authored using generative AI tooling?
no