Skip to content

Commit

Permalink
[SPARK-32897][PYTHON] Don't show a deprecation warning at SparkSessio…
Browse files Browse the repository at this point in the history
…n.builder.getOrCreate

### What changes were proposed in this pull request?

In PySpark shell, if you call `SparkSession.builder.getOrCreate` as below:

```python
import warnings
from pyspark.sql import SparkSession, SQLContext
warnings.simplefilter('always', DeprecationWarning)
spark.stop()
SparkSession.builder.getOrCreate()
```

it shows the deprecation warning as below:

```
/.../spark/python/pyspark/sql/context.py:72: DeprecationWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
  DeprecationWarning)
```

via https://github.com/apache/spark/blob/d3304268d3046116d39ec3d54a8e319dce188f36/python/pyspark/sql/session.py#L222

We shouldn't print the deprecation warning from it. This is the only place ^.

### Why are the changes needed?

To prevent to inform users that `SparkSession.builder.getOrCreate` is deprecated mistakenly.

### Does this PR introduce _any_ user-facing change?

Yes, it won't show a deprecation warning to end users for calling `SparkSession.builder.getOrCreate`.

### How was this patch tested?

Manually tested as above.

Closes apache#29768 from HyukjinKwon/SPARK-32897.

Authored-by: HyukjinKwon <gurwls223@apache.org>
Signed-off-by: Takuya UESHIN <ueshin@databricks.com>
  • Loading branch information
HyukjinKwon authored and ueshin committed Sep 16, 2020
1 parent 40ef5c9 commit 657e39a
Showing 1 changed file with 4 additions and 3 deletions.
7 changes: 4 additions & 3 deletions python/pyspark/sql/context.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,10 @@ def __init__(self, sparkContext, sparkSession=None, jsqlContext=None):
>>> df.rdd.map(lambda x: (x.i, x.s, x.d, x.l, x.b, x.time, x.row.a, x.list)).collect()
[(1, 'string', 1.0, 1, True, datetime.datetime(2014, 8, 1, 14, 1, 5), 1, [1, 2, 3])]
"""
warnings.warn(
"Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
DeprecationWarning)
if sparkSession is None:
warnings.warn(
"Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.",
DeprecationWarning)

self._sc = sparkContext
self._jsc = self._sc._jsc
Expand Down

0 comments on commit 657e39a

Please sign in to comment.