Skip to content

Commit

Permalink
[SPARK-6696] [SQL] Adds HiveContext.refreshTable to PySpark
Browse files Browse the repository at this point in the history
<!-- Reviewable:start -->
[<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/apache/spark/5349)
<!-- Reviewable:end -->

Author: Cheng Lian <lian@databricks.com>

Closes apache#5349 from liancheng/py-refresh-table and squashes the following commits:

004bec0 [Cheng Lian] Adds HiveContext.refreshTable to PySpark
  • Loading branch information
liancheng authored and marmbrus committed Apr 9, 2015
1 parent 7d7384c commit 891ada5
Showing 1 changed file with 9 additions and 0 deletions.
9 changes: 9 additions & 0 deletions python/pyspark/sql/context.py
Original file line number Diff line number Diff line change
Expand Up @@ -574,6 +574,15 @@ def _ssql_ctx(self):
def _get_hive_ctx(self):
return self._jvm.HiveContext(self._jsc.sc())

def refreshTable(self, tableName):
"""Invalidate and refresh all the cached the metadata of the given
table. For performance reasons, Spark SQL or the external data source
library it uses might cache certain metadata about a table, such as the
location of blocks. When those change outside of Spark SQL, users should
call this function to invalidate the cache.
"""
self._ssql_ctx.refreshTable(tableName)


class UDFRegistration(object):
"""Wrapper for user-defined function registration."""
Expand Down

0 comments on commit 891ada5

Please sign in to comment.