Skip to content

[SPARK-28498][SQL][TEST] clear the states of SparkSession after each test #25243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cloud-fan wants to merge 10 commits into from
Closed

[SPARK-28498][SQL][TEST] clear the states of SparkSession after each test #25243

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
8665961
always create a fresh copy of the SparkSession before each test
cloud-fan Jul 24, 2019
d9026d2
fix test
cloud-fan Jul 24, 2019
8a90b9d
address comments
cloud-fan Jul 25, 2019
e5f5121
only reset session state
cloud-fan Jul 26, 2019
fc8a731
Merge remote-tracking branch 'origin/master' into test
cloud-fan Jul 26, 2019
ec7b7bd
fix
cloud-fan Jul 26, 2019
c8e9c65
fix
cloud-fan Jul 26, 2019
940d336
fix
cloud-fan Jul 26, 2019
d30c1d3
do not clear temp views
cloud-fan Jul 30, 2019
5ac310b
fix
cloud-fan Aug 6, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,8 @@ abstract class AvroSuite extends QueryTest with SharedSQLContext with SQLTestUti
val episodesAvro = testFile("episodes.avro")
val testAvro = testFile("test.avro")

override protected def beforeAll(): Unit = {
super.beforeAll()
spark.conf.set(SQLConf.FILES_MAX_PARTITION_BYTES.key, 1024)
override protected def sparkConf: SparkConf = {
super.sparkConf.set(SQLConf.FILES_MAX_PARTITION_BYTES, 1024L)
}

def checkReloadMatchesSaved(originalFile: String, newFile: String): Unit = {
Expand Down
4 changes: 2 additions & 2 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -852,7 +852,7 @@ class SessionCatalog(
* Drop all existing temporary views.
* For testing only.
*/
def clearTempTables(): Unit = synchronized {
def clearTempViews(): Unit = synchronized {
tempViews.clear()
}

Expand Down Expand Up @@ -1428,7 +1428,7 @@ class SessionCatalog(
dropTempFunction(func.funcName, ignoreIfNotExists = false)
}
}
clearTempTables()
clearTempViews()
globalTempViewManager.clear()
functionRegistry.clear()
tableRelationCache.invalidateAll()
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -607,7 +607,7 @@ class SparkSession private(
*/
@transient lazy val catalog: Catalog = new CatalogImpl(self)

@transient private lazy val catalogs = new mutable.HashMap[String, CatalogPlugin]()
@transient private[sql] lazy val catalogs = new mutable.HashMap[String, CatalogPlugin]()

private[sql] def catalog(name: String): CatalogPlugin = synchronized {
catalogs.getOrElseUpdate(name, Catalogs.load(name, sessionState.conf))
Expand Down
2 changes: 1 addition & 1 deletion ...core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceResolution.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ case class DataSourceResolution(
import org.apache.spark.sql.catalog.v2.CatalogV2Implicits._
import lookup._

lazy val v2SessionCatalog: CatalogPlugin = lookup.sessionCatalog
def v2SessionCatalog: CatalogPlugin = lookup.sessionCatalog
.getOrElse(throw new AnalysisException("No v2 session catalog implementation is available"))

override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
Expand Down
Loading