[SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable #18328
Conversation
| query: Dataset[_], | ||
| tableName: Option[String] = None | ||
| ): Unit = writeLock { | ||
| cacheQuery(query, tableName, StorageLevel.fromString(query.sparkSession.sessionState.conf.cacheStorageLevel)) |
There was a problem hiding this comment.
This line exceeds 100 characters.
| def cacheQuery( | ||
| query: Dataset[_], | ||
| tableName: Option[String] = None | ||
| ): Unit = writeLock { |
There was a problem hiding this comment.
Combine the two lines together
|
Could you add a test case in |
|
ok to test |
|
Test build #78187 has finished for PR 18328 at commit
|
…nMemoryColumnarStorage.level variable
|
Test build #78201 has started for PR 18328 at commit |
|
Test FAILed. |
| @@ -106,6 +105,11 @@ class CacheManager extends Logging { | |||
| } | |||
| } | |||
|
|
|||
| def cacheQuery(query: Dataset[_], tableName: Option[String] = None): Unit = writeLock { | |||
| cacheQuery(query, tableName, StorageLevel.fromString( | |||
| query.sparkSession.sessionState.conf.cacheStorageLevel)) | |||
There was a problem hiding this comment.
btw, this is a bit odd to roundtrip to string <-> storagelevel...
There was a problem hiding this comment.
Could you explain your question in a bit more details?
|
@dosoft Is this PR active? then would you mind if I ask to reply to the review comment above? |
What changes were proposed in this pull request?
As described in title
How was this patch tested?