Skip to content

Commit

Permalink
[SPARK-19137][SQL] Fix withSQLConf to reset OptionalConfigEntry c…
Browse files Browse the repository at this point in the history
…orrectly

## What changes were proposed in this pull request?

`DataStreamReaderWriterSuite` makes test files in source folder like the followings. Interestingly, the root cause is `withSQLConf` fails to reset `OptionalConfigEntry` correctly. In other words, it resets the config into `Some(undefined)`.

```bash
$ git status
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        sql/core/%253Cundefined%253E/
        sql/core/%3Cundefined%3E/
```

## How was this patch tested?

Manual.
```
build/sbt "project sql" test
git status
```

Author: Dongjoon Hyun <dongjoon@apache.org>

Closes apache#16522 from dongjoon-hyun/SPARK-19137.
  • Loading branch information
dongjoon-hyun authored and zsxwing committed Jan 10, 2017
1 parent 3ef183a commit d5b1dc9
Showing 1 changed file with 7 additions and 1 deletion.
Expand Up @@ -94,7 +94,13 @@ private[sql] trait SQLTestUtils
*/
protected def withSQLConf(pairs: (String, String)*)(f: => Unit): Unit = {
val (keys, values) = pairs.unzip
val currentValues = keys.map(key => Try(spark.conf.get(key)).toOption)
val currentValues = keys.map { key =>
if (spark.conf.contains(key)) {
Some(spark.conf.get(key))
} else {
None
}
}
(keys, values).zipped.foreach(spark.conf.set)
try f finally {
keys.zip(currentValues).foreach {
Expand Down

0 comments on commit d5b1dc9

Please sign in to comment.