Skip to content

Commit

Permalink
[SPARK-32489][CORE] Pass core module UTs in Scala 2.13
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

So far, we fixed many stuffs in `core` module. This PR fixes the remaining UT failures in Scala 2.13.

- `OneApplicationResource.environmentInfo` will return a deterministic result for `sparkProperties`, `hadoopProperties`, `systemProperties`, and `classpathEntries`.
- `SubmitRestProtocolSuite` has Scala 2.13 answer in addition to the existing Scala 2.12 answer, and uses the expected answer based on the Scala runtime version.

### Why are the changes needed?

To support Scala 2.13.

### Does this PR introduce _any_ user-facing change?

Yes, `environmentInfo` is changed, but this fixes the indeterministic behavior.

### How was this patch tested?

- Scala 2.12: Pass the Jenkins or GitHub Action
- Scala 2.13: Do the following.
```
$ dev/change-scala-version.sh 2.13
$ build/mvn test -pl core --am -Pscala-2.13
```

**BEFORE**
```
Tests: succeeded 2612, failed 3, canceled 1, ignored 8, pending 0
*** 3 TESTS FAILED ***
```

**AFTER**
```
Tests: succeeded 2615, failed 0, canceled 1, ignored 8, pending 0
All tests passed.
```

Closes #29298 from dongjoon-hyun/SPARK-32489.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
dongjoon-hyun committed Jul 30, 2020
1 parent 32f4ef0 commit 7cf3b54
Show file tree
Hide file tree
Showing 3 changed files with 262 additions and 233 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -104,10 +104,10 @@ private[v1] class AbstractApplicationResource extends BaseAppResource {
val resourceProfileInfo = ui.store.resourceProfileInfo()
new v1.ApplicationEnvironmentInfo(
envInfo.runtime,
Utils.redact(ui.conf, envInfo.sparkProperties),
Utils.redact(ui.conf, envInfo.hadoopProperties),
Utils.redact(ui.conf, envInfo.systemProperties),
envInfo.classpathEntries,
Utils.redact(ui.conf, envInfo.sparkProperties).sortBy(_._1),
Utils.redact(ui.conf, envInfo.hadoopProperties).sortBy(_._1),
Utils.redact(ui.conf, envInfo.systemProperties).sortBy(_._1),
envInfo.classpathEntries.sortBy(_._1),
resourceProfileInfo)
}

Expand Down
Loading

0 comments on commit 7cf3b54

Please sign in to comment.