Skip to content

Commit

Permalink
[SPARK-39258][TESTS] Fix Hide credentials in show create table
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
[SPARK-35378-FOLLOWUP](apache#36632) changes the return value of `CommandResultExec.executeCollect()` from `InternalRow` to `UnsafeRow`, this change causes the result of `r.tostring` in the following code:

https://github.com/apache/spark/blob/de73753bb2e5fd947f237e731ff05aa9f2711677/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala#L1143-L1148

change from

```
[CREATE TABLE tab1 (
  NAME STRING,
  THEID INT)
USING org.apache.spark.sql.jdbc
OPTIONS (
  'dbtable' = 'TEST.PEOPLE',
  'password' = '*********(redacted)',
  'url' = '*********(redacted)',
  'user' = 'testUser')
]
```
to

```
[0,10000000d5,5420455441455243,62617420454c4241,414e20200a282031,4e4952545320454d,45485420200a2c47,a29544e49204449,726f20474e495355,6568636170612e67,732e6b726170732e,a6362646a2e6c71,20534e4f4954504f,7462642720200a28,203d2027656c6261,45502e5453455427,200a2c27454c504f,6f77737361702720,2a27203d20276472,2a2a2a2a2a2a2a2a,6574636164657228,2720200a2c272964,27203d20276c7275,2a2a2a2a2a2a2a2a,746361646572282a,20200a2c27296465,3d20277265737527,7355747365742720,a29277265]
```

and the UT `JDBCSuite$Hide credentials in show create table` failed in master branch.

This pr is  change to use `executeCollectPublic()` instead of `executeCollect()` to fix this UT.

### Why are the changes needed?
Fix UT failed in mater branch after [SPARK-35378-FOLLOWUP](apache#36632)

### Does this PR introduce _any_ user-facing change?
NO.

### How was this patch tested?

- GitHub Action pass
- Manual test

Run `mvn clean install -DskipTests -pl sql/core -am -Dtest=none -DwildcardSuites=org.apache.spark.sql.jdbc.JDBCSuite`

**Before**

```
- Hide credentials in show create table *** FAILED ***
  "[0,10000000d5,5420455441455243,62617420454c4241,414e20200a282031,4e4952545320454d,45485420200a2c47,a29544e49204449,726f20474e495355,6568636170612e67,732e6b726170732e,a6362646a2e6c71,20534e4f4954504f,7462642720200a28,203d2027656c6261,45502e5453455427,200a2c27454c504f,6f77737361702720,2a27203d20276472,2a2a2a2a2a2a2a2a,6574636164657228,2720200a2c272964,27203d20276c7275,2a2a2a2a2a2a2a2a,746361646572282a,20200a2c27296465,3d20277265737527,7355747365742720,a29277265]" did not contain "TEST.PEOPLE" (JDBCSuite.scala:1146)

```

**After**

```
Run completed in 24 seconds, 868 milliseconds.
Total number of tests run: 93
Suites: completed 2, aborted 0
Tests: succeeded 93, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```

Closes apache#36637 from LuciferYang/SPARK-39258.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 6eb15d1)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
LuciferYang authored and dongjoon-hyun committed May 28, 2022
1 parent 002d85d commit 5c60da7
Showing 1 changed file with 2 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -1153,7 +1153,7 @@ class JDBCSuite extends QueryTest
""".stripMargin)

val show = ShowCreateTableCommand(TableIdentifier(tableName), ShowCreateTable.getoutputAttrs)
spark.sessionState.executePlan(show).executedPlan.executeCollect().foreach { r =>
spark.sessionState.executePlan(show).executedPlan.executeCollectPublic().foreach { r =>
assert(!r.toString.contains(password))
assert(r.toString.contains(dbTable))
assert(r.toString.contains(userName))
Expand All @@ -1166,7 +1166,7 @@ class JDBCSuite extends QueryTest
}

withSQLConf(SQLConf.SQL_OPTIONS_REDACTION_PATTERN.key -> "(?i)dbtable|user") {
spark.sessionState.executePlan(show).executedPlan.executeCollect().foreach { r =>
spark.sessionState.executePlan(show).executedPlan.executeCollectPublic().foreach { r =>
assert(!r.toString.contains(password))
assert(!r.toString.contains(dbTable))
assert(!r.toString.contains(userName))
Expand Down

0 comments on commit 5c60da7

Please sign in to comment.