Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-47021][BUILD][TESTS] Fix kvstore module to have explicit commons-lang3 test dependency #45080

Closed
wants to merge 1 commit into from

Conversation

dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Feb 10, 2024

What changes were proposed in this pull request?

This PR aims to fix kvstore module by adding explicit commons-lang3 test dependency and excluding htmlunit-driver from org.scalatestplus to use Apache Spark's explicit declaration.

spark/pom.xml

Lines 711 to 716 in fa23d27

<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>htmlunit-driver</artifactId>
<version>${htmlunit-driver.version}</version>
<scope>test</scope>
</dependency>

Why are the changes needed?

Since Spark 3.3.0 (SPARK-37282), kvstore uses commons-lang3 test dependency like the following, but we didn't declare it explicitly so far.

Previously, it was provided by some unused htmlunit-driver's transitive dependency accidentally. This causes a weird situation which kvstore module starts to fail to compile when we upgrade htmlunit-driver. We need to fix this first.

$ mvn dependency:tree -pl common/kvstore
...
[INFO] |  \- org.seleniumhq.selenium:htmlunit-driver:jar:4.12.0:test
...
[INFO] |        +- org.apache.commons:commons-lang3:jar:3.14.0:test

Does this PR introduce any user-facing change?

No. This is only a test dependency fix.

How was this patch tested?

Pass the CIs.

Was this patch authored or co-authored using generative AI tooling?

No.

@dongjoon-hyun
Copy link
Member Author

Thank you, @MaxGekk !

dongjoon-hyun added a commit that referenced this pull request Feb 11, 2024
…mmons-lang3` test dependency

### What changes were proposed in this pull request?

This PR aims to fix `kvstore` module by adding explicit `commons-lang3` test dependency and excluding `htmlunit-driver` from `org.scalatestplus` to use Apache Spark's explicit declaration.
https://github.com/apache/spark/blob/fa23d276e7e4ed94bf11d71f2e1daa22fe2238e5/pom.xml#L711-L716

### Why are the changes needed?

Since Spark 3.3.0 (SPARK-37282), `kvstore` uses `commons-lang3` test dependency like the following, but we didn't declare it explicitly so far.

https://github.com/apache/spark/blob/fa23d276e7e4ed94bf11d71f2e1daa22fe2238e5/common/kvstore/src/test/java/org/apache/spark/util/kvstore/LevelDBSuite.java#L33

https://github.com/apache/spark/blob/fa23d276e7e4ed94bf11d71f2e1daa22fe2238e5/common/kvstore/src/test/java/org/apache/spark/util/kvstore/LevelDBIteratorSuite.java#L23

Previously, it was provided by some unused `htmlunit-driver`'s transitive dependency accidentally. This causes a weird situation which `kvstore` module starts to fail to compile when we upgrade `htmlunit-driver`. We need to fix this first.

```
$ mvn dependency:tree -pl common/kvstore
...
[INFO] |  \- org.seleniumhq.selenium:htmlunit-driver:jar:4.12.0:test
...
[INFO] |        +- org.apache.commons:commons-lang3:jar:3.14.0:test
```

### Does this PR introduce _any_ user-facing change?

No. This is only a test dependency fix.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #45080 from dongjoon-hyun/SPARK-47021.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(cherry picked from commit a926c79)
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
dongjoon-hyun added a commit that referenced this pull request Feb 11, 2024
…mmons-lang3` test dependency

### What changes were proposed in this pull request?

This PR aims to fix `kvstore` module by adding explicit `commons-lang3` test dependency and excluding `htmlunit-driver` from `org.scalatestplus` to use Apache Spark's explicit declaration.
https://github.com/apache/spark/blob/fa23d276e7e4ed94bf11d71f2e1daa22fe2238e5/pom.xml#L711-L716

### Why are the changes needed?

Since Spark 3.3.0 (SPARK-37282), `kvstore` uses `commons-lang3` test dependency like the following, but we didn't declare it explicitly so far.

https://github.com/apache/spark/blob/fa23d276e7e4ed94bf11d71f2e1daa22fe2238e5/common/kvstore/src/test/java/org/apache/spark/util/kvstore/LevelDBSuite.java#L33

https://github.com/apache/spark/blob/fa23d276e7e4ed94bf11d71f2e1daa22fe2238e5/common/kvstore/src/test/java/org/apache/spark/util/kvstore/LevelDBIteratorSuite.java#L23

Previously, it was provided by some unused `htmlunit-driver`'s transitive dependency accidentally. This causes a weird situation which `kvstore` module starts to fail to compile when we upgrade `htmlunit-driver`. We need to fix this first.

```
$ mvn dependency:tree -pl common/kvstore
...
[INFO] |  \- org.seleniumhq.selenium:htmlunit-driver:jar:4.12.0:test
...
[INFO] |        +- org.apache.commons:commons-lang3:jar:3.14.0:test
```

### Does this PR introduce _any_ user-facing change?

No. This is only a test dependency fix.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #45080 from dongjoon-hyun/SPARK-47021.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(cherry picked from commit a926c79)
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
@dongjoon-hyun
Copy link
Member Author

Merged to master/3.5/3.4.

@dongjoon-hyun dongjoon-hyun deleted the SPARK-47021 branch February 11, 2024 18:44
snmvaughan pushed a commit to snmvaughan/spark that referenced this pull request Mar 26, 2024
…mmons-lang3` test dependency

### What changes were proposed in this pull request?

This PR aims to fix `kvstore` module by adding explicit `commons-lang3` test dependency and excluding `htmlunit-driver` from `org.scalatestplus` to use Apache Spark's explicit declaration.
https://github.com/apache/spark/blob/fa23d276e7e4ed94bf11d71f2e1daa22fe2238e5/pom.xml#L711-L716

### Why are the changes needed?

Since Spark 3.3.0 (SPARK-37282), `kvstore` uses `commons-lang3` test dependency like the following, but we didn't declare it explicitly so far.

https://github.com/apache/spark/blob/fa23d276e7e4ed94bf11d71f2e1daa22fe2238e5/common/kvstore/src/test/java/org/apache/spark/util/kvstore/LevelDBSuite.java#L33

https://github.com/apache/spark/blob/fa23d276e7e4ed94bf11d71f2e1daa22fe2238e5/common/kvstore/src/test/java/org/apache/spark/util/kvstore/LevelDBIteratorSuite.java#L23

Previously, it was provided by some unused `htmlunit-driver`'s transitive dependency accidentally. This causes a weird situation which `kvstore` module starts to fail to compile when we upgrade `htmlunit-driver`. We need to fix this first.

```
$ mvn dependency:tree -pl common/kvstore
...
[INFO] |  \- org.seleniumhq.selenium:htmlunit-driver:jar:4.12.0:test
...
[INFO] |        +- org.apache.commons:commons-lang3:jar:3.14.0:test
```

### Does this PR introduce _any_ user-facing change?

No. This is only a test dependency fix.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#45080 from dongjoon-hyun/SPARK-47021.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants