Skip to content

[SPARK-45938][INFRA] Add utils to the dependencies of the core/unsafe/network_common module in module.py#43818

Closed
LuciferYang wants to merge 4 commits into
apache:masterfrom
LuciferYang:common-utils-test
Closed

[SPARK-45938][INFRA] Add utils to the dependencies of the core/unsafe/network_common module in module.py#43818
LuciferYang wants to merge 4 commits into
apache:masterfrom
LuciferYang:common-utils-test

Conversation

@LuciferYang
Copy link
Copy Markdown
Contributor

@LuciferYang LuciferYang commented Nov 15, 2023

What changes were proposed in this pull request?

This pr add utils to the dependencies of the core/unsafe/network_common module in module.py due to utils is direct dependency of core/unsafe/network_common:

spark/core/pom.xml

Lines 102 to 106 in 7120e6b

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-common-utils_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-common-utils_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-common-utils_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>

Why are the changes needed?

All UTs of utils module are still in the core module now. This pr is used to avoid the issue of not running unit tests in the core module when changing the utils module.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions

Was this patch authored or co-authored using generative AI tooling?

No

@LuciferYang LuciferYang changed the title [SPARK-45938][INFRA] Add utils to the dependency list of the core module in module.py [SPARK-45938][INFRA] Add utils to the dependencies of the core module in module.py Nov 15, 2023
@github-actions github-actions Bot added the BUILD label Nov 15, 2023
core = Module(
name="core",
dependencies=[kvstore, network_common, network_shuffle, unsafe, launcher],
dependencies=[kvstore, network_common, network_shuffle, unsafe, launcher, utils],
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @zhengruifeng @HyukjinKwon , utils module is also a direct dependency of unsafe and network-common. Do we need to modify the dependencies of unsafe and network-common? Or is the current change sufficient?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utils module is also a direct dependency of unsafe and network-common

let's also add this dependency

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

],
)

utils = Module(
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving it is because of

image

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, this is python :)

@LuciferYang LuciferYang changed the title [SPARK-45938][INFRA] Add utils to the dependencies of the core module in module.py [SPARK-45938][INFRA] Add utils to the dependencies of the core/unsafe/network_common module in module.py Nov 16, 2023
@LuciferYang
Copy link
Copy Markdown
Contributor Author

Thanks @zhengruifeng

@LuciferYang LuciferYang deleted the common-utils-test branch March 30, 2026 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants