Skip to content

[SPARK-48540][CORE] Avoid ivy output loading settings to stdout#46882

Closed
cxzl25 wants to merge 1 commit intoapache:masterfrom
cxzl25:SPARK-48540
Closed

[SPARK-48540][CORE] Avoid ivy output loading settings to stdout#46882
cxzl25 wants to merge 1 commit intoapache:masterfrom
cxzl25:SPARK-48540

Conversation

@cxzl25
Copy link
Contributor

@cxzl25 cxzl25 commented Jun 5, 2024

What changes were proposed in this pull request?

This PR aims to avoid ivy output loading settings to stdout.

Why are the changes needed?

Now org.apache.spark.util.MavenUtils#getModuleDescriptor will output the following string to stdout.

This is due to the modified code order in SPARK-32596 .

:: loading settings :: url = jar:file:xxxx/jars/ivy-2.5.0.jar!/org/apache/ivy/core/settings/ivysettings.xml

Stack trace

	at org.apache.ivy.core.settings.IvySettings.load(IvySettings.java:404)
	at org.apache.ivy.core.settings.IvySettings.loadDefault(IvySettings.java:443)
	at org.apache.ivy.Ivy.configureDefault(Ivy.java:435)
	at org.apache.ivy.core.IvyContext.getDefaultIvy(IvyContext.java:201)
	at org.apache.ivy.core.IvyContext.getIvy(IvyContext.java:180)
	at org.apache.ivy.core.IvyContext.getSettings(IvyContext.java:216)
	at org.apache.ivy.core.module.status.StatusManager.getCurrent(StatusManager.java:40)
	at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.<init>(DefaultModuleDescriptor.java:206)
	at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.newDefaultInstance(DefaultModuleDescriptor.java:107)
	at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.newDefaultInstance(DefaultModuleDescriptor.java:66)
	at org.apache.spark.deploy.SparkSubmitUtils$.getModuleDescriptor(SparkSubmit.scala:1413)
	at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1460)
	at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:185)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:327)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:942)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:181)

Does this PR introduce any user-facing change?

No

How was this patch tested?

local test

Was this patch authored or co-authored using generative AI tooling?

No

@cxzl25
Copy link
Contributor Author

cxzl25 commented Jun 6, 2024

@LuciferYang @yaooqinn Please help review, thanks in advance!

@yaooqinn yaooqinn closed this in f4434c3 Jun 6, 2024
@yaooqinn
Copy link
Member

yaooqinn commented Jun 6, 2024

Merged to master. Thank you @cxzl25

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants