Skip to content

Commit

Permalink
[SPARK-40799][BUILD][CONNECT][FOLLOW-UP] Keep the console output cons…
Browse files Browse the repository at this point in the history 
…istent of lint-scala script as was

### What changes were proposed in this pull request?

This PR proposes to keep `dev/lint-scala` quiet as was.

### Why are the changes needed?

To remove noisy output from the `dev/lint-scala` script.

**Before**

Success

```
Scalastyle checks passed.
Using `mvn` from path: /.../spark/build/apache-maven-3.8.6/bin/mvn
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO] ------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: x86_64
[INFO] os.detected.version: 10.16
[INFO] os.detected.version.major: 10
[INFO] os.detected.version.minor: 16
[INFO] os.detected.classifier: osx-x86_64
[INFO]
[INFO] ----------------< org.apache.spark:spark-connect_2.12 >-----------------
[INFO] Building Spark Project Connect 3.4.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- mvn-scalafmt_2.12:1.1.1640084764.9f463a9:format (default-cli)  spark-connect_2.12 ---
[INFO] parsed config (v3.5.9): dev/.scalafmt.conf
[INFO] Scalafmt results: 0 of 11 were unformatted
Details:
Formatted: Connect.scala
Formatted: DataTypeProtoConverter.scala
Formatted: SparkConnectPlanner.scala
Formatted: SparkConnectPlugin.scala
Formatted: SparkConnectCommandPlanner.scala
Formatted: SparkConnectStreamHandler.scala
Formatted: SparkConnectService.scala
Formatted: package.scala
Formatted: SparkConnectProtoSuite.scala
Formatted: SparkConnectPlannerSuite.scala
Formatted: SparkConnectCommandPlannerSuite.scala

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.257 s
[INFO] Finished at: 2022-10-21T11:18:19+09:00
[INFO] ------------------------------------------------------------------------

```

Failure

```
Scalastyle checks passed.
Using `mvn` from path: /Users/hyukjin.kwon/workspace/forked/spark/build/apache-maven-3.8.6/bin/mvn
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO] ------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: x86_64
[INFO] os.detected.version: 10.16
[INFO] os.detected.version.major: 10
[INFO] os.detected.version.minor: 16
[INFO] os.detected.classifier: osx-x86_64
[INFO]
[INFO] ----------------< org.apache.spark:spark-connect_2.12 >-----------------
[INFO] Building Spark Project Connect 3.4.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- mvn-scalafmt_2.12:1.1.1640084764.9f463a9:format (default-cli)  spark-connect_2.12 ---
[INFO] parsed config (v3.5.9): dev/.scalafmt.conf
[INFO] Scalafmt results: 0 of 11 were unformatted
Details:
Formatted: Connect.scala
Formatted: DataTypeProtoConverter.scala
Formatted: SparkConnectPlanner.scala
Formatted: SparkConnectPlugin.scala
Formatted: SparkConnectCommandPlanner.scala
Formatted: SparkConnectStreamHandler.scala
Formatted: SparkConnectService.scala
Formatted: package.scala
Formatted: SparkConnectProtoSuite.scala
Formatted: SparkConnectPlannerSuite.scala
Formatted: SparkConnectCommandPlannerSuite.scala

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.257 s
[INFO] Finished at: 2022-10-21T11:18:19+09:00
[INFO] ------------------------------------------------------------------------
(python3.9) ➜  spark git:(master) ./dev/lint-scala
Scalastyle checks passed.
Using `mvn` from path: /Users/hyukjin.kwon/workspace/forked/spark/build/apache-maven-3.8.6/bin/mvn
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO] ------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: x86_64
[INFO] os.detected.version: 10.16
[INFO] os.detected.version.major: 10
[INFO] os.detected.version.minor: 16
[INFO] os.detected.classifier: osx-x86_64
[INFO]
[INFO] ----------------< org.apache.spark:spark-connect_2.12 >-----------------
[INFO] Building Spark Project Connect 3.4.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- mvn-scalafmt_2.12:1.1.1640084764.9f463a9:format (default-cli)  spark-connect_2.12 ---
[INFO] parsed config (v3.5.9): dev/.scalafmt.conf
[ERROR] unformatted file at: /Users/hyukjin.kwon/workspace/forked/spark/connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
[INFO] Scalafmt results: 1 of 11 were unformatted
Details:
Formatted: Connect.scala
Formatted: DataTypeProtoConverter.scala
Requires formatting: SparkConnectPlanner.scala
Formatted: SparkConnectPlugin.scala
Formatted: SparkConnectCommandPlanner.scala
Formatted: SparkConnectStreamHandler.scala
Formatted: SparkConnectService.scala
Formatted: package.scala
Formatted: SparkConnectProtoSuite.scala
Formatted: SparkConnectPlannerSuite.scala
Formatted: SparkConnectCommandPlannerSuite.scala

[ERROR]
org.apache.maven.plugin.MojoExecutionException: Scalafmt: Unformatted files found
    at org.antipathy.mvn_scalafmt.FormatMojo.execute (FormatMojo.java:91)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.223 s
[INFO] Finished at: 2022-10-21T11:19:58+09:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.antipathy:mvn-scalafmt_2.12:1.1.1640084764.9f463a9:format (default-cli) on project spark-connect_2.12: Error formatting Scala files: Scalafmt: Unformatted files found -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
The scalafmt check failed on connector/connect.
Before submitting your change, please make sure to format your code using the following command:
./build/mvn -Pscala-2.12 scalafmt:format -Dscalafmt.skip=fase -Dscalafmt.validateOnly=false -Dscalafmt.changedOnly=false -pl connector/connect
```

**After**

Success

```
Scalastyle checks passed.
Scalafmt checks passed.
```

Failure

```
Scalastyle checks passed.
The scalafmt check failed on connector/connect at following occurrences:

Requires formatting: SparkConnectPlanner.scala

Before submitting your change, please make sure to format your code using the following command:
./build/mvn -Pscala-2.12 scalafmt:format -Dscalafmt.skip=fase -Dscalafmt.validateOnly=false -Dscalafmt.changedOnly=false -pl connector/connect

```

In this way, this is consistent before #38258.

### Does this PR introduce _any_ user-facing change?

No, dev-only.

### How was this patch tested?

Manually tested as described above.

Closes #38326 from HyukjinKwon/SPARK-40799-followup.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
@HyukjinKwon
HyukjinKwon committed Oct 21, 2022
1 parent 7934f00 commit 140c99c
Showing 1 changed file with 14 additions and 3 deletions.
17 changes: 14 additions & 3 deletions dev/lint-scala
View file
Expand Up @@ -23,10 +23,21 @@ SPARK_ROOT_DIR="$(dirname $SCRIPT_DIR)"
"$SCRIPT_DIR/scalastyle" "$1"

# For Spark Connect, we actively enforce scalafmt and check that the produced diff is empty.
./build/mvn -Pscala-2.12 scalafmt:format -Dscalafmt.skip=false -Dscalafmt.validateOnly=true -Dscalafmt.changedOnly=false -pl connector/connect
if [[ $? -ne 0 ]]; then
echo "The scalafmt check failed on connector/connect."
ERRORS=$(./build/mvn \
-Pscala-2.12 \
scalafmt:format \
-Dscalafmt.skip=false \
-Dscalafmt.validateOnly=true \
-Dscalafmt.changedOnly=false \
-pl connector/connect \
2>&1 | grep -e "^Requires formatting" \
)

if test ! -z "$ERRORS"; then
echo -e "The scalafmt check failed on connector/connect at following occurrences:\n\n$ERRORS\n"
echo "Before submitting your change, please make sure to format your code using the following command:"
echo "./build/mvn -Pscala-2.12 scalafmt:format -Dscalafmt.skip=fase -Dscalafmt.validateOnly=false -Dscalafmt.changedOnly=false -pl connector/connect"
exit 1
else
echo -e "Scalafmt checks passed."
fi

0 comments on commit 140c99c

Please sign in to comment.