[SPARK-39886][SQL] Disable DEFAULT column SQLConf until implementation is complete #37310
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…stead of Utils.localCanonicalHostName in tests ### What changes were proposed in this pull request? This PR aims to use `Utils.localHostNameForURI` instead of `Utils.localCanonicalHostName` in the following suites which changed in #36866 - `MasterSuite` - `MasterWebUISuite` - `RocksDBBackendHistoryServerSuite` ### Why are the changes needed? These test cases fails when we run with `SPARK_LOCAL_IP=::1` and `-Djava.net.preferIPv6Addresses=true` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GA - Manual test: 1. `export SPARK_LOCAL_IP=::1` ``` echo $SPARK_LOCAL_IP ::1 ``` 2. add `-Djava.net.preferIPv6Addresses=true` to MAVEN_OPTS, for example: ``` diff --git a/pom.xml b/pom.xml index 1ce3b43faf..3356622985 100644 --- a/pom.xml +++ b/pom.xml -2943,7 +2943,7 <include>**/*Suite.java</include> </includes> <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory> - <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine> + <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true -Djava.net.preferIPv6Addresses=true</argLine> <environmentVariables> <!-- Setting SPARK_DIST_CLASSPATH is a simple way to make sure any child processes ``` 3. maven test `RocksDBBackendHistoryServerSuite`, `MasterSuite` and `MasterWebUISuite` ``` mvn clean install -DskipTests -pl core -am mvn clean test -pl core -Dtest=none -DwildcardSuites=org.apache.spark.deploy.history.RocksDBBackendHistoryServerSuite mvn clean test -pl core -Dtest=none -DwildcardSuites=org.apache.spark.deploy.master.MasterSuite mvn clean test -pl core -Dtest=none -DwildcardSuites=org.apache.spark.deploy.master.ui.MasterWebUISuite ``` **Before** RocksDBBackendHistoryServerSuite: ``` - Redirect to the root page when accessed to /history/ *** FAILED *** java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:613) at java.net.Socket.connect(Socket.java:561) at sun.net.NetworkClient.doConnect(NetworkClient.java:180) at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ... Run completed in 31 seconds, 745 milliseconds. Total number of tests run: 73 Suites: completed 2, aborted 0 Tests: succeeded 3, failed 70, canceled 0, ignored 0, pending 0 *** 70 TESTS FAILED *** ``` MasterSuite: ``` - master/worker web ui available behind front-end reverseProxy *** FAILED *** The code passed to eventually never returned normally. Attempted 487 times over 50.079685917 seconds. Last failure message: Connection refused (Connection refused). (MasterSuite.scala:405) Run completed in 3 minutes, 48 seconds. Total number of tests run: 32 Suites: completed 2, aborted 0 Tests: succeeded 29, failed 3, canceled 0, ignored 0, pending 0 *** 3 TESTS FAILED *** ``` MasterWebUISuite: ``` - Kill multiple hosts *** FAILED *** java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:613) at java.net.Socket.connect(Socket.java:561) at sun.net.NetworkClient.doConnect(NetworkClient.java:180) at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ... Run completed in 7 seconds, 83 milliseconds. Total number of tests run: 4 Suites: completed 2, aborted 0 Tests: succeeded 0, failed 4, canceled 0, ignored 0, pending 0 *** 4 TESTS FAILED *** ``` **After** RocksDBBackendHistoryServerSuite: ``` Run completed in 38 seconds, 205 milliseconds. Total number of tests run: 73 Suites: completed 2, aborted 0 Tests: succeeded 73, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` MasterSuite: ``` Run completed in 1 minute, 10 seconds. Total number of tests run: 32 Suites: completed 2, aborted 0 Tests: succeeded 32, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` MasterWebUISuite: ``` Run completed in 6 seconds, 330 milliseconds. Total number of tests run: 4 Suites: completed 2, aborted 0 Tests: succeeded 4, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` Closes #36876 from LuciferYang/SPARK-39464-FOLLOWUP. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request?
This PR aims to fix `KafkaTestUtils` to support IPv6.
### Why are the changes needed?
Currently, the test suite is using a hard-coded `127.0.0.1` like the following.
```
props.put("listeners", "SASL_PLAINTEXT://127.0.0.1:0")
props.put("advertised.listeners", "SASL_PLAINTEXT://127.0.0.1:0")
```
### Does this PR introduce _any_ user-facing change?
No. This is a test-only change.
### How was this patch tested?
Pass the CIs.
Closes #36923 from dongjoon-hyun/SPARK-39530.
Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Yuming Wang <yumwang@ebay.com>
… column pruning
### What changes were proposed in this pull request?
This PR removes the window if windowExpressions is empty in column pruning.
### Why are the changes needed?
It will throw exception if user `set spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.RemoveNoopOperators` before running this query:
```sql
SELECT id
FROM (SELECT id,
Row_number()
OVER (
partition BY id
ORDER BY id) AS rn
FROM Range(3)) t
```
Exception:
```
org.apache.spark.sql.AnalysisException: Window expression is empty in Window [id#11L], [id#11L ASC NULLS FIRST]
+- Project [id#11L]
+- Range (0, 5, step=1, splits=None)
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test.
Closes #36842 from wangyum/SPARK-39445.
Authored-by: Yuming Wang <yumwang@ebay.com>
Signed-off-by: Yuming Wang <yumwang@ebay.com>
…column ### What changes were proposed in this pull request? Sometimes users forgot to add single quotes on the map key string literal, for example `map_col[a]`. In such a case, the Analyzer will throw an exception: ``` [MISSING_COLUMN] Column 'struct.a' does not exist. Did you mean one of the following? ... ``` We can improve this message by saying that the user should append single quotes if the map key is a string literal. ``` [UNRESOLVED_MAP_KEY] Cannot resolve column 'a' as a map key. If the key is a string literal, please add single quotes around it. Otherwise, did you mean one of the following column(s)? ... ``` ### Why are the changes needed? Error message improvement ### Does this PR introduce _any_ user-facing change? Yes but trivial, an improvement on the error message of unresolved map key column ### How was this patch tested? New UT Closes #36896 from gengliangwang/unreslovedMapKey. Authored-by: Gengliang Wang <gengliang@apache.org> Signed-off-by: Gengliang Wang <gengliang@apache.org>
…2.13 ### What changes were proposed in this pull request? `Left ExpressionSet-- Right ExpressionSet` in Scala 2.13 will use default `--` of `scala.collection.Set` as follows: ``` def -- (that: IterableOnce[A]): C = fromSpecific(coll.toSet.removedAll(that)) ``` The `Left ExpressionSet` will convert to a `immutable.Set` and the `ExpressionSet` cannot override the `removedAll` method of `immutable.Set` due to it's a `scala.collection.Set`. So this pr override `--` method for `ExpressionSet` in Scala 2.13 to to take over the action to ensure it return the same result as Scala 2.12. ### Why are the changes needed? Bug fix for Scala 2.13 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - Pass GA - Manual test ``` dev/change-scala-version.sh 2.13 mvn clean install -DskipTests -pl sql/catalyst -am -Pscala-2.13 mvn test -pl -pl sql/catalyst -Pscala-2.13 -Dtest=none -DwildcardSuites=org.apache.spark.sql.catalyst.expressions.ExpressionSetSuite ``` **Before** ``` - remove multiple elements to set with non-deterministic expressions *** FAILED *** Set() had size 0 instead of expected size 1 (ExpressionSetSuite.scala:239) Run completed in 4 seconds, 537 milliseconds. Total number of tests run: 44 Suites: completed 2, aborted 0 Tests: succeeded 43, failed 1, canceled 0, ignored 0, pending 0 *** 1 TEST FAILED *** ``` **After** ``` Run completed in 4 seconds, 622 milliseconds. Total number of tests run: 44 Suites: completed 2, aborted 0 Tests: succeeded 44, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` Closes #36925 from LuciferYang/SPARK-39520. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Sean Owen <srowen@gmail.com>
… Type and Spark DecimalType ### What changes were proposed in this pull request? - Implemented getCatalystType method in TeradataDialect - Handle Types.NUMERIC explicitly ### Why are the changes needed? Load table from Teradata, if the type of column in Teradata is `Number`, it will be converted to `DecimalType(38,0)` which will lose the fractional part of original data. ### Does this PR introduce _any_ user-facing change? Yes, it will convert Number type to DecimalType(38,18) if the scale is 0, so that keep the fractional part in some way. ### How was this patch tested? UT is added to JDBCSuite.scala. Closes #36499 from Eugene-Mark/teradata-loading. Lead-authored-by: Eugene-Mark <eugene.ma.twenty@gmail.com> Co-authored-by: Eugene <eugene.ma@intel.com> Co-authored-by: Eugene <eugene.ma.twenty@gmail.com> Signed-off-by: Sean Owen <srowen@gmail.com>
### What changes were proposed in this pull request? This PR proposes to separate workflows for each scheduled jobs. After this PR, each scheduled build can be easily searched at https://github.com/apache/spark/actions. For example, as below:  In addition, this PR switches ANSI build to a scheduled build too because it's too expensive to run it for each commit. ### Why are the changes needed? Currently it is very inconvenient to navigate scheduled jobs now. We should use https://github.com/apache/spark/actions/workflows/build_and_test.yml?query=event%3Aschedule link and manually search one by one. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Tested in my fork (https://github.com/HyukjinKwon/spark/actions). Closes #36922 from HyukjinKwon/SPARK-39521. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request? compute `Series.argmax ` with one pass ### Why are the changes needed? existing implemation of `Series.argmax` needs two pass on the dataset, the first one is to compute the maximum value, and the second one is to get the index. However, they can be computed on one pass. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? existing UT Closes #36927 from zhengruifeng/ps_series_argmax_opt. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
…n build, and readme link ### What changes were proposed in this pull request? This PR fixes the notify_test_workflow.yml to detect build_main.yml that is for PR builds. In addition, this PR fixes the link of build status in README.md. ### Why are the changes needed? To make the build fixed. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? N/A. Should better be merged fixed and tested since it's already broken. Closes #36932 from HyukjinKwon/SPARK-39521-followup. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
…ges" to detect the workflow ### What changes were proposed in this pull request? This PR changes the step name that's renamed in previous PR. ### Why are the changes needed? To recover the build. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Tested at https://github.com/HyukjinKwon/spark/pull/52/checks?check_run_id=6977082541. Closes #36933 from HyukjinKwon/SPARK-39521-followup2. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
…ompatible with 3 layer namespace ### What changes were proposed in this pull request? Make GetTable, TableExists and DatabaseExists be compatible with 3 layer namespace ### Why are the changes needed? This is a part of effort to make catalog API be compatible with 3 layer namespace ### Does this PR introduce _any_ user-facing change? Yes. The API change here is backward compatible and it extends the API to further support 3 layer namespace (e.g. catalog.database.table). ### How was this patch tested? UT Closes #36641 from amaliujia/catalogapi2. Authored-by: Rui Wang <rui.wang@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
… of IO related errors
### What changes were proposed in this pull request?
Deregistering disk persisted local blocks from the block manager in case of IO related errors.
### Why are the changes needed?
In case of a disk corruption a disk persisted block will lead to job failure as the block registration is always leads to the same file. So even when the task is rescheduled on a different executor the job will fail.
Example:
First failure (the block is locally available):
```
22/04/25 07:15:28 ERROR executor.Executor: Exception in task 17024.0 in stage 12.0 (TID 51853)
java.io.StreamCorruptedException: invalid stream header: 00000000
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:943)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:401)
at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:63)
at org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:63)
at org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:122)
at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:617)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:897)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
```
Then the task might be rescheduled on a different executor but as the block is registered to the first block manager the error will be the same:
```
java.io.StreamCorruptedException: invalid stream header: 00000000
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:943)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:401)
at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:63)
at org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:63)
at org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:122)
at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
at org.apache.spark.storage.BlockManager$$anonfun$getRemoteValues$1.apply(BlockManager.scala:698)
at org.apache.spark.storage.BlockManager$$anonfun$getRemoteValues$1.apply(BlockManager.scala:696)
at scala.Option.map(Option.scala:146)
at org.apache.spark.storage.BlockManager.getRemoteValues(BlockManager.scala:696)
at org.apache.spark.storage.BlockManager.get(BlockManager.scala:831)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
```
My idea deregistering the block when the IO operation occurs and let the following task to recompute it.
This PR only targets only local blocks. In a follow up PR `getRemoteValues` can be extended with the block removing.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
1) An existing unit test was extended.
2) Manually.
#### Manual testing
Start Spark:
```
$ ./bin/spark-shell --master "local-cluster[3,1,1200]" --conf spark.serializer=org.apache.spark.serializer.JavaSerializer
```
Create a persisted RDD (here via a DF):
```
scala> val df = sc.parallelize(1 to 20, 4).toDF
...
scala> df.persist(org.apache.spark.storage.StorageLevel.DISK_ONLY)
...
scala> df.show()
+-----+
|value|
+-----+
| 1|
| 2|
| 3|
| 4|
| 5|
| 6|
| 7|
| 8|
| 9|
| 10|
| 11|
| 12|
| 13|
| 14|
| 15|
| 16|
| 17|
| 18|
| 19|
| 20|
+-----+
```
Now as the blocks are persisted let's corrupt one of the file. For this we have to find the the directory where the blocks stored:
```
$ grep "DiskBlockManager: Created local directory" work/app-20220511112820-0000/*/stdout
work/app-20220511112820-0000/0/stdout:22/05/11 11:28:21 INFO DiskBlockManager: Created local directory at /private/var/folders/s4/qrgp74ds36l9t56tx_f6w54h0000gn/T/spark-df93a007-256e-4c16-ad23-ff7eae76c850/executor-0f4c0d32-8f12-447f-add3-5cfbd4a7c777/blockmgr-dde20b67-a824-4d92-9023-8fa902588a26
work/app-20220511112820-0000/1/stdout:22/05/11 11:28:21 INFO DiskBlockManager: Created local directory at /private/var/folders/s4/qrgp74ds36l9t56tx_f6w54h0000gn/T/spark-df93a007-256e-4c16-ad23-ff7eae76c850/executor-05de3de7-60ca-4954-8baa-965da3c35ce5/blockmgr-71c559a6-f0e8-42a1-bf53-3bddb4a69618
work/app-20220511112820-0000/2/stdout:22/05/11 11:28:21 INFO DiskBlockManager: Created local directory at /private/var/folders/s4/qrgp74ds36l9t56tx_f6w54h0000gn/T/spark-df93a007-256e-4c16-ad23-ff7eae76c850/executor-bf0ca6de-17c6-437f-82e9-33afb01dce7d/blockmgr-1f66ae2a-0a3a-4a99-8f75-b65b5d4e5e0c
```
Let's write something into one of the rdd file:
```
vim /private/var/folders/s4/qrgp74ds36l9t56tx_f6w54h0000gn/T/spark-df93a007-256e-4c16-ad23-ff7eae76c850/executor-bf0ca6de-17c6-437f-82e9-33afb01dce7d/blockmgr-1f66ae2a-0a3a-4a99-8f75-b65b5d4e5e0c/19/rdd_4_1
```
Use the DF/RDD one more time:
```
scala> df.show()
22/05/11 11:30:41 WARN TaskSetManager: Lost task 0.0 in stage 3.0 (TID 7) (192.168.1.65 executor 2): java.io.StreamCorruptedException: invalid stream header: 41ACED00
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:938)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:396)
at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:66)
at org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:66)
at org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:137)
at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:212)
at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:967)
at org.apache.spark.storage.BlockManager.get(BlockManager.scala:1277)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1344)
...
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
+-----+
|value|
+-----+
| 1|
| 2|
| 3|
| 4|
| 5|
| 6|
| 7|
| 8|
| 9|
| 10|
| 11|
| 12|
| 13|
| 14|
| 15|
| 16|
| 17|
| 18|
| 19|
| 20|
+-----+
```
Check the logs:
```
$ cat work/app-20220511112820-0000/2/stdout
...
22/05/11 11:30:41 INFO CoarseGrainedExecutorBackend: Got assigned task 7
22/05/11 11:30:41 INFO Executor: Running task 0.0 in stage 3.0 (TID 7)
...
22/05/11 11:30:41 INFO BlockManager: invalid stream header: 41ACED00. BlockManagerId(2, 192.168.1.65, 63368, None) - blockId: rdd_4_1 retryCount: 0 - blockDiskPath: /private/var/folders/s4/qrgp74ds36l9t56tx_f6w54h0000gn/T/spark-df93a007-256e-4c16-ad23-ff7eae76c850/executor-bf0ca6de-17c6-437f-82e9-33afb01dce7d/blockmgr-1f66ae2a-0a3a-4a99-8f75-b65b5d4e5e0c/19/rdd_4_1
22/05/11 11:30:41 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 7)
java.io.StreamCorruptedException: invalid stream header: 41ACED00
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:938) ~[?:1.8.0_322]
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:396) ~[?:1.8.0_322]
at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:66) ~[spark-core_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
...
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_322]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_322]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_322]
22/05/11 11:30:41 INFO CoarseGrainedExecutorBackend: Got assigned task 8
22/05/11 11:30:41 INFO Executor: Running task 0.1 in stage 3.0 (TID 8)
22/05/11 11:30:41 INFO Executor: Finished task 0.1 in stage 3.0 (TID 8). 1623 bytes result sent to driver
```
Closes #36512 from attilapiros/handleCorruptedCachedRdd.
Authored-by: attilapiros <piros.attila.zsolt@gmail.com>
Signed-off-by: yi.wu <yi.wu@databricks.com>
### What changes were proposed in this pull request? After SPARK-39521, we don't need such information anymore because they are all classified under Actions as below: <img width="416" alt="Screen Shot 2022-06-21 at 9 05 13 AM" src="https://user-images.githubusercontent.com/6477701/174708227-62a54a73-2679-49e8-b017-22f98d789400.png"> ### Why are the changes needed? To remove unnecessary information ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Tested as below: <img width="288" alt="Screen Shot 2022-06-21 at 1 36 13 PM" src="https://user-images.githubusercontent.com/6477701/174717053-a387949e-88b5-44c0-a478-f7525e0c7019.png"> Closes #36935 from HyukjinKwon/minor-titles. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request? 1. Move `DESCRIBE TABLE` parsing tests to `DescribeRelationParserSuite`. 2. Put common `DESCRIBE TABLE` tests into one trait `org.apache.spark.sql.execution.command.DescribeTableSuiteBase`, and put datasource specific tests to the `v1.DescribeTableSuite` and `v2.DescribeTableSuite`. The changes follow the approach of #30287. Closes #36671. ### Why are the changes needed? 1. The unification will allow to run common `DESCRIBE TABLE` tests for both DSv1/Hive DSv1 and DSv2 2. We can detect missing features and differences between DSv1 and DSv2 implementations. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? By running the modified test suites: ``` $ build/sbt "testOnly *DDLParserSuite" $ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *HiveDDLSuite" ``` and new test suites: ``` $ build/sbt "sql/test:testOnly *DescribeTableParserSuite" $ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *DescribeTableSuite" ``` Closes #36912 from MaxGekk/unify-describe-table-tests-4. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>
…hen committed file not consistent with task status ### What changes were proposed in this pull request? There is a case for current code may cause data correctness issue: 1. One writing data task commit task success, but the container was killed after commit task, so this task failed, but the data file still remained under committedTaskPath . 2. DAGScheduler call `handleCompletedTask` and then call `outputCommitCoordinator.taskCompleted`, outputCommitCoordinator will remove the lock of failed task's partition. 3. Then failed task's rerun with an new attempt, the new attempt task call `outputCommitCoordinator.canCommit()` will return true since the lock of this partition had been removed, then it commit task success, also task final succeed. 4. Two files remained under this job's attempt path for same partition. 5. CommitJob commit both two committed task path's data. 6. Finally data duplicated. In this pr, we do below since: 1. When commit task success, executor send an CommitOutputSuccess message to outputCommitCoordinator. 2. When outputCommitCoordinator handle `taskComplete`, if task failed but commit success, means data duplicate will happen, we should failed to job. ### Why are the changes needed? Fix data duplicated issue. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Closes #36564 from AngersZhuuuu/SPARK-39195. Lead-authored-by: Angerszhuuuu <angers.zhu@gmail.com> Co-authored-by: AngersZhuuuu <angers.zhu@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
### What changes were proposed in this pull request? This PR aims to improve YARN client mode to support IPv6. ### Why are the changes needed? On `IPv6`-only environment, YARN module passes the UTs if we exclude `ExtendedYarnTest`. Among the failures, this PR focus on `YARN client mode` first. Please note that YARN module passes if IPv4 and IPv6 coexist. **BEFORE** ``` % build/sbt "yarn/testOnly *.YarnClusterSuite -- -z yarn-client" -Pyarn ... Using SPARK_LOCAL_HOSTNAME=[::1] Using SPARK_LOCAL_IP=[::1] ... [info] YarnClusterSuite: [info] - run Spark in yarn-client mode *** FAILED *** (2 seconds, 144 milliseconds) [info] FAILED did not equal FINISHED (stdout/stderr was not captured) (BaseYarnClusterSuite.scala:233) ... ``` **AFTER** ``` % build/sbt "yarn/testOnly *.YarnClusterSuite -- -z yarn-client" -Pyarn ... Using SPARK_LOCAL_HOSTNAME=[::1] Using SPARK_LOCAL_IP=[::1] ... [info] YarnClusterSuite: [info] - run Spark in yarn-client mode (10 seconds, 172 milliseconds) [info] - run Spark in yarn-client mode with unmanaged am (7 seconds, 108 milliseconds) [info] - run Spark in yarn-client mode with different configurations, ensuring redaction (8 seconds, 112 milliseconds) [info] - SPARK-35672: run Spark in yarn-client mode with additional jar using URI scheme 'local' (8 seconds, 118 milliseconds) [info] - SPARK-35672: run Spark in yarn-client mode with additional jar using URI scheme 'local' and gateway-replacement path (7 seconds, 115 milliseconds) [info] - SPARK-35672: run Spark in yarn-client mode with additional jar using URI scheme 'file' (9 seconds, 104 milliseconds) [info] - run Python application in yarn-client mode (11 seconds, 95 milliseconds) [info] Run completed in 1 minute, 21 seconds. [info] Total number of tests run: 7 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 7, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. Closes #36939 from dongjoon-hyun/SPARK-39542. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
… to V2 data sources
### What changes were proposed in this pull request?
Extend DEFAULT column support in ALTER TABLE ADD COLUMNS commands to include V2 data sources.
Example:
```
> create or replace table t (a string default 'abc') using $v2Source
> insert into t values (default)
> alter table t add column (b string default 'def')
> insert into t values ("ghi")
> Select * from t
"abc", "def",
"ghi", "def"
```
### Why are the changes needed?
This makes V2 data sources easier to use and extend.
### Does this PR introduce _any_ user-facing change?
Yes.
### How was this patch tested?
This PR includes new test coverage.
Closes #36771 from dtenedor/default-cols-v2-tables.
Authored-by: Daniel Tenedorio <daniel.tenedorio@databricks.com>
Signed-off-by: Gengliang Wang <gengliang@apache.org>
…an (DataSourceV2) ### What changes were proposed in this pull request? As `SupportsReportPartitioning` allows implementations of `Scan` provide Spark with information about the exiting partitioning of data read by a `DataSourceV2`, a similar mix in interface `SupportsReportOrdering` should provide order information. ### Why are the changes needed? This prevents Spark from sorting data if they already exhibit a certain order provided by the source. ### Does this PR introduce _any_ user-facing change? It adds `SupportsReportOrdering` mix in interface. ### How was this patch tested? This adds tests to `DataSourceV2Suite`, similar to the test for `SupportsReportPartitioning`. Closes #35965 from EnricoMi/branch-datasourcev2-output-ordering. Authored-by: Enrico Minack <github@enrico.minack.dev> Signed-off-by: Chao Sun <sunchao@apple.com>
### What changes were proposed in this pull request? Upgrade mysql-connector-java from 8.0.27 to 8.0.29 ### Why are the changes needed? Improper Handling of Insufficient Permissions or Privileges in MySQL Connectors Java. Vulnerability in the MySQL Connectors product of Oracle MySQL (component: Connector/J). Supported versions that are affected are 8.0.27 and prior. Difficult to exploit vulnerability allows high privileged attacker with network access via multiple protocols to compromise MySQL Connectors. Successful attacks of this vulnerability can result in takeover of MySQL Connectors. CVSS 3.1 Base Score 6.6 (Confidentiality, Integrity and Availability impacts). CVSS Vector: (CVSS:3.1/AV:N/AC:H/PR:H/UI:N/S:U/C:H/I:H/A:H). [CVE-2022-21363](https://nvd.nist.gov/vuln/detail/CVE-2022-21363) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA Closes #36938 from bjornjorgensen/Upgrade-mysql-connector-java-to-8.0.28. Lead-authored-by: Bjørn Jørgensen <bjornjorgensen@gmail.com> Co-authored-by: Bjørn Jørgensen <47577197+bjornjorgensen@users.noreply.github.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
…ator ### What changes were proposed in this pull request? Remove flaky test of OutputCommitCoordinator ### Why are the changes needed? Remove flaky test after disscuss ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Not need Closes #36943 from AngersZhuuuu/SPARK-39195-FOLLOWUP. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
…Metrics ### What changes were proposed in this pull request? Deprecate `scoreLabelsWeight` in `BinaryClassificationMetrics` ### Why are the changes needed? `scoreLabelsWeight` in `BinaryClassificationMetrics` is a public variable, but it should be private since it is only for internal purpose. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? existing UT Closes #36926 from zhengruifeng/ml_make_binclsmetrics_rdd_private. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Sean Owen <srowen@gmail.com>
### What changes were proposed in this pull request? This PR adds a check for invalid plans in AQE replanning process. The check will throw exceptions when it detects an invalid plan, causing AQE to void the current replanning result and keep using the latest valid plan. ### Why are the changes needed? AQE logical optimization rules can lead to invalid physical plans and cause runtime exceptions as certain physical plan nodes are not compatible with others. E.g., `BroadcastExchangeExec` can only work as a direct child of broadcast join nodes, but it could appear under other incompatible physical plan nodes because of empty relation propagation. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added UT. Closes #36953 from maryannxue/validate-aqe. Authored-by: Maryann Xue <maryann.xue@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
…f top-level columns ### What changes were proposed in this pull request? It turns out that I was wrong in #36727 . We still have the limitation (column name cannot contain dot) in master and 3.3 braches, in a very implicit way: The `V2ExpressionBuilder` has a boolean flag `nestedPredicatePushdownEnabled` whose default value is false. When it's false, it uses `PushableColumnWithoutNestedColumn` to match columns, which doesn't support dot in names. `V2ExpressionBuilder` is only used in 2 places: 1. `PushableExpression`. This is a pattern match that is only used in v2 agg pushdown 2. `PushablePredicate`. This is a pattern match that is used in various places, but all the caller sides set `nestedPredicatePushdownEnabled` to true. This PR removes the `nestedPredicatePushdownEnabled` flag from `V2ExpressionBuilder`, and makes it always support nested fields. `PushablePredicate` is also updated accordingly to remove the boolean flag, as it's always true. ### Why are the changes needed? Fix a mistake to eliminate an unexpected limitation in DS v2 pushdown. ### Does this PR introduce _any_ user-facing change? No for end users. For data source developers, they can trigger agg pushdowm more often. ### How was this patch tested? a new test Closes #36945 from cloud-fan/dsv2. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
…ic into precondition ### What changes were proposed in this pull request? This PR borrows the idea from #36928 but adds some more changes in order for scheduled jobs to share the `precondition` so all conditional logic is consolidated here. This PR also adds a new option to `is-changed.py` so dependent modules can be checked together. In this way, we don't have to change `build_and_test.yml` often when we add a new module. In addition, this PR removes `type` because `precondition` job now replaces it. Lastly, this PR enables PySpark, SparkR TPC-DS and Docker integration tests for scheduled jobs when applicable. Closes #36928 ### Why are the changes needed? To make it easier to read. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Tested locally and in my fork (https://github.com/HyukjinKwon/spark/actions) Closes #36940 from HyukjinKwon/SPARK-39529. Lead-authored-by: Hyukjin Kwon <gurwls223@apache.org> Co-authored-by: Enrico Minack <github@enrico.minack.dev> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
…erIPv6Addresses` ### What changes were proposed in this pull request? This PR aims to improve `SparkContext` to propagate `java.net.preferIPv6Addresses`. ### Why are the changes needed? This helps IPv6-only Spark applications by removing the explicit JVM setting burden. ### Does this PR introduce _any_ user-facing change? No. There is no change for IPv4 users. ### How was this patch tested? Pass the CIs. Closes #36959 from dongjoon-hyun/SPARK-39561. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request? This PR aims to support IPv6 in WebUI. Since `startJettyServer` expects `hostName`, we use `Utils.localHostNameForURI`. https://github.com/apache/spark/blob/59eee98024dac42309f2e7196c7e68832317f284/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala#L241-L242 ### Why are the changes needed? To support IPv6-only environment. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. Closes #36958 from dongjoon-hyun/SPARK-39559. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
…ala 2.13 to improve the performance
### What changes were proposed in this pull request?
`ExpressionSet ++` method in the master branch a little slower than the branch-3.3 with Scala-2.13, so this pr override `concat` method for `ExpressionSet` in Scala 2.13.
### Why are the changes needed?
Improve the performance
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GA
- Manual test 1:
microbench as follows and run with Scala 2.13:
```scala
val valuesPerIteration = 100000
val benchmark = new Benchmark("Test ExpressionSet ++ ", valuesPerIteration, output = output)
val aUpper = AttributeReference("A", IntegerType)(exprId = ExprId(1))
val initialSet = ExpressionSet(aUpper + 1 :: Rand(0) :: Nil)
val setToAddWithSameDeterministicExpression = ExpressionSet(aUpper + 1 :: Rand(0) :: Nil)
benchmark.addCase("Test ++") { _: Int =>
for (_ <- 0L until valuesPerIteration) {
initialSet ++ setToAddWithSameDeterministicExpression
}
}
benchmark.run()
```
**branch-3.3 result:**
```
OpenJDK 64-Bit Server VM 1.8.0_312-b07 on Linux 4.14.0_1-0-0-45
Intel(R) Xeon(R) Gold 6XXXC CPU 2.60GHz
Test ExpressionSet ++ : Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Test ++ 14 16 4 7.2 139.1 1.0X
```
**master result before this pr:**
```
OpenJDK 64-Bit Server VM 1.8.0_312-b07 on Linux 4.14.0_1-0-0-45
Intel(R) Xeon(R) Gold 6XXXC CPU 2.60GHz
Test ExpressionSet ++ : Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Test ++ 16 19 5 6.1 163.9 1.0X
```
**master result after this pr:**
```
OpenJDK 64-Bit Server VM 1.8.0_312-b07 on Linux 4.14.0_1-0-0-45
Intel(R) Xeon(R) Gold 6XXXC CPU 2.60GHz
Test ExpressionSet ++ : Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Test ++ 12 13 3 8.6 115.7 1.0X
```
- Manual test 2:
```
dev/change-scala-version.sh 2.13
mvn clean install -pl sql/core -am -DskipTests -Pscala-2.13
mvn test -pl sql/catalyst -Pscala-2.13
mvn test -pl sql/core -Pscala-2.13
```
```
Run completed in 10 minutes, 40 seconds.
Total number of tests run: 6584
Suites: completed 285, aborted 0
Tests: succeeded 6584, failed 0, canceled 0, ignored 5, pending 0
All tests passed.
```
```
Run completed in 1 hour, 27 minutes, 16 seconds.
Total number of tests run: 11745
Suites: completed 520, aborted 0
Tests: succeeded 11745, failed 0, canceled 7, ignored 57, pending 0
All tests passed.
```
Closes #36942 from LuciferYang/ExpressionSet.
Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Sean Owen <srowen@gmail.com>
### What changes were proposed in this pull request? Convert `CaseInsensitiveStringMap#logger` to static, it's safe since the `org.slf4j.Logger` is thread-safe. ### Why are the changes needed? To improve performance. The overhead of `LoggerFactory.getLogger(CaseInsensitiveStringMap.class)` is non-negligible. I always see this when I refresh the page "Thread dump for executor". <img width="1548" alt="image" src="https://user-images.githubusercontent.com/26535726/174699103-822a868b-e390-47ee-b65e-e5651ca60dca.png"> <img width="1526" alt="image" src="https://user-images.githubusercontent.com/26535726/174699135-6f8ccce2-14a4-4734-a2bb-08f51c292ec0.png"> ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing UT. Closes #36930 from pan3793/SPARK-39538. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Sean Owen <srowen@gmail.com>
…percent_rank ### What changes were proposed in this pull request? Change `LimitPushDownThroughWindow` so that it no longer supports pushing down a limit through a window using percent_rank. ### Why are the changes needed? Given a query with a limit of _n_ rows, and a window whose child produces _m_ rows, percent_rank will label the _nth_ row as 100% rather than the _mth_ row. This behavior conflicts with Spark 3.1.3, Hive 2.3.9 and Prestodb 0.268. #### Example Assume this data: ``` create table t1 stored as parquet as select * from range(101); ``` And also assume this query: ``` select id, percent_rank() over (order by id) as pr from t1 limit 3; ``` With Spark 3.2.1, 3.3.0, and master, the limit is applied before the percent_rank: ``` 0 0.0 1 0.5 2 1.0 ``` With Spark 3.1.3, and Hive 2.3.9, and Prestodb 0.268, the limit is applied _after_ percent_rank: Spark 3.1.3: ``` 0 0.0 1 0.01 2 0.02 ``` Hive 2.3.9: ``` 0: jdbc:hive2://localhost:10000> select id, percent_rank() over (order by id) as pr from t1 limit 3; . . . . . . . . . . . . . . . .> . . . . . . . . . . . . . . . .> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. +-----+-------+ | id | pr | +-----+-------+ | 0 | 0.0 | | 1 | 0.01 | | 2 | 0.02 | +-----+-------+ 3 rows selected (4.621 seconds) 0: jdbc:hive2://localhost:10000> ``` Prestodb 0.268: ``` id | pr ----+------ 0 | 0.0 1 | 0.01 2 | 0.02 (3 rows) ``` With this PR, Spark will apply the limit after percent_rank. ### Does this PR introduce _any_ user-facing change? No (besides changing percent_rank's behavior to be more like Spark 3.1.3, Hive, and Prestodb). ### How was this patch tested? New unit tests. Closes #36951 from bersprockets/percent_rank_issue. Authored-by: Bruce Robbins <bersprockets@gmail.com> Signed-off-by: Yuming Wang <yumwang@ebay.com>
…pass in IPv6 env ### What changes were proposed in this pull request? This PR aims to make `hive-thrift server` module unit tests pass in IPv6 environment. ### Why are the changes needed? To have IPv6 test coverage of Spark Thrift Server. ### Does this PR introduce _any_ user-facing change? No. This is a test-only change. ### How was this patch tested? Pass the CIs. Closes #36961 from dongjoon-hyun/SPARK-39562. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
…PARK-7837 Do not close output writer twice when commitTask() fail` ### What changes were proposed in this pull request? This PR adds additional assertions to fix flaky test `SPARK-7837 Do not close output writer twice when commitTask() fail` in `ParquetIOSuite`. ### Why are the changes needed? The test suite `SPARK-7837 Do not close output writer twice when commitTask() fails` only handle the `TaskSetFailed` event before SPARK-39195 due to `maxTaskFailures` is 1 when local mode. But after SPARK-39195, In `OutputCommitCoordinator#taskCompleted`, the processing of `stageState.authorizedCommitters(partition) == taskId` changes from debug logging to post an `StageFailed` event, the flaky test may handle to one of the `TaskSetFailed` event and `StageFailed` event, and the execution order of the two events is uncertain, and there may be the following two kinds of logs: **- Scenario 1(Success)** ``` 18:47:51.592 ERROR org.apache.spark.sql.execution.datasources.FileFormatWriter: Aborting job 59e24bb4-d3e3-41f6-aa80-411ddb481362. org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1) (localhost executor driver): org.apache.spark.SparkException: Task failed while writing rows. at org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:596) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:334) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$12(FileFormatWriter.scala:242) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92) at org.apache.spark.scheduler.Task.run(Task.scala:139) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1490) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.RuntimeException: Intentional exception for testing purposes at scala.sys.package$.error(package.scala:30) at org.apache.spark.sql.execution.datasources.parquet.TaskCommitFailureParquetOutputCommitter.commitTask(ParquetIOSuite.scala:1552) at org.apache.spark.mapred.SparkHadoopMapRedUtil$.$anonfun$commitTask$1(SparkHadoopMapRedUtil.scala:51) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:619) at org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:51) at org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:78) at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitTask(HadoopMapReduceCommitProtocol.scala:279) at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.$anonfun$commit$1(FileFormatDataWriter.scala:107) at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:619) at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:107) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:318) at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1524) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:324) ... 9 more Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2706) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2642) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2641) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2641) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1189) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1189) at scala.Option.foreach(Option.scala:407) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1189) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2897) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2836) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2825) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:952) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2222) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:231) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:189) at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113) at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111) at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125) ``` **- Scenario 2(Failed)** ``` 18:49:14.145 ERROR org.apache.spark.sql.execution.datasources.FileFormatWriter: Aborting job 82a0051d-7fe4-4632-8c9c-78dfbfaf5819. org.apache.spark.SparkException: Job aborted due to stage failure: Authorized committer (attemptNumber=0, stage=1, partition=0) failed; but task commit success, data duplication may happen. at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2706) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2642) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2641) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2641) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleStageFailed$1(DAGScheduler.scala:1182) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleStageFailed$1$adapted(DAGScheduler.scala:1182) at scala.Option.foreach(Option.scala:407) at org.apache.spark.scheduler.DAGScheduler.handleStageFailed(DAGScheduler.scala:1182) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2894) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2836) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2825) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:952) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2222) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:231) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:189) at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113) at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111) at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125) ``` If `maxTaskFailures` is changed to `>=2`, Scenario 2 will appear stably, but the flaky test runs in the local mode and `maxTaskFailures` is always 1, so the this pr just adds additional assertions to workaround. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GitHub Actions Closes #37245 from LuciferYang/SPARK-39622. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
This reverts commit 67efa31.
### What changes were proposed in this pull request? In the PR, I propose to fix the width of the day-of-month field to the maximum value 2: <img width="573" alt="Screenshot 2022-07-21 at 21 11 11" src="https://user-images.githubusercontent.com/1580697/180284361-ac0d11f2-5fb2-4fa9-b950-8d5cf5e7b16f.png"> and clarify the behavior of parsing numbers: <img width="751" alt="Screenshot 2022-07-21 at 21 11 23" src="https://user-images.githubusercontent.com/1580697/180284644-7af789af-5ebb-4e7e-a041-a00dea8c7b5b.png"> ### Why are the changes needed? To not confuse users and improve user experience with Spark SQL. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? By building docs and manually checking the changes. Closes #37246 from MaxGekk/docs-fix-width-of-days. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request? This PR aims to upgrade scalatest-maven-plugin to version 2.1.0. ### Why are the changes needed? The last upgrade occurred 1 year ago. release note: > https://github.com/scalatest/scalatest-maven-plugin/releases/tag/release-2.1.0 v2.0.2 VS v2.1.0 > scalatest/scalatest-maven-plugin@v2.0.2...release-2.1.0 compile with scala 2.12.16 https://github.com/scalatest/scalatest-maven-plugin/blob/master/pom.xml#L22~L24 https://github.com/scalatest/scalatest-maven-plugin/blob/master/pom.xml#L116~L121 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. Closes #37237 from panbingkun/bump_scalatest-maven-plugin. Authored-by: panbingkun <pbk1982@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request? move `libfontconfig1-dev libharfbuzz-dev libfribidi-dev libfreetype6-dev libpng-dev libtiff5-dev libjpeg-dev` from Install dependencies for documentation generation to Install R linter dependencies and SparkR Update after #37243: **add `apt update` before installation.** ### Why are the changes needed? to make CI happy Install R linter dependencies and SparkR started to fail after devtools_2.4.4 was released. ``` --------------------------- [ANTICONF] -------------------------------- Configuration failed to find the fontconfig freetype2 library. Try installing: * deb: libfontconfig1-dev (Debian, Ubuntu, etc) * rpm: fontconfig-devel (Fedora, EPEL) * csw: fontconfig_dev (Solaris) * brew: freetype (OSX) it seems that libfontconfig1-dev is needed now. ``` also refer to r-lib/systemfonts#35 (comment) ### Does this PR introduce any user-facing change? No ### How was this patch tested? CI passed Closes #37247 from Yikun/patch-25. Lead-authored-by: Ruifeng Zheng <ruifengz@apache.org> Co-authored-by: Yikun Jiang <yikunkero@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request? Enable Spark on K8S integration tests in Github Action based on minikube: - The K8S IT will always triggered in user fork repo and `apache/spark` merged commits to master branch - This PR does NOT contains Volcano related test due to limited resource of github action. - minikube installation is allowed by Apache Infra: [INFRA-23000](https://issues.apache.org/jira/projects/INFRA/issues/INFRA-23000) - Why setting driver 0.5 cpu, executor 0.2 cpu? * Github-hosted runner hardware limited: [2U7G](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources), so cpu resource is very limited. * IT Job available CPU = 2U - 0.85U (K8S deploy) = 1.15U * There are 1.15 cpu left after k8s installation, to meet the requirement of K8S tests (one driver + max to 3 executors). * For memory: 6947 is maximum (Otherwise raise `Exiting due to RSRC_OVER_ALLOC_MEM: Requested memory allocation 7168MB is more than your system limit 6947MB.`), but this is not integer multiple of 1024, so I just set this to 6144 for better resource statistic. - Time cost info: * 14 mins to compile related code. * 3 mins to build docker images. * 20-30 mins to test * Total: about 30-40 mins ### Why are the changes needed? This will also improve the efficiency of K8S development and guarantee the quality of spark on K8S and spark docker image in some level. ### Does this PR introduce _any_ user-facing change? No, dev only. ### How was this patch tested? CI passed Closes #35830 Closes #37244 from Yikun/SPARK-38597-k8s-it. Authored-by: Yikun Jiang <yikunkero@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
|
Looks like I checked out the code for this PR wrong. Dropping this. |
github-actions
bot
added
AVRO
BUILD
CORE
DEPLOY
DOCS
DSTREAM
EXAMPLES
GRAPHX
INFRA
KUBERNETES
MESOS
ML
MLLIB
PYTHON
R
SPARK SHELL
SQL
STRUCTURED STREAMING
WEB UI
WINDOWS
YARN
labels
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Disable DEFAULT column SQLConf until implementation is complete.
Why are the changes needed?
This prevents users from trying the feature before all the coding is finished.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Existing test coverage exercises the configuration.