[Spark] Fix the inaccurate issue of obtaining COLUMN_SIZE in the decimal field jdbc of spark engine #5750

waywtdcc · 2023-11-22T08:56:25Z

🔍 Description

Issue References 🔗

This pull request fixes #

Describe Your Solution 🔧

Fix the inaccuracy in getting COLUMN_SIZE from spark engine's decimal field jdbc. The current jdbc's get column size gets the decimal field as default size, which is inaccurate; if it is decimal(20,3), the obtained column size is 16; the actual is 20.

Types of changes 🔖

Bugfix (non-breaking change which fixes an issue)

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests

…ead in a stream

...ubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/SchemaHelper.scala

yaooqinn · 2023-11-22T09:07:52Z

Nice catch! Hi @waywtdcc , can we restore the checklist from PR desc?

codecov-commenter · 2023-11-22T10:28:43Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (a23b16a) 61.36% compared to head (2d288f5) 61.35%.
Report is 12 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #5750      +/-   ##
============================================
- Coverage     61.36%   61.35%   -0.01%     
  Complexity       23       23              
============================================
  Files           607      607              
  Lines         35897    35942      +45     
  Branches       4923     4933      +10     
============================================
+ Hits          22027    22052      +25     
- Misses        11479    11506      +27     
+ Partials       2391     2384       -7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…ead in a stream

...ubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/SchemaHelper.scala

yaooqinn

Hi, please restore the PR description

…mal field jdbc of spark engine

waywtdcc · 2023-11-27T02:51:01Z

Hi, please restore the PR description

ok

…IZE in the decimal field jdbc of spark engine # 🔍 Description ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 Fix the inaccuracy in getting COLUMN_SIZE from spark engine's decimal field jdbc. The current jdbc's get column size gets the decimal field as default size, which is inaccurate; if it is decimal(20,3), the obtained column size is 16; the actual is 20. ## Types of changes 🔖 - [X] Bugfix (non-breaking change which fixes an issue) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- Closes #5750 from waywtdcc/support_spark_decimal2. Closes #5750 2d288f5 [waywtdcc] [Spark] Fix the inaccurate issue of obtaining COLUMN_SIZE in the decimal field jdbc of spark engine 4286354 [waywtdcc] Support flink engine under the select statement, the results can be read in a stream e5b74b0 [waywtdcc] Support flink engine under the select statement, the results can be read in a stream Authored-by: waywtdcc <waywtdcc@163.com> Signed-off-by: Kent Yao <yao@apache.org> (cherry picked from commit eb9e88b) Signed-off-by: Kent Yao <yao@apache.org>

yaooqinn · 2023-11-29T03:50:26Z

Merged to master, 1.8, 1.7

…se notes # 🔍 Description ## Issue References 🔗 Currently, we use a rather primitive way to manually write release notes from scratch, and some of the mechanical and repetitive work can be simplified by the scripts. ## Describe Your Solution 🔧 Adds a script to simplify the process of creating release notes. Note: it just simplifies some processes, the release manager still needs to tune the outputs by hand. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 ``` RELEASE_TAG=v1.8.1 PREVIOUS_RELEASE_TAG=v1.8.0 build/release/pre_gen_release_notes.py ``` ``` $ head build/release/commits-v1.8.1.txt [KYUUBI #5981] Deploy Spark Hive connector with Scala 2.13 to Maven Central [KYUUBI #6058] Make Jetty server stop timeout configurable [KYUUBI #5952][1.8] Disconnect connections without running operations after engine maxlife time graceful period [KYUUBI #6048] Assign serviceNode and add volatile for variables [KYUUBI #5991] Error on reading Atlas properties composed of multi values [KYUUBI #6045] [REST] Sync the AdminRestApi with the AdminResource Apis [KYUUBI #6047] [CI] Free up disk space [KYUUBI #6036] JDBC driver conditional sets fetchSize on opening session [KYUUBI #6028] Exited spark-submit process should not block batch submit queue [KYUUBI #6018] Speed up GetTables operation for Spark session catalog ``` ``` $ head build/release/contributors-v1.8.1.txt * Shaoyun Chen -- [KYUUBI #5857][KYUUBI #5720][KYUUBI #5785][KYUUBI #5617] * Chao Chen -- [KYUUBI #5750] * Flyangz -- [KYUUBI #5832] * Pengqi Li -- [KYUUBI #5713] * Bowen Liang -- [KYUUBI #5730][KYUUBI #5802][KYUUBI #5767][KYUUBI #5831][KYUUBI #5801][KYUUBI #5754][KYUUBI #5626][KYUUBI #5811][KYUUBI #5853][KYUUBI #5765] * Paul Lin -- [KYUUBI #5799][KYUUBI #5814] * Senmiao Liu -- [KYUUBI #5969][KYUUBI #5244] * Xiao Liu -- [KYUUBI #5962] * Peiyue Liu -- [KYUUBI #5331] * Junjie Ma -- [KYUUBI #5789] ``` --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6074 from pan3793/release-script. Closes #6074 3d5ec20 [Cheng Pan] credits 1765279 [Cheng Pan] Add a script to simplify the process of creating release notes Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>

… release notes # 🔍 Description ## Issue References 🔗 Currently, we use a rather primitive way to manually write release notes from scratch, and some of the mechanical and repetitive work can be simplified by the scripts. ## Describe Your Solution 🔧 Adds a script to simplify the process of creating release notes. Note: it just simplifies some processes, the release manager still needs to tune the outputs by hand. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 ``` RELEASE_TAG=v1.8.1 PREVIOUS_RELEASE_TAG=v1.8.0 build/release/pre_gen_release_notes.py ``` ``` $ head build/release/commits-v1.8.1.txt [KYUUBI apache#5981] Deploy Spark Hive connector with Scala 2.13 to Maven Central [KYUUBI apache#6058] Make Jetty server stop timeout configurable [KYUUBI apache#5952][1.8] Disconnect connections without running operations after engine maxlife time graceful period [KYUUBI apache#6048] Assign serviceNode and add volatile for variables [KYUUBI apache#5991] Error on reading Atlas properties composed of multi values [KYUUBI apache#6045] [REST] Sync the AdminRestApi with the AdminResource Apis [KYUUBI apache#6047] [CI] Free up disk space [KYUUBI apache#6036] JDBC driver conditional sets fetchSize on opening session [KYUUBI apache#6028] Exited spark-submit process should not block batch submit queue [KYUUBI apache#6018] Speed up GetTables operation for Spark session catalog ``` ``` $ head build/release/contributors-v1.8.1.txt * Shaoyun Chen -- [KYUUBI apache#5857][KYUUBI apache#5720][KYUUBI apache#5785][KYUUBI apache#5617] * Chao Chen -- [KYUUBI apache#5750] * Flyangz -- [KYUUBI apache#5832] * Pengqi Li -- [KYUUBI apache#5713] * Bowen Liang -- [KYUUBI apache#5730][KYUUBI apache#5802][KYUUBI apache#5767][KYUUBI apache#5831][KYUUBI apache#5801][KYUUBI apache#5754][KYUUBI apache#5626][KYUUBI apache#5811][KYUUBI apache#5853][KYUUBI apache#5765] * Paul Lin -- [KYUUBI apache#5799][KYUUBI apache#5814] * Senmiao Liu -- [KYUUBI apache#5969][KYUUBI apache#5244] * Xiao Liu -- [KYUUBI apache#5962] * Peiyue Liu -- [KYUUBI apache#5331] * Junjie Ma -- [KYUUBI apache#5789] ``` --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes apache#6074 from pan3793/release-script. Closes apache#6074 3d5ec20 [Cheng Pan] credits 1765279 [Cheng Pan] Add a script to simplify the process of creating release notes Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>

Support flink engine under the select statement, the results can be r…

e5b74b0

…ead in a stream

github-actions bot added the module:spark label Nov 22, 2023

yaooqinn reviewed Nov 22, 2023

View reviewed changes

...ubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/SchemaHelper.scala Show resolved Hide resolved

yaooqinn assigned waywtdcc Nov 22, 2023

yaooqinn modified the milestones: v1.9.0, v1.7.4 Nov 22, 2023

Support flink engine under the select statement, the results can be r…

4286354

…ead in a stream

waywtdcc requested a review from yaooqinn November 23, 2023 00:21

yaooqinn reviewed Nov 23, 2023

View reviewed changes

...ubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/SchemaHelper.scala Outdated Show resolved Hide resolved

yaooqinn requested changes Nov 23, 2023

View reviewed changes

waywtdcc requested a review from yaooqinn November 24, 2023 06:36

[Spark] Fix the inaccurate issue of obtaining COLUMN_SIZE in the deci…

2d288f5

…mal field jdbc of spark engine

yaooqinn approved these changes Nov 27, 2023

View reviewed changes

yaooqinn closed this in eb9e88b Nov 29, 2023

pan3793 mentioned this pull request Dec 11, 2023

[TASK][EASY] Fix the inaccurate issue of obtaining COLUMN_SIZE in the decimal field JDBC of Spark engine #5844

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Spark] Fix the inaccurate issue of obtaining COLUMN_SIZE in the decimal field jdbc of spark engine #5750

[Spark] Fix the inaccurate issue of obtaining COLUMN_SIZE in the decimal field jdbc of spark engine #5750

waywtdcc commented Nov 22, 2023

yaooqinn commented Nov 22, 2023

codecov-commenter commented Nov 22, 2023 •

edited

yaooqinn left a comment

waywtdcc commented Nov 27, 2023

yaooqinn commented Nov 29, 2023

[Spark] Fix the inaccurate issue of obtaining COLUMN_SIZE in the decimal field jdbc of spark engine #5750

[Spark] Fix the inaccurate issue of obtaining COLUMN_SIZE in the decimal field jdbc of spark engine #5750

Conversation

waywtdcc commented Nov 22, 2023

🔍 Description

Issue References 🔗

Describe Your Solution 🔧

Types of changes 🔖

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests

yaooqinn commented Nov 22, 2023

codecov-commenter commented Nov 22, 2023 • edited

Codecov Report

yaooqinn left a comment

Choose a reason for hiding this comment

waywtdcc commented Nov 27, 2023

yaooqinn commented Nov 29, 2023

codecov-commenter commented Nov 22, 2023 •

edited