[ARROW] Fix Spark session timezone format in arrow-based result format #4326
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
kind:documentation
cfmcgrady
changed the title
cfmcgrady
force-pushed
the
arrow-string-ts
branch
from
612ec46
to
87c6f9e
Compare
5 tasks
pan3793
reviewed
Feb 17, 2023
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/arrow/ArrowColumnarBatchRow.java
Outdated
Show resolved
Hide resolved
…uubi/engine/spark/schema/RowSet.scala
pan3793
reviewed
Feb 17, 2023
| return colTypes; | ||
| } | ||
| /** | ||
| * 1. the complex types (map/array/struct) are always converted to string type to transport 2. if |
There was a problem hiding this comment.
The spotless plugin makes this change, otherwise, we wouldn't pass the style check :(
Codecov Report
@@ Coverage Diff @@
## master #4326 +/- ##
============================================
+ Coverage 53.53% 53.61% +0.07%
Complexity 13 13
============================================
Files 562 562
Lines 30704 30804 +100
Branches 4142 4153 +11
============================================
+ Hits 16438 16515 +77
- Misses 12721 12732 +11
- Partials 1545 1557 +12
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
pan3793
approved these changes
Feb 18, 2023
pan3793
pushed a commit
that referenced
this pull request
Feb 18, 2023
…ed result format ### _Why are the changes needed?_ 1. this PR introduces a new configuration called `kyuubi.operation.result.arrow.timestampAsString`, when true, arrow-based rowsets will convert timestamp-type columns to strings for transmission. 2. `kyuubi.operation.result.arrow.timestampAsString` default setting to false for better transmission performance 3. the PR fixes timezone issue in arrow based result format described in #3958 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4326 from cfmcgrady/arrow-string-ts. Closes #4326 38c7fc9 [Fu Chen] fix style d864db0 [Fu Chen] address comment b714b3e [Fu Chen] revert externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/RowSet.scala 6c4eb50 [Fu Chen] minor 289b600 [Fu Chen] timstampAsString = false by default 78b7cab [Fu Chen] fix f560135 [Fu Chen] debug info b8e4b28 [Fu Chen] fix ut 87c6f9e [Fu Chen] update docs 86f6cb7 [Fu Chen] arrow based rowset timestamp as string Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> (cherry picked from commit 6bd0016) Signed-off-by: Cheng Pan <chengpan@apache.org>
|
Thanks, merged to master/1.7 |
SusurHe
pushed a commit
to SusurHe/incubator-kyuubi
that referenced
this pull request
Feb 21, 2023
* 'master' of github.com:apache/kyuubi: (1557 commits) [KYUUBI apache#3951][FOLLOWUP] Audit the rest request params [KYUUBI apache#4377] Grant execute permission to release scripts [KYUUBI apache#4374] Release uploading should include kyuubi-spark-connector-hive [KYUUBI apache#4267] Show warning if SessionHandle is invalid [KYUUBI apache#4385] [DOCS] Refine release process [KYUUBI apache#4352] Support System.gc() with periodic GC interval [KYUUBI apache#4152][FOLLOWUP] LDAP configurations should be server-only [KYUUBI apache#4373] Using SVN_STAGING_REPO instead of SVN_STAGING_REPO in the release script to fix echo message [KYUUBI apache#4372] Support to return null value for OperationsResource rowset [KYUUBI apache#4371] Fix typo in `kyuubi_ecosystem.drawio` [KYUUBI apache#4216] Support to transfer client version for kyuubi hive jdbc and rest client sdk [KYUUBI apache#4345] Add the doc of kyuubi trino server [KYUUBI apache#3081][DOCS] Add Hudi connector doc in Trino [KYUUBI apache#4357] Bump Jersey from 2.38 to 2.39 [KYUUBI apache#4338][FOLLOWUP] Fix K8s integration tests [KYUUBI apache#4326] [ARROW] Fix Spark session timezone format in arrow-based result format [KYUUBI apache#4360][FOLLOWUP] Get valid unlimited users from existing limiters instead of conf [KYUUBI apache#4362] Add `_configurations` in kerberos.rst [KYUUBI apache#4338] Bump Spark from 3.3.1 to 3.3.2 [KYUUBI apache#4119][FOLLOWUP] Add app start time for batch api docs ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
this PR introduces a new configuration called
kyuubi.operation.result.arrow.timestampAsString, when true, arrow-based rowsets will convert timestamp-type columns to strings for transmission.kyuubi.operation.result.arrow.timestampAsStringdefault setting to false for better transmission performancethe PR fixes timezone issue in arrow based result format described in Fix Spark session timezone format #3958
How was this patch tested?
Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before make a pull request