Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ARROW] Reflective calls to the function ArrowConverters#fromBatchIterator #4895

Closed
wants to merge 2 commits into from

Conversation

cfmcgrady
Copy link
Contributor

@cfmcgrady cfmcgrady commented May 29, 2023

Why are the changes needed?

to adapt Spark 3.5

the signature of function ArrowConverters#fromBatchIterator is changed in SPARK-43528 (since Spark 3.5)

Spark 3.4 or previous

  private[sql] def fromBatchIterator(
      arrowBatchIter: Iterator[Array[Byte]],
      schema: StructType,
      timeZoneId: String,
      context: TaskContext): Iterator[InternalRow]

Spark 3.5 or later

  private[sql] def fromBatchIterator(
      arrowBatchIter: Iterator[Array[Byte]],
      schema: StructType,
      timeZoneId: String,
      errorOnDuplicatedFieldNames: Boolean,
      context: TaskContext): Iterator[InternalRow]

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before make a pull request

@cfmcgrady cfmcgrady changed the title [ARROW] Reflective calls to the function ArrowConverters#fromBatchIterator [ARROW] Reflective calls to the function ArrowConverters#fromBatchIterator May 29, 2023
@codecov-commenter
Copy link

Codecov Report

Merging #4895 (87d5b72) into master (07e590a) will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #4895   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         557     557           
  Lines       30705   30704    -1     
  Branches     3997    3997           
======================================
+ Misses      30705   30704    -1     
Impacted Files Coverage Δ
...rk/sql/execution/arrow/KyuubiArrowConverters.scala 0.00% <ø> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@pan3793 pan3793 added this to the v1.8.0 milestone May 30, 2023
@pan3793 pan3793 closed this in 77b28e9 May 30, 2023
@pan3793
Copy link
Member

pan3793 commented May 30, 2023

Thanks, merged to master

@cfmcgrady cfmcgrady deleted the arrow-spark35 branch May 30, 2023 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants