Skip to content

Comments

[SPARK-44590][SQL][CONNECT] Remove the arrow batch record limit for SqlCommandResult#42213

Closed
heyihong wants to merge 2 commits intoapache:masterfrom
heyihong:SPARK-44590
Closed

[SPARK-44590][SQL][CONNECT] Remove the arrow batch record limit for SqlCommandResult#42213
heyihong wants to merge 2 commits intoapache:masterfrom
heyihong:SPARK-44590

Conversation

@heyihong
Copy link
Contributor

@heyihong heyihong commented Jul 28, 2023

What changes were proposed in this pull request?

  • Remove the arrow batch record limit for SqlCommandResult

Why are the changes needed?

  • Limit on size is sufficient

Does this PR introduce any user-facing change?

No

How was this patch tested?

build/sbt "connect-client-jvm/testOnly *ClientE2ETestSuite"

@ShreyeshArangath
Copy link

what do you mean by the limit on size being sufficient? could you provide more context?

@HyukjinKwon
Copy link
Member

e.g., sql("SHOW TABLE") returns too many records more than the batch size.

@HyukjinKwon HyukjinKwon changed the title [SPARK-44590] Remove the arrow batch record limit for SqlCommandResult [SPARK-44590][SQL][CONNECT] Remove the arrow batch record limit for SqlCommandResult Jul 30, 2023
@HyukjinKwon
Copy link
Member

Merged to master and branch-3.5.

HyukjinKwon pushed a commit that referenced this pull request Jul 30, 2023
…qlCommandResult

### What changes were proposed in this pull request?

- Remove the arrow batch record limit for SqlCommandResult

### Why are the changes needed?

- Limit on size is sufficient

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

`build/sbt "connect-client-jvm/testOnly *ClientE2ETestSuite"`

Closes #42213 from heyihong/SPARK-44590.

Authored-by: Yihong He <yihong.he@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit d232e6e)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants