[KYUUBI #3885][FOLLOWUP] Fix memory leak when using incremental collect mode in JDBC engine #5161
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wForget
reviewed
Aug 15, 2023
| if (incrementalCollect) { | ||
| info("Execute in incremental collect mode") | ||
| new IterableFetchIterator(resultSetWrapper.toIterable) | ||
| new IterableFetchIterator[Row](new Iterable[Row] { |
Member
There was a problem hiding this comment.
This will cause the resultSet to be fetched lazily, so we can not close the statement in the finally block.
Contributor
Author
There was a problem hiding this comment.
Thanks for the hint. It might be the cause for the failure of closing statement too early. (https://github.com/apache/kyuubi/actions/runs/5857069896/job/15886592243#step:7:292)
Let me reopen this PR to have another try.
bowenliang123
changed the title
…losing statemnt too early
Codecov Report
@@ Coverage Diff @@
## master #5161 +/- ##
======================================
Coverage 0.00% 0.00%
======================================
Files 566 566
Lines 31624 31699 +75
Branches 4122 4134 +12
======================================
- Misses 31624 31699 +75 see 19 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
Contributor
Author
|
I may need a hand in this, for early closed result set and unset operationHandle. cc @zhaomin1423 @wForget ci log: https://github.com/apache/kyuubi/actions/runs/5863404047/job/15896805347?pr=5161#step:7:272 |
Member
|
sql syntax error: |
wForget
reviewed
Aug 17, 2023
| private val fetched = new AtomicBoolean(false) | ||
| override def iterator: Iterator[Row] = { | ||
| if (fetched.getAndSet(true)) { | ||
| jdbcStatement.synchronized { |
Member
There was a problem hiding this comment.
we need to re-execute the statement when resetPosition is called
Contributor
There was a problem hiding this comment.
Would re-execution be a possible problem? For example, insert is executed twice.
Member
There was a problem hiding this comment.
For update statement hasResult should be false.
wForget
reviewed
Aug 17, 2023
| private lazy val metadata = currentResult.getMetaData | ||
|
|
||
| override def hasNext: Boolean = { | ||
| if (currentResult.isClosed) return false |
Member
There was a problem hiding this comment.
FetchResult may be called again when the previous resultSet has ended, so we need to determine whether the resultSet has been closed.
wForget
changed the title
cxzl25
changed the title
cxzl25
previously approved these changes
Aug 22, 2023
| private val fetched = new AtomicBoolean(false) | ||
| override def iterator: Iterator[Row] = { | ||
| if (fetched.getAndSet(true)) { | ||
| jdbcStatement.synchronized { |
Contributor
There was a problem hiding this comment.
Would re-execution be a possible problem? For example, insert is executed twice.
3 tasks
pan3793
pushed a commit
that referenced
this pull request
Nov 1, 2023
… JDBC engine ### _Why are the changes needed?_ Similar to #3885, there is also memory leak in the jdbc engine when using incremental collect mode. Duplicate of #5161 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5570 from wForget/hotfix. Closes #5570 dbeddc9 [wforget] comment 0c4e499 [wforget] fix b7c421a [wforget] fix 0ef21ce [wforget] fix f5f2e80 [wforget] Fix memory leak when using incremental collect mode in JDBC engine Authored-by: wforget <643348094@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org>
pan3793
pushed a commit
that referenced
this pull request
Nov 1, 2023
… JDBC engine ### _Why are the changes needed?_ Similar to #3885, there is also memory leak in the jdbc engine when using incremental collect mode. Duplicate of #5161 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5570 from wForget/hotfix. Closes #5570 dbeddc9 [wforget] comment 0c4e499 [wforget] fix b7c421a [wforget] fix 0ef21ce [wforget] fix f5f2e80 [wforget] Fix memory leak when using incremental collect mode in JDBC engine Authored-by: wforget <643348094@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> (cherry picked from commit 0a0088b) Signed-off-by: Cheng Pan <chengpan@apache.org>
Contributor
Author
|
Closing this PR, as #5570 is merged for fixing the same issue. |
YesOrNo828
pushed a commit
to YesOrNo828/kyuubi
that referenced
this pull request
Nov 6, 2023
…ode in JDBC engine ### _Why are the changes needed?_ Similar to apache#3885, there is also memory leak in the jdbc engine when using incremental collect mode. Duplicate of apache#5161 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes apache#5570 from wForget/hotfix. Closes apache#5570 dbeddc9 [wforget] comment 0c4e499 [wforget] fix b7c421a [wforget] fix 0ef21ce [wforget] fix f5f2e80 [wforget] Fix memory leak when using incremental collect mode in JDBC engine Authored-by: wforget <643348094@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
toIterablecreates a stream byStream.cons(self.next(), self.toStream)which holds iterated rows of resultset in JDBC engineHow was this patch tested?
Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before make a pull request