Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[C++] DatasetWriter cleanup resource when not able to write #38884

Closed
mapleFU opened this issue Nov 26, 2023 · 0 comments · Fixed by #38885
Closed

[C++] DatasetWriter cleanup resource when not able to write #38884

mapleFU opened this issue Nov 26, 2023 · 0 comments · Fixed by #38885
Assignees
@mapleFU
Labels
Component: C++ Type: enhancement
Milestone
15.0.0

Comments

@mapleFU
Copy link
Member

mapleFU commented Nov 26, 2023 •
edited
Loading

Describe the enhancement requested

The DatasetWriterFileQueue::WriteNext will write-batch and Release the chunk-size. However, if the file-size is too large, DoWriteRecordBatch will failed to open file or close a file, which causing chunk-size resource leak, causing the deadlock

Component(s)

C++
@github-actions github-actions bot mentioned this issue Nov 26, 2023
Merged
@kou kou closed this as completed in #38885 Dec 7, 2023
@kou kou added this to the 15.0.0 milestone Dec 7, 2023
kou pushed a commit that referenced this issue Dec 7, 2023
@mapleFU
GH-38884: [C++] DatasetWriter release rows_in_flight_throttle when al…
f7286a9 
…locate writing failed (#38885)

### Rationale for this change

When file-queue is fall or write failed, the `DatasetWriterImpl::DoWriteRecordBatch` might failed, however, the resources are not released.

### What changes are included in this PR?

When file-queue is full or cannot open file, release the `row` resources.

### Are these changes tested?

yes

### Are there any user-facing changes?

no

* Closes: #38884

Authored-by: mwish <maplewish117@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@westonpace westonpace mentioned this issue Dec 27, 2023
Merged
dgreiss pushed a commit to dgreiss/arrow that referenced this issue Feb 19, 2024
@mapleFU @dgreiss
apacheGH-38884: [C++] DatasetWriter release rows_in_flight_throttle w…
1488f16 
…hen allocate writing failed (apache#38885)

### Rationale for this change

When file-queue is fall or write failed, the `DatasetWriterImpl::DoWriteRecordBatch` might failed, however, the resources are not released.

### What changes are included in this PR?

When file-queue is full or cannot open file, release the `row` resources.

### Are these changes tested?

yes

### Are there any user-facing changes?

no

* Closes: apache#38884

Authored-by: mwish <maplewish117@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mapleFU mapleFU

Labels
Component: C++ Type: enhancement
Projects
None yet
Milestone
15.0.0
2 participants
@kou @mapleFU