Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BulkProcessor's close ignores in-flight bulkRequests #6314

Closed
btiernay opened this Issue May 26, 2014 · 3 comments

Comments

Projects
None yet
3 participants
@btiernay
Copy link

btiernay commented May 26, 2014

I have observed on a number of occasions that when I have large batch sizes / concurrent requests that I don't always get the same number of documents out of ES that I put in. Looking at:

https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/action/bulk/BulkProcessor.java#L204

    /**
     * Closes the processor. If flushing by time is enabled, then its shutdown. Any remaining bulk actions are flushed.
     */
    public synchronized void close() {
        if (closed) {
            return;
        }
        closed = true;
        if (this.scheduledFuture != null) {
            this.scheduledFuture.cancel(false);
            this.scheduler.shutdown();
        }
        if (bulkRequest.numberOfActions() > 0) {
            execute();
        }
    }

What happens if there are concurrent requests in-flight before the close call is made? Shouldn't this method block on those finishing? Or is the requirement / expectation that the client call close
on TransportClient which will block on the in-flight requests for up to 10 seconds?

As far as I can tell, there is no reliable way to know it is safe to shutdown the JVM.

@btiernay btiernay changed the title BulkProcessor silently drops last batch BulkProcessor's close ignores in-flight bulkRequests May 26, 2014

@btiernay

This comment has been minimized.

Copy link
Author

btiernay commented May 27, 2014

This may be related but seems to concern error checking: #4301

@btiernay

This comment has been minimized.

Copy link
Author

btiernay commented May 27, 2014

This appears to be a symptom of the missing functionality described in #4158. Is there a client side workaround given #5038?

@javanna

This comment has been minimized.

Copy link
Member

javanna commented Jun 13, 2014

Hi @btiernay, #5038 is going to be fixed soon (see #6495 ), also I think it would be good to add a blocking variant of the close method, that tries and wait for all the in-flight bulk requests to be completed #4180 is just a merge away and achieves exactly this ;)

Does this address your concerns?

@javanna javanna self-assigned this Jun 13, 2014

@s1monw s1monw added bug labels Jul 17, 2014

@s1monw s1monw assigned s1monw and unassigned javanna Jul 17, 2014

s1monw added a commit that referenced this issue Jul 17, 2014

[Bulk] Add blocking close method to BulkProcessor
Blocks until all bulk requests have completed.

Closes #4158
Closes #6314

@s1monw s1monw closed this in 42b71a0 Jul 17, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.