New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix slow CSV export by using a bigger default batch size #5304

Merged
merged 1 commit into from Nov 21, 2018

Conversation

Projects
None yet
3 participants
@bernd
Member

bernd commented Nov 21, 2018

The default batch size in Elasticsearch is 10. We now use 500 as default
for the newly introduced "batch_size" parameter which we pass to the
search request as limit. (which is the batch size for scroll requests)

Previously we used the "limit" parameter for the batch size AND the
result limit. That was wrong and resulted in truncated results.

Fixes #5172

(cherry picked from commit a77756d)

Fix slow CSV export by using a bigger default batch size (#5289)
The default batch size in Elasticsearch is 10. We now use 500 as default
for the newly introduced "batch_size" parameter which we pass to the
search request as limit. (which is the batch size for scroll requests)

Previously we used the "limit" parameter for the batch size AND the
result limit. That was wrong and resulted in truncated results.

Fixes #5172

(cherry picked from commit a77756d)

@bernd bernd added this to the 2.5.0 milestone Nov 21, 2018

@bernd bernd requested a review from edmundoa Nov 21, 2018

@edmundoa edmundoa merged commit 31f32a8 into 2.5 Nov 21, 2018

2 of 3 checks passed

ci-web-linter Jenkins build graylog-pr-linter-check 2987 has failed
Details
graylog-project/pr Jenkins build graylog-project-pr-snapshot 2241 has succeeded
Details
license/cla Contributor License Agreement is signed.
Details

@edmundoa edmundoa deleted the issue-5172-2.5 branch Nov 21, 2018

@keaoner

This comment has been minimized.

keaoner commented Dec 10, 2018

Hi all

The parameter "batch_size" is to put in the url or in the configuration file?

Thx u!

@edmundoa

This comment has been minimized.

Member

edmundoa commented Dec 10, 2018

Hi @keaoner, in Graylog 2.5.0 we started using a larger batch_size so you most likely don't need to do anything if you are using that version. In case you really want to override the default, you need to pass the batch_size parameter to the API request.

In case you need more information about it, please open a topic in our discussion forum or ask in the #graylog channel on freenode IRC.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment