CSV Export issue #4190
CSV Export issue #4190
Comments
@vladostp Please attach the contents of the "index_ranges" collection in MongoDB (e. g. using mongoexport) and some example requests including their complete response which show the problem. |
I am really sorry but I can't post any responses of requests cause it contains some sensitive data. But I tried different type of searches in different sets of indices and the behaviour was always the same:
|
You can of course redact the sensitive parts of the requests and responses in a sensible (and consistent) way, but without you providing any details we'll be unable to reproduce the issue and thus won't be able to help you. |
|
@vladostp Okay, that's the literal CSV file. Could you try posting the complete HTTP requests and responses (without the response body) when exporting these? You can find them in the Developer Console of your web browser (or just post the output of curl if you're using that). Also take a look in the logs of your Graylog nodes and check them for useful information. |
Good Request:
Good Response:
Bad Request:
Bad Response:
|
We're seeing similar, requesting the CSV download through the web interface - for me it's the transition for 15 to 16 indices in the query that causes a 10 record CSV to be returned rather than the 10s of thousands expected. |
There definitely does seem to be a problem when the scroll_id gets too large and it switches to POST rather than GET.
So, scroll is bigger than 1900 so it switches over to POST via: https://github.com/searchbox-io/Jest/blob/v5.3.3/jest-common/src/main/java/io/searchbox/core/SearchScroll.java#L15
That post body needs to be a JSON object like it is on new copies of jest. My guess is that the code we're pulling in is this: https://github.com/searchbox-io/Jest/blob/v2.4.0/jest-common/src/main/java/io/searchbox/core/SearchScroll.java#L24 EDIT: I've confirmed that replacing the class with one that supports the JSON encoding does indeed fix this issue for us. |
Can confirm -- we are seeing the same thing in our installation after the upgrade to 2.3.1. Haven't tested how many indices are int he query before this breaks down, but our exports only have 10 lines as well. |
Refs Graylog2/graylog2-server#4190 (cherry picked from commit 2a34a76)
Refs Graylog2/graylog2-server#4190 (cherry picked from commit 2a34a76)
Scroll queries were broken for a number of different reasons: * The fork of Jest used by Graylog had a bug regarding scroll queries (searchbox-io/Jest#489, searchbox-io/Jest#491) * The Elasticsearch Multi Search API doesn't support scroll queries (elastic/elasticsearch#18454) Due to the default HTTP request length limit of Elasticsearch, `Search#scroll()` resolves the affected index sets and uses the index wildcards instead of separate index names. Fixes #4190
Scroll queries were broken for a number of different reasons: * The fork of Jest used by Graylog had a bug regarding scroll queries (searchbox-io/Jest#489, searchbox-io/Jest#491) * The Elasticsearch Multi Search API doesn't support scroll queries (elastic/elasticsearch#18454) Due to the default HTTP request length limit of Elasticsearch, `Search#scroll()` resolves the affected index sets and uses the index wildcards instead of separate index names. Fixes #4190
Scroll queries were broken for a number of different reasons: * The fork of Jest used by Graylog had a bug regarding scroll queries (searchbox-io/Jest#489, searchbox-io/Jest#491) * The Elasticsearch Multi Search API doesn't support scroll queries (elastic/elasticsearch#18454) Due to the default HTTP request length limit of Elasticsearch, `Search#scroll()` resolves the affected index sets and uses the index wildcards instead of separate index names. Fixes #4190 (cherry picked from commit 7239f15)
Scroll queries were broken for a number of different reasons: * The fork of Jest used by Graylog had a bug regarding scroll queries (searchbox-io/Jest#489, searchbox-io/Jest#491) * The Elasticsearch Multi Search API doesn't support scroll queries (elastic/elasticsearch#18454) Due to the default HTTP request length limit of Elasticsearch, `Search#scroll()` resolves the affected index sets and uses the index wildcards instead of separate index names. Fixes #4190 (cherry picked from commit 7239f15)
Scroll queries were broken for a number of different reasons: * The fork of Jest used by Graylog had a bug regarding scroll queries (searchbox-io/Jest#489, searchbox-io/Jest#491) * The Elasticsearch Multi Search API doesn't support scroll queries (elastic/elasticsearch#18454) Due to the default HTTP request length limit of Elasticsearch, `Search#scroll()` resolves the affected index sets and uses the index wildcards instead of separate index names. Fixes #4190 (cherry picked from commit 7239f15)
The CSV export fails when I am trying to export a search result from more than 7 indices.
Expected Behavior
The exported CSV files contains the search result data.
Current Behavior
The exported CSV file contains only 10 lines
Possible Solution
Steps to Reproduce (for bugs)
Context
I am trying to export data to CSV file
Your Environment
The text was updated successfully, but these errors were encountered: