Python ResultsReader class is incredibly slow

[mew1033]

It's possible that I don't understand what all the ResultsReader class is doing, so take this issue with a grain of salt.

When using the ResultsReader class to get results from a Splunk search, as indicated here: https://github.com/splunk/splunk-sdk-python/blob/master/splunklib/results.py#L173-L181, it takes an incredibly long time to get all the results. Using the jobs.results function is orders of magnitude faster.

For example, on a search with 175k results, it takes 4+ minutes to get the results with ResultsReader objects, and 3.7 seconds with the results function. The following snippet shows what I'm talking about:

import splunklib.results as results
import splunklib.client as client
from datetime import datetime
import json


splunk_object = client.connect(
    host="host",
    port="port",
    username="username",
    password="password",
    app="app",
    verify=True,
    autologin=True)

spl = '| makeresults count=175000'

splunk_search_kwargs = {"exec_mode": "blocking",
                        "earliest_time": "-48h",
                        "latest_time": "now",
                        "enable_lookups": "true"}

splunk_search_job = splunk_object.jobs.create(spl, **splunk_search_kwargs)


start_time_json = datetime.now()
# Get the results from the Splunk search
search_results_json = []
# log_general.debug("Getting Splunk search results.")
get_offset = 0
max_get = 49000
result_count = int(splunk_search_job['resultCount'])
while (get_offset < result_count):
    r = splunk_search_job.results(**{"count": max_get, "offset": get_offset, "output_mode": "json"})
    obj = json.loads(r.read())
    search_results_json.extend(obj['results'])
    get_offset += max_get
# log_general.debug("Found %d results" % len(search_results))

end_time_json = datetime.now()


start_time = datetime.now()
# Get the results from the Splunk search
search_results = []
# log_general.debug("Getting Splunk search results.")
get_offset = 0
max_get = 49000
result_count = int(splunk_search_job['resultCount'])
while (get_offset < result_count):
    rr = results.ResultsReader(splunk_search_job.results(**{"count": max_get, "offset": get_offset}))
    for result in rr:
        if isinstance(result, results.Message):
            # Diagnostic messages may be returned in the results
            print '%s: %s' % (result.type, result.message)
        elif isinstance(result, dict):
            # Normal events are returned as dicts
            search_results.append(result)
    get_offset += max_get
# log_general.debug("Found %d results" % len(search_results))

end_time = datetime.now()

print ("ResultsReader time: %s" % (end_time-start_time).seconds)
print ("json_results time: %s" % (end_time_json-start_time_json).seconds)

Is ResultsReader doing anything special that I miss out on by just getting the results is json mode directly? I know that ResultsReader uses XML under the hood, but that doesn't really matter to me; at the end of the day, I just need the results in a python object.

[tdhellmann]

@mew1033 - thanks for bringing this to our attention and thanks for the snippet demonstrating your issue! We'll take a look.

[dveuve]

Great suggestion @mew1033 . Ran into the same issue pulling back ~12k results. My entire script run (I'm less thorough than you) took 74 seconds using the ResultsReader, and 3.6 seconds with results directly.

[delewis13]

Ran into this issue pulling 15k results. ResultsReader v slow [4 minutes], straight JSON ~5 seconds.

Ditto same issue, plus my Mac's CPU goes to 98-99% every time on the ResultsReader call.

Workarounds:

1. Use job.export instead of job.create or job.oneshot
2. Use a custom streaming wrapper in conjunction with io.BufferedReader

Using both of these improved my performance 100x+
Details HERE

IMHO the architectural design as described in the link is terrible. Splunk seems to constantly want to protect it users from themselves which is ok in a GUI but terrible in an API where engineers presumably know what they are doing.

[randomgambit]

Hello there! thanks for these helpful tips... I am having the same issue and I am not quite sure how to use the proposed workaround results.ResultsReader(io.BufferedReader(ResponseReaderWrapper(rs))) mentioned above.

I am using a simple service.jobs.export() and my goal is to actually save the data to a dataframe. At the moment I am using

exportsearch_results = service.jobs.export(searchquery, **kwargs_export)
reader = results.ResultsReader(exportsearch_results)    

thislist = []
for item in reader:
 thislist.append(item)

and then I create the dataframe using the list. This is quite slow.

Any suggestion greatly appreciated.
Thanks!!

[vdineshTM]

@randomgambit I am having the same performance problem as well. I am having a similar setup where I have service job export and reading result from ResultsReader. I need to write all those records into the file.

It takes 10 hours to process 1 Million records. ridiculously slow.

Can someone offer a solution to this issue pls?

[mew1033]

My solution is to just not use ResultsReader at all. Just use the raw .results method and grab it as json. Something like this:

import json

spl = 'your search goes here'

splunk_search_kwargs = {"exec_mode": "blocking",
                        "earliest_time": "-48h",
                        "latest_time": "now",
                        "enable_lookups": "true"}

splunk_search_job = splunk_object.jobs.create(spl, **splunk_search_kwargs)
search_results_json = []
get_offset = 0
max_get = 49000
result_count = int(splunk_search_job['resultCount'])
while (get_offset < result_count):
    r = splunk_search_job.results(**{"count": max_get, "offset": get_offset, "output_mode": "json"})
    obj = json.loads(r.read())
    search_results_json.extend(obj['results'])
    get_offset += max_get

[ashah-splunk]

@mew1033 we have introduced a JSONResultsReader in our latest Python SDK release 1.6.19, it has a better performance as compared to the existing ResultsReader. We would suggest you to use the new feature available and let us know if that improves performance for your application.
@vdineshTM , @randomgambit and @bretlowery - we would suggest you all as well to utilise the JSONResultsReader in your applications and let us know if in case of any issues or it improves performance in your applications
Note:- JSONResultsReader works along with the query param "output_mode" set to 'json'.

[qinghuan1998]

我的解决方案是根本不使用ResultsReader。只需使用原始.results方法并将其作为 json 抓取即可。像这样的东西：

import json

spl = 'your search goes here'

splunk_search_kwargs = {"exec_mode": "blocking",
                        "earliest_time": "-48h",
                        "latest_time": "now",
                        "enable_lookups": "true"}

splunk_search_job = splunk_object.jobs.create(spl, **splunk_search_kwargs)
search_results_json = []
get_offset = 0
max_get = 49000
result_count = int(splunk_search_job['resultCount'])
while (get_offset < result_count):
    r = splunk_search_job.results(**{"count": max_get, "offset": get_offset, "output_mode": "json"})
    obj = json.loads(r.read())
    search_results_json.extend(obj['results'])
    get_offset += max_get

Thank you for solving my problem, but I put the returned result directly into the for loop:

            for item in results.JSONResultsReader(job_result):
                result_list.append(item)

However, i can only get up to 50,000 pieces of data. After seeing your code, I found that you can adjust the results of different segments by passing in parameters such as offset (the official document does not give detailed instructions).