XML parse error with "Connection: keep-alive"

[mic159]

The python splunk SDK currently breaks when the server uses "Connection: keep-alive".

The issue is in splunklib.binding:handler, in the request method, you get the response object, then call close on the connection, then try to read the data.

When the server is using keep-alive, httplib will give you an empty response after you call connection.close().

We currently have NginX running in front of Splunk, which was changing it to keep-alive.

I'm using python2.7 on ubuntu 13.04 and ubuntu 12.04.
The error I get is:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/aaa/.virtualenvs/splunk/local/lib/python2.7/site-packages/splunklib/client.py", line 288, in connect
    return Service(**kwargs).login()
  File "/home/aaa/.virtualenvs/splunk/local/lib/python2.7/site-packages/splunklib/binding.py", line 753, in login
    session = XML(body).findtext("./sessionKey")
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1302, in XML
    return parser.close()
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1655, in close
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1507, in _raiseerror
    raise err
xml.etree.ElementTree.ParseError: no element found: line 1, column 0

[madhadron]

Huh. Okay, let me put this in our internal bug tracker and we'll have a look at it.

[e0d]

I'm seeing the same issue.

I'm using the downloadable zip with version b33b546 and have set up a .splunkrc and verified that the credentials are loaded properly by python.

When running oneshot.py I'm getting the following error:

Traceback (most recent call last):
  File "/home/edward/Applications/pycharm-2.7/helpers/pydev/pydevd.py", line 1481, in <module>
    debugger.run(setup['file'], None, None)
  File "/home/edward/Applications/pycharm-2.7/helpers/pydev/pydevd.py", line 1124, in run
    pydev_imports.execfile(file, globals, locals) #execute the script
  File "/home/edward/Applications/splunk-splunk-sdk-python-b33b546/examples/oneshot.py", line 54, in <module>
    main()
  File "/home/edward/Applications/splunk-splunk-sdk-python-b33b546/examples/oneshot.py", line 47, in main
    service = connect(**opts.kwargs)
  File "/home/edward/Applications/splunk-splunk-sdk-python-b33b546/examples/../splunklib/client.py", line 288, in connect
    return Service(**kwargs).login()
  File "/home/edward/Applications/splunk-splunk-sdk-python-b33b546/examples/../splunklib/binding.py", line 754, in login
    session = XML(body).findtext("./sessionKey")
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1302, in XML
    return parser.close()
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1655, in close
    self._raiseerror(v)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1507, in _raiseerror
    raise err
xml.etree.ElementTree.ParseError: no element found: line 1, column 0

Stepping through the code in my debugger I see that I'm getting a 200 HTTP response around here: /splunklib/binding.py:753

{'status': 200, 'headers': [('date', 'Tue, 18 Jun 2013 14:39:42 GMT'), ('content-length', '83'), ('content-type', 'text/xml; charset=utf-8'), ('connection', 'keep-alive'), ('server', 'Splunkd')], 'reason': 'OK', 'body': <splunklib.binding.ResponseReader object at 0x19fb6d0>}

The content-length corresponds to what I expect the size of the response body to be.

For example, if I run:

curl -k https://myserver.example.org:443/services/auth/login -d'username=foo&password=bar' > response.out && du -b response.out

> 83    response.out

That response looks correct as well.

<response>
  <sessionKey>5a58f8f133e59c8e11aed463c1745328</sessionKey>
</response>

[mic159]

Yes, this is because whatever is reverse proxying Splunk is changing the "Connection" header to keep-alive.
A workaround is to make your reverse proxy to not use keep-alive.

If you are using NginX, you can disable it by setting keepalive_timeout 0; in your NginX configuration.

[e0d]

Yeah, in my case I have an AWS ELB in front of splunk, so I have fewer configuration options. Seems like the SDK should support keep-alive.

[kelvinn]

I installed Splunk on OpenShift and ran into this issue, or at least something very similar. Initially I thought it was a problem with OpenShift, but direct REST requests worked without issue. Even in Python using urllib worked fine, which indicated an issue with the SDK.

Did you ever implement a workaround? I have changed line 1200 of bindings.py to look like the below, and the SDK now connections without issue. I will suggest this as a fix unless somebody comes up with a more elegant solution?

conn_header = response.getheader('Connection', None)
if conn_header is "Close":
    connection.close()

[glennblock]

Hi @kelvinn

I am going to file a bug for us to fix this. Thanks for investigating!

[stelles]

After reviewing this issue, we've found that it's actually a bug in the Python Library.

http://bugs.python.org/issue3566

There are workarounds, for example @kelvinn posted one that works as well as changing a middleman program to not use "connection: Keep-Alive".

We're not going to implement a fix considering this is a bug in the Python language as well as we're only going to support connection handling from the Splunk server, not middleman programs.

Sorry for the inconvenience!