Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to process parallel requests over the same session #150

Open
mirceaulinic opened this issue Aug 29, 2017 · 12 comments
Open

Unable to process parallel requests over the same session #150

mirceaulinic opened this issue Aug 29, 2017 · 12 comments

Comments

@mirceaulinic
Copy link

When issuing two concomitant requests over the same eAPI session I can get the following errors:

  File "/usr/local/lib/python2.7/dist-packages/pyeapi/client.py", line 730, in run_commands
    response = self._connection.execute(commands, encoding, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/eapilib.py", line 495, in execute
    response = self.send(request)
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/eapilib.py", line 388, in send
    response = self.transport.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1099, in getresponse
    raise ResponseNotReady()
ResponseNotReady
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/client.py", line 730, in run_commands
    response = self._connection.execute(commands, encoding, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/eapilib.py", line 495, in execute
    response = self.send(request)
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/eapilib.py", line 372, in send
    self.transport.putrequest('POST', '/command-api')
  File "/usr/lib/python2.7/httplib.py", line 921, in putrequest
    raise CannotSendRequest()
CannotSendRequest

or

  File "/usr/local/lib/python2.7/dist-packages/pyeapi/client.py", line 730, in run_commands
    response = self._connection.execute(commands, encoding, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/eapilib.py", line 495, in execute
    response = self.send(request)
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/eapilib.py", line 392, in send
    response_content = response.read()
  File "/usr/lib/python2.7/httplib.py", line 578, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/httplib.py", line 636, in _read_chunked
    value.append(self._safe_read(chunk_left))
  File "/usr/lib/python2.7/httplib.py", line 693, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
AttributeError: 'NoneType' object has no attribute 'read'

For the moment I would only like to understand if this is known, desired or should be corrected.

Thanks!
@mirceaulinic
Copy link
Author

mirceaulinic commented Aug 29, 2017 •
edited

Just to provide an example with logs. Starting two parallel jobs using the same pyeAPI session: one that applies a configuration change by creating a new configuration session, the other one invokes the show arp command:

2017-08-29 16:54:48,324 [pyeapi.eapilib   ][DEBUG   ][23270] Request content: {"params": {"format": "json", "version": 1, "cmds": ["enable", "show configuration sessions"]}, "jsonrpc": "2.0", "method": "runCmds", "id": "140371987725072"}
2017-08-29 16:54:49,063 [pyeapi.eapilib   ][DEBUG   ][23270] Response: status:200, reason:OK
2017-08-29 16:54:49,063 [pyeapi.eapilib   ][DEBUG   ][23270] Response content: {"jsonrpc": "2.0", "result": [{}, {"maxSavedSessions": 1, "maxOpenSessions": 5, "sessions": {}}], "id": "140371987725072"}
2017-08-29 16:54:49,064 [pyeapi.eapilib   ][DEBUG   ][23270] eapi_response: {u'jsonrpc': u'2.0', u'result': [{}, {u'maxSavedSessions': 1, u'maxOpenSessions': 5, u'sessions': {}}], u'id': u'140371987725072'}
2017-08-29 16:54:49,064 [pyeapi.eapilib   ][DEBUG   ][23270] Request content: {"params": {"format": "json", "version": 1, "cmds": ["enable", "configure session napalm_324721", "ntp server 1.2.3.4"]}, "jsonrpc": "2.0", "method": "runCmds", "id": "140371987725072"}
2017-08-29 16:54:49,704 [pyeapi.eapilib   ][DEBUG   ][23270] Request content: {"params": {"format": "json", "version": 1, "cmds": ["enable", "show arp"]}, "jsonrpc": "2.0", "method": "runCmds", "id": "140371987725072"}
2017-08-29 16:54:49,704 [/usr/lib/python2.7/dist-packages/salt/utils/napalm.pyc][ERROR   ][23270] Cannot execute "get_arp_table" on edge01.bjm01 as username. Reason: !
2017-08-29 16:54:49,704 [/usr/lib/python2.7/dist-packages/salt/utils/napalm.pyc][ERROR   ][23270] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/napalm.py", line 155, in call
    out = getattr(napalm_device.get('DRIVER'), method)(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/napalm_eos/eos.py", line 801, in get_arp_table
    ipv4_neighbors = self.device.run_commands(commands)[0].get('ipV4Neighbors', [])
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/client.py", line 730, in run_commands
    response = self._connection.execute(commands, encoding, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/eapilib.py", line 495, in execute
    response = self.send(request)
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/eapilib.py", line 372, in send
    self.transport.putrequest('POST', '/command-api')
  File "/usr/lib/python2.7/httplib.py", line 921, in putrequest
    raise CannotSendRequest()
CannotSendRequest

2017-08-29 16:54:50,195 [pyeapi.eapilib   ][DEBUG   ][23270] Response: status:200, reason:OK
2017-08-29 16:54:50,196 [pyeapi.eapilib   ][DEBUG   ][23270] Response content: {"jsonrpc": "2.0", "result": [{}, {}, {}], "id": "140371987725072"}
2017-08-29 16:54:50,196 [pyeapi.eapilib   ][DEBUG   ][23270] eapi_response: {u'jsonrpc': u'2.0', u'result': [{}, {}, {}], u'id': u'140371987725072'}
2017-08-29 16:54:50,196 [pyeapi.eapilib   ][DEBUG   ][23270] Request content: {"params": {"format": "text", "version": 1, "cmds": ["enable", "show session-config named napalm_324721 diffs"]}, "jsonrpc": "2.0", "method": "runCmds", "id": "140371987725072"}
2017-08-29 16:55:03,757 [pyeapi.eapilib   ][DEBUG   ][23270] Response: status:200, reason:OK

The first one continues as normal, while the other one raises CannotSendRequest.

@mharista
Copy link
Contributor

mharista commented Sep 1, 2017

Hi @mirceaulinic,

Is this being done with parallel threads or processes that are both using the same session and simultaneously sending commands? I believe the requests package used for the session in pyeapi is blocking until a requests entire response has been downloaded.

More info here

@mirceaulinic
Copy link
Author

Hi @mharista - thanks for looking into this issue.
Yes, this is from parallel processes (same behaviour with multiple threads) sharing the same session.
I went through the resource you pointed out, but I couldn't find anything similar, or describing such a limitation. Also, the errors I am facing, don't suggest they are blocking till the previous response has been downloaded, but simply dropping the a request that arrives the latter. Is this intended by design?

@mirceaulinic
Copy link
Author

I am also a bit confused you mentioned the requests library although the traceback doesn't show like pyeapi uses it (requirements.txt confirms that as well). Can you please clarify?

@mharista
Copy link
Contributor

mharista commented Sep 1, 2017

@mirceaulinic My mistake. I assumed pyeapi was using the requests module because of other libraries we maintain that do use it, but pyeapi actually uses httplib. Sorry for causing unnecessary confusion. The error you are hitting is similar to what I mentioned previously. The second thread is likely making another request on the session before the original thread has made a call to getresponse(). I believe this is intended by design in python's httplib module and inherited by pyeapi.

@mirceaulinic
Copy link
Author

mirceaulinic commented Sep 1, 2017 •
edited

Thanks for confirming @mharista.

AFAIK using requests we shouldn't see anything like that, but I need to double check. Before starting to work on anything like that, would you welcome a PR to improve this side and switch from httplib to requests?

@mharista
Copy link
Contributor

mharista commented Sep 1, 2017

@mirceaulinic from the small amount I've looked into this, the library recommended for asynchronous requests is grequests. The link I provided previously mentions this.

Blocking Or Non-Blocking?
With the default Transport Adapter in place, Requests does not provide any kind of non-blocking IO. The Response.content property will block until the entire response has been downloaded. If you require more granularity, the streaming features of the library (see Streaming Requests) allow you to retrieve smaller quantities of the response at a time. However, these calls will still block.

If you are concerned about the use of blocking IO, there are lots of projects out there that combine Requests with one of Python's asynchronicity frameworks. Two excellent examples are grequests and requests-futures.

As for a PR to support handling of async requests, I am open to that as long as all the existing functionality is maintained and none of the existing unit/system tests are broken.

@mirceaulinic
Copy link
Author

mirceaulinic commented Sep 1, 2017 via email

@mharista
Copy link
Contributor

mharista commented Sep 1, 2017

@mirceaulinic is the question relating to the requests module, httplib or pyeapi? Pyeapi uses httplib which by design doesn't support async requests at the moment.

@mirceaulinic
Copy link
Author

Hi @mharista - my question is related to pyeapi: choosing httplib in favour of other solutions is a design choice.

@mharista
Copy link
Contributor

mharista commented Sep 5, 2017

@mirceaulinic Could you please contact me via email? mhartzel@arista.com

@mharista
Copy link
Contributor

Hi @mirceaulinic, Is this still an issue you would like to discuss?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mirceaulinic @mharista