ConnectionError: Socket error during eAPI connection: The read operation timed out

[lvrfrc87]

Please, note that this BUG s related to: napalm-automation/napalm-ansible#165 (comment)

I am using NAPALM to perform config-replace on Arista DCS-7060SX2-48YC6-R version 4.21.8M-13902577.4218M.with pyeapi==0.8.3 The session times out no matter what timeout I configure, either http and https. I can use the library ok for other things like send commands.

I can see the HTTP session up and the configure session created, however no packets are sent/received and configure session remains in pending

   User              Requests       Bytes in       Bytes out    Last hit
----------------- -------------- -------------- --------------- --------------
   prd2204.svc       456            530905         3856176      12 seconds ago

   User              Requests       Bytes in       Bytes out    Last hit
----------------- -------------- -------------- --------------- ---------------
   prd2204.svc       456            530905         3856176      171 seconds ago

l1a.r5b1.ams7#show configuration sessions
Maximum number of completed sessions: 1
Maximum number of pending sessions: 5

  Name             State         User                 Terminal
  ------------- ------------- ----------------------- -----------
  napalm_101518    pending        prd2204.svc    command-api

The read operation timed out
Traceback (most recent call last):
  File "/Users/federicoolivieri/git/NaC/lib/python3.7/site-packages/pyeapi/eapilib.py", line 443, in send
    response_content = response.read()
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 454, in read
    return self._readall_chunked()
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 561, in _readall_chunked
    chunk_left = self._get_chunk_left()
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 544, in _get_chunk_left
    chunk_left = self._read_next_chunk_size()
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 504, in _read_next_chunk_size
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1049, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 908, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
Traceback (most recent call last):
  File "/Users/federicoolivieri/git/NaC/lib/python3.7/site-packages/pyeapi/eapilib.py", line 443, in send
    response_content = response.read()
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 454, in read
    return self._readall_chunked()
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 561, in _readall_chunked
    chunk_left = self._get_chunk_left()
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 544, in _get_chunk_left
    chunk_left = self._read_next_chunk_size()
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 504, in _read_next_chunk_size
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1049, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 908, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/federicoolivieri/git/NaC/lib/python3.7/site-packages/napalm/eos/eos.py", line 297, in load_replace_candidate
    self._load_config(filename, config, True)
  File "/Users/federicoolivieri/git/NaC/lib/python3.7/site-packages/napalm/eos/eos.py", line 286, in _load_config
    self.device.run_commands(commands)
  File "/Users/federicoolivieri/git/NaC/lib/python3.7/site-packages/pyeapi/client.py", line 743, in run_commands
    response = self._connection.execute(commands, encoding, **kwargs)
  File "/Users/federicoolivieri/git/NaC/lib/python3.7/site-packages/pyeapi/eapilib.py", line 550, in execute
    response = self.send(request)
  File "/Users/federicoolivieri/git/NaC/lib/python3.7/site-packages/pyeapi/eapilib.py", line 479, in send
    raise ConnectionError(str(self), error_msg)
pyeapi.eapilib.ConnectionError: Socket error during eAPI connection: The read operation timed out

[andriirr]

Federico, going to take a look at this issue.
Do you mind passing me configuration of targeted arista switch?

[lvrfrc87]

Sure! Which part of config? management API?

[andriirr]

Running config would be a preferred option

[andriirr]

andrii@arista.com to keep it private :)

[lvrfrc87]

Sent. :)

[andriirr]

Napalm playbook worked with no issue until I enabled authorisation via TACACS+.
The way how the validation works in this case, every single command has to be checked at the TACACS+ server.
This means intensive communication between the switch and server and it takes a lot of time.

In my lab, in order to validate your configuration ansible spent 3 min 55 sec.
I guess your thoughts around napalm timeouts were correct, just needed a bit more tweaking.
To back up my theory, you can inspect the flow via packet capture on the interface through which TACACS+ server is available and track the progress that way.

On the other hand, initial spin (w/o TACACS+) of your playbook executes in 23sec - I guess you might look into the options of either having local user just for napalm or disable authorisation for specific command type/users (if possible, although I didn't do a research on this one yet).