vpshere connection connect.SmartConnect does not persist for longer than 15 min inactive time.

[shyamachilles]

@tianhao64 , @hartsock

This issue is an offshoot of Issue #194 and #294 -----

My environment consists of a SAN attached to ESX host.

I am currently using pyvmomi 6.0 with Python 2.7.10 -

This is how am trying to connect to the vsphere web client (As per #294,)

import ssl
connect.SmartConnect(protocol="https",
host=self.fqdn,
user=self.username,
pwd=self.password,
sslContext=ssl._create_unverified_context())

the Test does below -

1. Clone "x" VMs and power them on , in a sequence. "x" is passed as an argument to the test.
2. Perform IO from each cloned VM to a common datastore provided to test as an argument.
3. Take Snapshots of each clones
4. Stop IO. ( this Step happens on the client host and vsphere is inactive for more than 15mins usually)
5. Power off VMS and delete them

I am consistently hitting failure in Step 5 , while trying to power off the first Vm. Typical stack trace looks like this -

File "/home/usr/banalas/139895/testSW/lib/ClientHostVMware.py", line 682, in vmPowerOff
dlog.debug("Powering off " + virtualMachine.name)
File "/home/usr/banalas/139895/testSW/lib/pyVmomi/VmomiSupport.py", line 560, in call
return self.f(args, *kwargs)
File "/home/usr/banalas/139895/testSW/lib/pyVmomi/VmomiSupport.py", line 383, in _InvokeAccessor
return self._stub.InvokeAccessor(self, info)
File "/home/usr/banalas/139895/testSW/lib/pyVmomi/StubAdapterAccessorImpl.py", line 47, in InvokeAccessor
raise objectContent.missingSet[0].fault
vim.fault.NotAuthenticated: (vim.fault.NotAuthenticated) {
dynamicType = ,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = '',
faultCause = ,
faultMessage = (vmodl.LocalizableMessage) [],
object = 'vim.VirtualMachine:vm-1050',
privilegeId = 'System.View'

I ve also seen pyvmomi/SoapAdapter.py having CONNECTION_POOL_IDLE_TIMEOUT_SEC =900 secs which is 15 mins.

Is this the reason for my consistent failure? Can I try using a new connection to the vsphere before the power off task? How do I ensure the vsphere connection instance can be tolerant of idle timeouts > 15 mins and not hit vim.fault.NotAuthenticated.

Please suggest how to resolve this issue.

[shyamachilles]

@tianhao64 Could you please help.

[mygoda]

i want to know too. but nobody help..

[frapposelli]

@vmware/pyvmomi-maintainers any chance of you guys taking a look at this?

[tianhao64]

Looks like the CONNECTION_POOL_IDLE_TIMEOUT_SEC is set to 900 secs in SoapAdapter.py. However, Connect/SmartConnect doesn't take the timeout parameter so the default is always used. One quick workaround can be changing the default timeout to a larger number or even disabled it by setting to -1: https://github.com/vmware/pyvmomi/blob/master/pyVmomi/SoapAdapter.py#L56

[svdgraaf]

I can confirm that raising the timeout fixes the issue. Would it be possible to implement something so that the timeout is configurable from a public method somewhere? If I can find the time, I will try to make a PR.

[hartsock]

@svdgraaf thanks!

[jwbennet]

Has there been any update on getting a fix for this issue?

[samuelsh]

Faced the same issue when started working on interactive cli tool for VM management,
While Cli is idle more than 15 minutes disconnection occurred, so when user got back and trying, for example list all VMs in folder, exception raised.
Of course I can run some background thread, which will "ping" host each x minutes as a workaround, but I guess better solution is to provide SmartConnect with timeout parameter.

[Huang-Jack]

Hi team,
do you merge the fix to 6.5 version? I used the pyVmomi6.5 still the issue and found the timeout is 900s.
CONNECTION_POOL_IDLE_TIMEOUT_SEC =900

[tianhao64]

Anyone can help to make the change? We need to add the CONNECTION_POOL_IDLE_TIMEOUT_SEC parameter to SmartConnect and pass the value all the way to SoapStubAdapter.

[magianxd]

I have added the parameter 'connectionPoolTimeout' in both SmartConnect and SmartConnectNoSSL. This problem is I cannot push my branch up to the repo. @tianhao64

[tianhao64]

@magianxd Thanks for the change. Can you create a pull request? I will merge the change if everything looks good.

[magianxd]

@tianhao64, I cannot create a new branch. Push returns 403 error.

[tianhao64]

@magianxd Can you fork the repo, push the change to your own repo and create a pull request?
https://help.github.com/articles/fork-a-repo/

[Huang-Jack]

looks good. Thanks @magianxd @tianhao64

[magianxd]

@tianhao64, there is still a bug in the code. connectionPoolTimeout is not passed to __RetrieveContent in __Login, I'm forking a new repo and submitting a PR

[magianxd]

File "C:\Users\xiaos3\PycharmProjects\pyvmomi\pyVmomi\VmomiSupport.py", line 574, in call
return self.f(args, **kwargs)
File "C:\Users\xiaos3\PycharmProjects\pyvmomi\pyVmomi\VmomiSupport.py", line 394, in _InvokeAccessor
return self._stub.InvokeAccessor(self, info)
File "C:\Users\xiaos3\PycharmProjects\pyvmomi\pyVmomi\StubAdapterAccessorImpl.py", line 42, in InvokeAccessor
options=self._pcType.RetrieveOptions(maxObjects=1))
File "C:\Users\xiaos3\PycharmProjects\pyvmomi\pyVmomi\VmomiSupport.py", line 580, in
self.f((self.args + (obj,) + args), **kwargs)
File "C:\Users\xiaos3\PycharmProjects\pyvmomi\pyVmomi\VmomiSupport.py", line 386, in _InvokeMethod
return self._stub.InvokeMethod(self, info, args)
File "C:\Users\xiaos3\PycharmProjects\pyvmomi\pyVmomi\SoapAdapter.py", line 1325, in InvokeMethod
resp = conn.getresponse()
File "C:\Python27\lib\httplib.py", line 1136, in getresponse
response.begin()
File "C:\Python27\lib\httplib.py", line 453, in begin
version, status, reason = self._read_status()
File "C:\Python27\lib\httplib.py", line 409, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "C:\Python27\lib\socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
File "C:\Python27\lib\ssl.py", line 756, in recv
return self.read(buflen)
File "C:\Python27\lib\ssl.py", line 643, in read
v = self._sslobj.read(len)
error: [Errno 10053] An established connection was aborted by the software in your host machine

@tianhao64, when I use the master branch code (PR #493 inlcuded) in my local development environment, the code encountered the above issue. Do you have any ideas? This issue occurs when accessing pyvmomi objects

[tianhao64]

@magianxd can you share your code?

[magianxd]

@tianhao64 I might find the root cause of this not authenticated issue. The connection timeout value, which is 900s, is not the root cause. The code will close idle connections after 900s. If there are no connections in the connection pool available, the code will create a new connection to send requests. Which means the code will always get response from vCenter.

Another point is that I still meet the Not Authenticated issue after PR #493 merged.

The root cause to Not Authentication issue should be too many sessions to vCenter. User should not use SmartConnect to get service instance each time. They should get service instance only once and check if the current session is timeout. If current session becomes inactive, logout and re-login.

So, the PR #493 should be rolled back since it will cause the connection closed unexpectedly issue. Or you can review a fix that I have made for this issue. Will let you know when PR is created

[tianhao64]

@magianxd You are right, It looks like the connectionPooltimeout parameter only close idle connections from the client side, and if the client connection was idle longer than the timeout value, a new connection will be created automatically.

How many connections you created using SmartConnect? I suspect your timeout issue is due to the server close the connection because the session times out. This is controlled on server side and is configurable via server settings. See https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2004663.

Another option is to try SessionOrientedStub which will re-login if the previous session times out.

I think PR #493 was valid. We just add the support to pass the customized connectionPooltimeout to SoapStubAdapter. Do you see any issues with it?

[magianxd]

@tianhao64. The issue is that if users pass negative numbers for connectionPoolTimeout, _CloseIdleConnections will not close any connection and all the requests sent to vCenter will use the initial connection. When the initial connection become invalid, the code will fail. I have added an else clause to force the code remove old connections when connectionPoolTimeout is set to negative numbers.

[magianxd]

@tianhao64 Do you have any example of using SessionOrientedStub. BTW, Can I configure vCenter to never timeout?

[tianhao64]

There is an example documented in the code:

pyvmomi/pyVmomi/SoapAdapter.py

Line 1530 in 912e365

class SessionOrientedStub(StubAdapterBase):

[tianhao64]

@magianxd I don't think you can make vcenter never timeout. Also it's probably not a good idea to do so.

[magianxd]

@tianhao64 Thanks. Do you mind if I add an argument named 'relogin' in SmartConnect and SmartConnectNoSSL to enable automatic re-login?

[tianhao64]

@magianxd Adding a keep-alive capability is probably better than re-login. re-login means we need to somehow cache the password and that might be considered as a security risk...

[tianhao64]

Here is the right KB article on how to increase the server session timeout: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2004663

[AnushaY1]

Hi,
pyvmomi 6.5 with Python 2.7
conn = SmartConnect(host=self.host, user=self.user,pwd=self.password , sslContext=context, connectionPoolTimeout = -1)

-never close the connection to vsphere until i use Disconnect(conn)i.e after 2 hrs i am trying to display vm details ,facing below error

[2017-04-25 15:31:31,135] ERROR in app: Exception on /vmlists [GET]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "/var/www/apptest/vsphereapi.py", line 34, in vmlists
content = conn.content
File "/usr/local/lib/python2.7/dist-packages/pyVmomi/VmomiSupport.py", line 574, in call
return self.f(args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pyVmomi/VmomiSupport.py", line 394, in _InvokeAccessor
return self._stub.InvokeAccessor(self, info)
File "/usr/local/lib/python2.7/dist-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor
options=self._pcType.RetrieveOptions(maxObjects=1))
File "/usr/local/lib/python2.7/dist-packages/pyVmomi/VmomiSupport.py", line 580, in
self.f((self.args + (obj,) + args), **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod
return self._stub.InvokeMethod(self, info, args)
File "/usr/local/lib/python2.7/dist-packages/pyVmomi/SoapAdapter.py", line 1323, in InvokeMethod
conn.request('POST', self.path, req, headers)
File "/usr/lib/python2.7/httplib.py", line 1057, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1097, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 1053, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 897, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 873, in send
self.sock.sendall(data)
File "/usr/lib/python2.7/ssl.py", line 743, in sendall
v = self.send(data[count:])
File "/usr/lib/python2.7/ssl.py", line 709, in send
v = self._sslobj.write(data)
error: [Errno 32] Broken pipe

it is small application when ever user logout that time only session logout will happen.(connection may be idle 2 or 5 or couple of hrs)

can anyone suggest how to solve this
thanks

[tianhao64]

@AnushaY1 Passing connectionPoolTimeout = -1 won't change session timeout. -1 only means client will never close a idle connection, it doesn't mean the connection will be valid forever. A soap request session will be timeout after 30min on server side, regardless whether or not the client closes the connection.

Session timeout is a common security best practice. Change the session timeout or workaround it can introduce security risk. That said, if increasing the session timeout is really needed in script, you have two options:

1. Increase timeout setting on server: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2004663

2. Have a separate thread to keep the session alive. For e.g. make a simple API request every 10 min or so.

Hope it helps!

[magianxd]

Here is a solution to resolve session timeout issue.

smart_stub = SmartStubAdapter(host=host, port=port, sslContext=ssl_context, connectionPoolTimeout=0)

session_stub = VimSessionOrientedStub(smart_stub, VimSessionOrientedStub.makeUserLoginMethod(username, password))

service_instance = vim.ServiceInstance('ServiceInstance', session_stub)

[AnushaY1]

working.thanks

[DavidDota]

@magianxd hello, if this way can keep session alive? what does it mean set CONNECTION_POOL_IDLE_TIMEOUT_SEC=0?