New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PEP 0466 - SSL - Changes consideration #212

Closed
hartsock opened this Issue Feb 10, 2015 · 22 comments

Comments

Projects
None yet
@hartsock
Member

hartsock commented Feb 10, 2015

As of Python 2.7.9 the default SSL behavior has changed (see PEP 0466). SSL certificates are verified in strict mode. Since vCenter's default beginner use case is to deploy with a self-signed certificate we may need to add some additional flags, settings, and warnings inside pyVmomi to alert about and accommodate this change.

  • We want new users to be able to just get going
  • We don't want to encourage weak SSL which is a security concern
  • We do want to inform a beginner user of what's going on and what to do about it
  • We want this to be as easy as possible without sacrificing security

NOTE: this may have already been covered, this issue is here as a note to force me to examine the issue again just before the next roll out of pyVmomi

@hartsock

This comment has been minimized.

Show comment
Hide comment
@hartsock

hartsock Feb 10, 2015

Member

So far I have been unable to find a way to override the Python interpreter-wide ssl.SSLContext.verify_mode so I am leaning toward making a patch to the library to allow users to pass in a set of flags to Connect and SmartConnect ... more on this later.

Edit:
Note the problem could be that we need to expose all of these flags ... upward from Connect and SmartConnect along with flags from the SoapStubAdapter and HTTPSConnectionWrapper classes.

Member

hartsock commented Feb 10, 2015

So far I have been unable to find a way to override the Python interpreter-wide ssl.SSLContext.verify_mode so I am leaning toward making a patch to the library to allow users to pass in a set of flags to Connect and SmartConnect ... more on this later.

Edit:
Note the problem could be that we need to expose all of these flags ... upward from Connect and SmartConnect along with flags from the SoapStubAdapter and HTTPSConnectionWrapper classes.

@dnaeon

This comment has been minimized.

Show comment
Hide comment
@dnaeon

dnaeon Feb 11, 2015

Member

Hey Shawn,

PEP-476 talks about the technical details of the new SSL behaviour.

The Opting out section of PEP-476 also provides details on how to disable certificate verification.

For users who wish to opt out of certificate verification, they can achieve this by providing the context argument to urllib.urlopen :

import ssl

# This restores the same behavior as before.
context = ssl._create_unverified_context()
urllib.urlopen("https://no-valid-cert", context=context)

It is also possible though highly discouraged to globally disable verification by monkeypatching the ssl module:

import ssl

ssl._create_default_https_context = ssl._create_unverified_context

Hope that helps.

Regards,
Marin

Member

dnaeon commented Feb 11, 2015

Hey Shawn,

PEP-476 talks about the technical details of the new SSL behaviour.

The Opting out section of PEP-476 also provides details on how to disable certificate verification.

For users who wish to opt out of certificate verification, they can achieve this by providing the context argument to urllib.urlopen :

import ssl

# This restores the same behavior as before.
context = ssl._create_unverified_context()
urllib.urlopen("https://no-valid-cert", context=context)

It is also possible though highly discouraged to globally disable verification by monkeypatching the ssl module:

import ssl

ssl._create_default_https_context = ssl._create_unverified_context

Hope that helps.

Regards,
Marin

@hartsock

This comment has been minimized.

Show comment
Hide comment
@hartsock

hartsock Feb 11, 2015

Member

@dnaeon thanks! That certainly helps clarify a few things and we were looking for a global monkey patch solution as a stop-gap. I don't want that as a permanent solution but I would want to have a work-around to show people while we work out what to do correctly.

Member

hartsock commented Feb 11, 2015

@dnaeon thanks! That certainly helps clarify a few things and we were looking for a global monkey patch solution as a stop-gap. I don't want that as a permanent solution but I would want to have a work-around to show people while we work out what to do correctly.

@Mechazawa

This comment has been minimized.

Show comment
Hide comment
@Mechazawa

Mechazawa Mar 20, 2015

Wouldn't the simplest fix be just adding a parameter for simply disabling ssl verification. Since requests is being used it would be extremely simple to disable SSL verification.

I'm willing to pick this one up if it's alright with @hartsock and @dnaeon.

Mechazawa commented Mar 20, 2015

Wouldn't the simplest fix be just adding a parameter for simply disabling ssl verification. Since requests is being used it would be extremely simple to disable SSL verification.

I'm willing to pick this one up if it's alright with @hartsock and @dnaeon.

@pdellaert

This comment has been minimized.

Show comment
Hide comment
@pdellaert

pdellaert Mar 20, 2015

Contributor

I already do this myself allowing a parameter to my scripts to be passed, disabling SSL verification(or at least the warnings):

if nosslcheck:
        requests.packages.urllib3.disable_warnings()

I think it's important to allow for SSL verification and enable it as a standard... Just for security awareness.

It would be nice to see this in the SmartConnect method as a parameter with a default to verify the certificate.

Contributor

pdellaert commented Mar 20, 2015

I already do this myself allowing a parameter to my scripts to be passed, disabling SSL verification(or at least the warnings):

if nosslcheck:
        requests.packages.urllib3.disable_warnings()

I think it's important to allow for SSL verification and enable it as a standard... Just for security awareness.

It would be nice to see this in the SmartConnect method as a parameter with a default to verify the certificate.

@Mechazawa

This comment has been minimized.

Show comment
Hide comment
@Mechazawa

Mechazawa Mar 20, 2015

Something like a parameter called unsafe_ssl would be perfect. It could also write a warning to sys.stderr to make sure the developer is notified that their code is running in an unsafe state. Monkeypatching the SSL module would not really be needed unless the project is using another library for HTTP.

Mechazawa commented Mar 20, 2015

Something like a parameter called unsafe_ssl would be perfect. It could also write a warning to sys.stderr to make sure the developer is notified that their code is running in an unsafe state. Monkeypatching the SSL module would not really be needed unless the project is using another library for HTTP.

@hartsock hartsock added the blocker label May 5, 2015

@hartsock hartsock changed the title from [Enhancement] PEP 0466 - SSL - Changes consideration to PEP 0466 - SSL - Changes consideration May 27, 2015

@hartsock

This comment has been minimized.

Show comment
Hide comment
@hartsock

hartsock May 27, 2015

Member

See also [https://github.com/vmware/pyvmomi/issues/235], [https://github.com/vmware/pyvmomi/issues/179], [https://github.com/gevent/gevent/issues/477]

Member

hartsock commented May 27, 2015

See also [https://github.com/vmware/pyvmomi/issues/235], [https://github.com/vmware/pyvmomi/issues/179], [https://github.com/gevent/gevent/issues/477]

hartsock added a commit to hartsock/pyvmomi that referenced this issue May 29, 2015

optionally allow unverified certificates
Toggles on and off strict host name verification in the standard python SSL
library. A change was introduced to Python's SSL verification behavior with
[PEP 0466](https://www.python.org/dev/peps/pep-0466/) which forces this
library to consider a set of changes to preserve backward compatability.

The intention of PEP 0466 is to improve SSL security for Python programmers
but this changes default behaviors. This change proposes keeping the default
behaviors undisturbed but forcing users and developers working in insecure
environments to become aware of the change and adjust their security measures
and code accordingly.

fixes: #212, #235, and #179

hartsock added a commit to hartsock/pyvmomi that referenced this issue May 29, 2015

optionally allow unverified certificates
Toggles on and off strict host name verification in the standard python SSL
library. A change was introduced to Python's SSL verification behavior with
[PEP 0466](https://www.python.org/dev/peps/pep-0466/) which forces this
library to consider a set of changes to preserve backward compatability.

The intention of PEP 0466 is to improve SSL security for Python programmers
but this changes default behaviors. This change proposes keeping the default
behaviors undisturbed but forcing users and developers working in insecure
environments to become aware of the change and adjust their security measures
and code accordingly.

fixes: #212, #235, and #179
@hartsock

This comment has been minimized.

Show comment
Hide comment
@hartsock

hartsock Jun 2, 2015

Member

NOTE: I attempted to avoid the global monkey-patch solution but pyVmomi has a connection somewhere that still opens verified even when I make the unverified context explicitly. I've not had time to run down all the connection open and closes. I'll attempt to do this at some point late this week or early next week.

Member

hartsock commented Jun 2, 2015

NOTE: I attempted to avoid the global monkey-patch solution but pyVmomi has a connection somewhere that still opens verified even when I make the unverified context explicitly. I've not had time to run down all the connection open and closes. I'll attempt to do this at some point late this week or early next week.

@duffolonious

This comment has been minimized.

Show comment
Hide comment
@duffolonious

duffolonious Jun 10, 2015

This fixes it for me:

⟫ git diff
diff --git a/pyVmomi/SoapAdapter.py b/pyVmomi/SoapAdapter.py
index 71fd2e5..5aa0bd6 100644
--- a/pyVmomi/SoapAdapter.py
+++ b/pyVmomi/SoapAdapter.py
@@ -1208,6 +1208,8 @@ class SoapStubAdapter(SoapStubAdapterBase):
       if cacertsFile:
          self.schemeArgs['ca_certs'] = cacertsFile
          self.schemeArgs['cert_reqs'] = ssl.CERT_REQUIRED
+      if self.schemeArgs == {}:
+         self.schemeArgs['context'] = ssl._create_unverified_context()
       self.samlToken = samlToken
       self.requestModifierList = []
       self._acceptCompressedResponses = acceptCompressedResponses

I'm assuming if none of the schemeArgs are set that we can use the unverified cert.

Has this problem already been solved somewhere (another branch or something)? I'm using this because it's really annoying on Ubuntu Vivid (Python 2.7.9). Python 2.7.9 is when this context argument was added https://docs.python.org/2/library/httplib.html#httplib.HTTPSConnection - so may need a version check or something to be compatible.

duffolonious commented Jun 10, 2015

This fixes it for me:

⟫ git diff
diff --git a/pyVmomi/SoapAdapter.py b/pyVmomi/SoapAdapter.py
index 71fd2e5..5aa0bd6 100644
--- a/pyVmomi/SoapAdapter.py
+++ b/pyVmomi/SoapAdapter.py
@@ -1208,6 +1208,8 @@ class SoapStubAdapter(SoapStubAdapterBase):
       if cacertsFile:
          self.schemeArgs['ca_certs'] = cacertsFile
          self.schemeArgs['cert_reqs'] = ssl.CERT_REQUIRED
+      if self.schemeArgs == {}:
+         self.schemeArgs['context'] = ssl._create_unverified_context()
       self.samlToken = samlToken
       self.requestModifierList = []
       self._acceptCompressedResponses = acceptCompressedResponses

I'm assuming if none of the schemeArgs are set that we can use the unverified cert.

Has this problem already been solved somewhere (another branch or something)? I'm using this because it's really annoying on Ubuntu Vivid (Python 2.7.9). Python 2.7.9 is when this context argument was added https://docs.python.org/2/library/httplib.html#httplib.HTTPSConnection - so may need a version check or something to be compatible.

@hartsock

This comment has been minimized.

Show comment
Hide comment
@hartsock

hartsock Jun 17, 2015

Member

@duffolonious thanks for the targeted patch! Do you think you could submit that as a pull request so we could discuss it?

Member

hartsock commented Jun 17, 2015

@duffolonious thanks for the targeted patch! Do you think you could submit that as a pull request so we could discuss it?

@duffolonious

This comment has been minimized.

Show comment
Hide comment
@duffolonious

duffolonious Jun 17, 2015

@hartsock sure, I'll get something together later today I think.

duffolonious commented Jun 17, 2015

@hartsock sure, I'll get something together later today I think.

@hartsock

This comment has been minimized.

Show comment
Hide comment
@hartsock

hartsock Jun 22, 2015

Member

Thanks to @duffolonious, I think @parthgala should look at the patch submitted here.

@parthgala how is your progress coming along on this issue?

Member

hartsock commented Jun 22, 2015

Thanks to @duffolonious, I think @parthgala should look at the patch submitted here.

@parthgala how is your progress coming along on this issue?

@flaub

This comment has been minimized.

Show comment
Hide comment
@flaub

flaub Nov 6, 2015

Any news on this?

flaub commented Nov 6, 2015

Any news on this?

@jeremyverda

This comment has been minimized.

Show comment
Hide comment
@jeremyverda

jeremyverda Dec 4, 2015

Hi,

Any news about that?

Thanks.

jeremyverda commented Dec 4, 2015

Hi,

Any news about that?

Thanks.

@pdellaert

This comment has been minimized.

Show comment
Hide comment
@pdellaert

pdellaert Dec 4, 2015

Contributor

With pyVmomi 6.0.0, you can use a parameter in the SmartConnect class to specify a SSLContext which is then used. This context can be set to not verify the certificate.

A short code example as described in https://dellaert.org/2015/12/02/pyvmomi-6-0-0-vsphere-6-0-and-ssl/

import ssl

from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim, vmodl

# Disabling SSL certificate verification
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
context.verify_mode = ssl.CERT_NONE

vc = None

vcenter_host = "10.0.0.10"
vcenter_port = 443
vcenter_username = "root"
vcenter_password = "vmware"

# Connecting to vCenter
try:
    vc = SmartConnect(host=vcenter_host, user=vcenter_username, pwd=vcenter_password, port=vcenter_port, sslContext=context)

except IOError as e:
    print "I/O error({0}): {1}".format(e.errno, e.strerror)

# Do stuff
Contributor

pdellaert commented Dec 4, 2015

With pyVmomi 6.0.0, you can use a parameter in the SmartConnect class to specify a SSLContext which is then used. This context can be set to not verify the certificate.

A short code example as described in https://dellaert.org/2015/12/02/pyvmomi-6-0-0-vsphere-6-0-and-ssl/

import ssl

from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim, vmodl

# Disabling SSL certificate verification
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
context.verify_mode = ssl.CERT_NONE

vc = None

vcenter_host = "10.0.0.10"
vcenter_port = 443
vcenter_username = "root"
vcenter_password = "vmware"

# Connecting to vCenter
try:
    vc = SmartConnect(host=vcenter_host, user=vcenter_username, pwd=vcenter_password, port=vcenter_port, sslContext=context)

except IOError as e:
    print "I/O error({0}): {1}".format(e.errno, e.strerror)

# Do stuff
@jeremyverda

This comment has been minimized.

Show comment
Hide comment
@jeremyverda

jeremyverda Dec 4, 2015

Thanks, I confirm that it's working. Good to know

jeremyverda commented Dec 4, 2015

Thanks, I confirm that it's working. Good to know

@rajalokan

This comment has been minimized.

Show comment
Hide comment
@rajalokan

rajalokan Dec 16, 2015

Hi @pdellaert , How to get your above patch working for python 2.7.6 ( on Ubuntu 14.04). I'm getting
AttributeError: 'module' object has no attribute 'SSLContext' on calling
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1). Any way to backport this ssl feature into python 2.7.6.

rajalokan commented Dec 16, 2015

Hi @pdellaert , How to get your above patch working for python 2.7.6 ( on Ubuntu 14.04). I'm getting
AttributeError: 'module' object has no attribute 'SSLContext' on calling
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1). Any way to backport this ssl feature into python 2.7.6.

@pwrmiller

This comment has been minimized.

Show comment
Hide comment
@pwrmiller

pwrmiller Dec 16, 2015

I got this working with pyvmomi 5.5.0 - the 6.0.0 package appears to be broken with Python 2.7.6.

To use 5.5.0:
sudo pip install pyvmomi==5.5.0

Then, put this in your script prior to the SmartConnect call to avoid issues if your Python ever becomes upgraded:
https://gist.github.com/michaelrice/a6794a017e349fc65d01

pwrmiller commented Dec 16, 2015

I got this working with pyvmomi 5.5.0 - the 6.0.0 package appears to be broken with Python 2.7.6.

To use 5.5.0:
sudo pip install pyvmomi==5.5.0

Then, put this in your script prior to the SmartConnect call to avoid issues if your Python ever becomes upgraded:
https://gist.github.com/michaelrice/a6794a017e349fc65d01

@pdellaert

This comment has been minimized.

Show comment
Hide comment
@pdellaert

pdellaert Dec 16, 2015

Contributor

@rajalokan : As in my blogpost, you can do this as follows:

import ssl

from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim, vmodl

context = None

# Disabling urllib3 ssl warnings
requests.packages.urllib3.disable_warnings()

# Disabling SSL certificate verification
if hasattr(ssl, 'SSLContext'):
    context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
    context.verify_mode = ssl.CERT_NONE

vc = None

vcenter_host = "10.0.0.10"
vcenter_port = 443
vcenter_username = "root"
vcenter_password = "vmware"

# Connecting to vCenter
try:
    if context:
        vc = SmartConnect(host=vcenter_host, user=vcenter_username, pwd=vcenter_password, port=vcenter_port, sslContext=context)
    else:
        vc = SmartConnect(host=vcenter_host, user=vcenter_username, pwd=vcenter_password, port=vcenter_port)

except IOError as e:
    print "I/O error({0}): {1}".format(e.errno, e.strerror)

# Do stuff
Contributor

pdellaert commented Dec 16, 2015

@rajalokan : As in my blogpost, you can do this as follows:

import ssl

from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim, vmodl

context = None

# Disabling urllib3 ssl warnings
requests.packages.urllib3.disable_warnings()

# Disabling SSL certificate verification
if hasattr(ssl, 'SSLContext'):
    context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
    context.verify_mode = ssl.CERT_NONE

vc = None

vcenter_host = "10.0.0.10"
vcenter_port = 443
vcenter_username = "root"
vcenter_password = "vmware"

# Connecting to vCenter
try:
    if context:
        vc = SmartConnect(host=vcenter_host, user=vcenter_username, pwd=vcenter_password, port=vcenter_port, sslContext=context)
    else:
        vc = SmartConnect(host=vcenter_host, user=vcenter_username, pwd=vcenter_password, port=vcenter_port)

except IOError as e:
    print "I/O error({0}): {1}".format(e.errno, e.strerror)

# Do stuff
@rajalokan

This comment has been minimized.

Show comment
Hide comment
@rajalokan

rajalokan Dec 17, 2015

Thanks @pwrmiller , downgrading to pyvmomi==5.5.0 works on python 2.7.6. I will wait for #252 before upgrading to 6.0. Thanks @pdellaert too for suggestion.

rajalokan commented Dec 17, 2015

Thanks @pwrmiller , downgrading to pyvmomi==5.5.0 works on python 2.7.6. I will wait for #252 before upgrading to 6.0. Thanks @pdellaert too for suggestion.

@hartsock hartsock removed their assignment Dec 29, 2015

@tintinweb

This comment has been minimized.

Show comment
Hide comment
@tintinweb

tintinweb Mar 4, 2016

requests seems to check against a bundled cacert.pem so it should be sufficient to add trusted ca-certificates to the cacert.pem bundled with requests. here's a oneliner to get the path of your cacert.pem:

python -c "import requests; print requests.certs.where()"

tintinweb commented Mar 4, 2016

requests seems to check against a bundled cacert.pem so it should be sufficient to add trusted ca-certificates to the cacert.pem bundled with requests. here's a oneliner to get the path of your cacert.pem:

python -c "import requests; print requests.certs.where()"
@tianhao64

This comment has been minimized.

Show comment
Hide comment
@tianhao64

tianhao64 Apr 2, 2016

Contributor

This is fixed by 92c1de5

Contributor

tianhao64 commented Apr 2, 2016

This is fixed by 92c1de5

@tianhao64 tianhao64 closed this Apr 2, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment