winrepo - SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed #27081

TheBigBear · 2015-09-12T11:35:04Z

The current python (ver. 2.7.8) based winrepo downloader fails on https sites that require SNI (server name indication) support the current version of python used does not, but allegedly this is resolved in python >= 2.7.9 .

The error is:

: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

salt 'uk-it-20' pkg.install pdf24creator

which tries downloading https://en.pdf24.org/products/pdf-creator/download/pdf24-creator-7.2.0.msi fails.

uk-it-20:
    The minion function caused an exception: Traceback (most recent call last):
      File "C:\salt\bin\lib\site-packages\salt\minion.py", line 1004, in _thread_return
        return_data = func(*args, **kwargs)
      File "C:\salt\bin\lib\site-packages\salt\modules\win_pkg.py", line 602, in install
        cached_pkg = __salt__['cp.cache_file'](installer, saltenv)
      File "C:\salt\bin\lib\site-packages\salt\modules\cp.py", line 365, in cache_file
        result = __context__['cp.fileclient'].cache_file(path, saltenv)
      File "C:\salt\bin\lib\site-packages\salt\fileclient.py", line 155, in cache_file
        return self.get_url(path, '', True, saltenv)
      File "C:\salt\bin\lib\site-packages\salt\fileclient.py", line 618, in get_url
        **get_kwargs
      File "C:\salt\bin\lib\site-packages\salt\utils\http.py", line 424, in query
        **req_kwargs
      File "C:\salt\bin\lib\site-packages\tornado\httpclient.py", line 102, in fetch
        self._async_client.fetch, request, **kwargs))
      File "C:\salt\bin\lib\site-packages\tornado\ioloop.py", line 445, in run_sync
        return future_cell[0].result()
      File "C:\salt\bin\lib\site-packages\tornado\concurrent.py", line 215, in result
        raise_exc_info(self._exc_info)
      File "<string>", line 3, in raise_exc_info
    SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

The site en.pdf24.org (85.25.111.198) requiring SNI support has nothing much wrong with it's SSL setup, so the winrepo downloader should not complain.

As a temporary work-around I will drop the 's' from 'https' winrepo installer URLs, for those pkgs that require this, and do have a non https link, but this is sub-optimal from a security perspective for an installer.

PPS: I have read in other places that the SSL SNI (and other) improvements have not been entirely satisfactory in python 2.7.9, but are improved upon - again- in the current release python 2.7.10.
So IF nothing else is holding us back it would be great if the python basis for the windows salt minion installer (and hence also the winrepo downloader? does that follow? - or is it going to depend on the python version running on the master?) could be bumpd up to >= python 2.7.10.

The text was updated successfully, but these errors were encountered:

TheBigBear · 2015-09-12T16:37:18Z

I may have assumed too much. ;-(

Now that I have changed the https URL to a simple http one, I still get the same SSL certificate check error? What the heck, why is the winrepo downloader checking SSL certs for a regular http download? Is the site maybe on a browser https/hsts preload list?

Ok so here is an even fuller trace level log:

time salt 'uk-it-20' pkg.install pdf24creator version=7.2.0 -l trace
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: uk-bigbro
[DEBUG   ] Missing configuration file: /root/.saltrc
[TRACE   ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found the in the configuration. Not loading the Logstash logging handlers module.
[TRACE   ] The required configuration section, 'fluent_handler', was not found the in the configuration. Not loading the fluent logging handlers module.
[DEBUG   ] Configuration file path: /etc/salt/master
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: uk-bigbro
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'uk-bigbro_master', 'tcp://127.0.0.1:4506', 'clear')
[TRACE   ] func get_cli_event_returns()
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] get_iter_returns for jid 20150912172942255490 sent to set(['uk-it-20']) will timeout at 17:29:52.270475
[TRACE   ] _get_event() waited 0 seconds and received nothing
[TRACE   ] get_event() received = {'tag': 'salt/job/20150912172942255490/new', 'data': {'tgt_type': 'glob', 'jid': '20150912172942255490', 'tgt': 'uk-it-20', '_stamp': '2015-09-12T16:29:42.257622', 'user': 'root', 'arg': ['pdf24creator', {'version': '7.2.0', '__kwarg__': True}], 'fun': 'pkg.install', 'minions': ['uk-it-20']}}
[TRACE   ] _get_event() waited 0 seconds and received nothing
[TRACE   ] _get_event() waited 0 seconds and received nothing
[TRACE   ] _get_event() waited 0 seconds and received nothing

< cut out many lines of these ;-) >

[TRACE   ] _get_event() waited 0 seconds and received nothing
[TRACE   ] get_event() received = {'tag': 'salt/job/20150912172942255490/ret/uk-it-20', 'data': {'fun_args': ['pdf24creator', {'version': '7.2.0'}], 'jid': '20150912172942255490', 'return': 'The minion function caused an exception: Traceback (most recent call last):\n  File "C:\\salt\\bin\\lib\\site-packages\\salt\\minion.py", line 1004, in _thread_return\n    return_data = func(*args, **kwargs)\n  File "C:\\salt\\bin\\lib\\site-packages\\salt\\modules\\win_pkg.py", line 602, in install\n    cached_pkg = __salt__[\'cp.cache_file\'](installer, saltenv)\n  File "C:\\salt\\bin\\lib\\site-packages\\salt\\modules\\cp.py", line 365, in cache_file\n    result = __context__[\'cp.fileclient\'].cache_file(path, saltenv)\n  File "C:\\salt\\bin\\lib\\site-packages\\salt\\fileclient.py", line 155, in cache_file\n    return self.get_url(path, \'\', True, saltenv)\n  File "C:\\salt\\bin\\lib\\site-packages\\salt\\fileclient.py", line 618, in get_url\n    **get_kwargs\n  File "C:\\salt\\bin\\lib\\site-packages\\salt\\utils\\http.py", line 424, in query\n    **req_kwargs\n  File "C:\\salt\\bin\\lib\\site-packages\\tornado\\httpclient.py", line 102, in fetch\n    self._async_client.fetch, request, **kwargs))\n  File "C:\\salt\\bin\\lib\\site-packages\\tornado\\ioloop.py", line 445, in run_sync\n    return future_cell[0].result()\n  File "C:\\salt\\bin\\lib\\site-packages\\tornado\\concurrent.py", line 215, in result\n    raise_exc_info(self._exc_info)\n  File "<string>", line 3, in raise_exc_info\nSSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed\n', 'success': False, 'cmd': '_return', '_stamp': '2015-09-12T16:29:47.673350', 'fun': 'pkg.install', 'id': 'uk-it-20', 'out': 'nested'}}
[DEBUG   ] jid 20150912172942255490 return from uk-it-20
[DEBUG   ] LazyLoaded nested.output
[TRACE   ] data = {'uk-it-20': 'The minion function caused an exception: Traceback (most recent call last):\n  File "C:\\salt\\bin\\lib\\site-packages\\salt\\minion.py", line 1004, in _thread_return\n    return_data = func(*args, **kwargs)\n  File "C:\\salt\\bin\\lib\\site-packages\\salt\\modules\\win_pkg.py", line 602, in install\n    cached_pkg = __salt__[\'cp.cache_file\'](installer, saltenv)\n  File "C:\\salt\\bin\\lib\\site-packages\\salt\\modules\\cp.py", line 365, in cache_file\n    result = __context__[\'cp.fileclient\'].cache_file(path, saltenv)\n  File "C:\\salt\\bin\\lib\\site-packages\\salt\\fileclient.py", line 155, in cache_file\n    return self.get_url(path, \'\', True, saltenv)\n  File "C:\\salt\\bin\\lib\\site-packages\\salt\\fileclient.py", line 618, in get_url\n    **get_kwargs\n  File "C:\\salt\\bin\\lib\\site-packages\\salt\\utils\\http.py", line 424, in query\n    **req_kwargs\n  File "C:\\salt\\bin\\lib\\site-packages\\tornado\\httpclient.py", line 102, in fetch\n    self._async_client.fetch, request, **kwargs))\n  File "C:\\salt\\bin\\lib\\site-packages\\tornado\\ioloop.py", line 445, in run_sync\n    return future_cell[0].result()\n  File "C:\\salt\\bin\\lib\\site-packages\\tornado\\concurrent.py", line 215, in result\n    raise_exc_info(self._exc_info)\n  File "<string>", line 3, in raise_exc_info\nSSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed\n'}
uk-it-20:
    The minion function caused an exception: Traceback (most recent call last):
      File "C:\salt\bin\lib\site-packages\salt\minion.py", line 1004, in _thread_return
        return_data = func(*args, **kwargs)
      File "C:\salt\bin\lib\site-packages\salt\modules\win_pkg.py", line 602, in install
        cached_pkg = __salt__['cp.cache_file'](installer, saltenv)
      File "C:\salt\bin\lib\site-packages\salt\modules\cp.py", line 365, in cache_file
        result = __context__['cp.fileclient'].cache_file(path, saltenv)
      File "C:\salt\bin\lib\site-packages\salt\fileclient.py", line 155, in cache_file
        return self.get_url(path, '', True, saltenv)
      File "C:\salt\bin\lib\site-packages\salt\fileclient.py", line 618, in get_url
        **get_kwargs
      File "C:\salt\bin\lib\site-packages\salt\utils\http.py", line 424, in query
        **req_kwargs
      File "C:\salt\bin\lib\site-packages\tornado\httpclient.py", line 102, in fetch
        self._async_client.fetch, request, **kwargs))
      File "C:\salt\bin\lib\site-packages\tornado\ioloop.py", line 445, in run_sync
        return future_cell[0].result()
      File "C:\salt\bin\lib\site-packages\tornado\concurrent.py", line 215, in result
        raise_exc_info(self._exc_info)
      File "<string>", line 3, in raise_exc_info
    SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
[TRACE   ] _get_event() waited 0 seconds and received nothing
[DEBUG   ] jid 20150912172942255490 found all minions set(['uk-it-20'])

real    0m6.963s
user    0m1.626s
sys 0m0.177s

TheBigBear · 2015-09-13T07:23:54Z

just double checked and there is no HSTS headers involved here, so it's not that.

So why does a 'plain' http download trigger a SSL server check/verify error?

@twangboy @UtahDave ^^

krak3n · 2015-09-14T10:53:17Z

Can't download files from S3 - This is currently a breaking issue for us.

twangboy · 2015-09-14T15:05:05Z

Might have to do this #25440

TheBigBear · 2015-09-14T15:30:41Z

@twangboy, I agree that it is likley linked to the python veriosn in use.

BUT, I have changed the installer URL for the pdf24creator winrepo pkg to a simple http one, and I still get the winrepo downloader failing with a SSL server certificate check.

I would call this a bug if the winrepo downloader even tries to do a SSL server certificate check on a simple non secure http download. And in htis case ti seems to be failing even for a plain http download in the same fashion it failed for https download. (that makes no sense and smacks of a caching issue - or something - but I can't find where and why it would somehow still have the old https URL for this install and trace level output doesn't let me see enough to see what is really going on here.)

TheBigBear · 2015-09-14T22:34:58Z

@krak3n how do you get to your S3 downloads?

what is the URL you use? Is there any web redirect (302) involved as well?

you could do a curl -D myheaders https://url-to-my-s3-file /dev/null and then examine the headers in the myheaders file.

krak3n · 2015-09-15T09:19:10Z

@TheBigBear my state was basically this:

/foo.jpg:
  file.managed:
    - source: https://s3-eu-west-1.amazonaws.com/public.bucket/foo.jpg

This threw the SSL error, but curling / wgeting works fine.

In end end I gave up and did a cmd.run which ran a curl to download the file.

TheBigBear · 2015-09-15T14:19:57Z

@twangboy, ok, so I created (just for proof of concept testing) a windows salt minion devel ver. actually using python 2.7.10 (was only a two line chg in the salt-windows-devel get-settings.psm1 lines 74 + 84) and this does fix the routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed error.

The way I quickly checked this is by running a saltutil http.query
salt 'uk-it-20' http.query https://en.pdf24.org/products/pdf-creator/download/pdf24-creator-7.3.0.msi
and it returns 'clean'. (uk-it-dev1 runs on the new windows minion built on python 2.7.10 )

uk-it-dev1:
    ----------

Full versions report is:

C:\salt>salt-call --versions
Salt Version:
           Salt: 2015.8.0-20-g7483556

Dependency Versions:
         Jinja2: 2.7.3
       M2Crypto: 0.21.1
           Mako: 1.0.1
         PyYAML: 3.11
          PyZMQ: 14.7.0
         Python: 2.7.10 (default, May 23 2015, 09:44:00) [MSC v.1500 64 bit (AMD64)]
           RAET: 0.6.3
        Tornado: 4.2.1
            ZMQ: 4.1.2
           cffi: 1.2.1
       cherrypy: 3.8.0
       dateutil: 2.4.2
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: 1.3.8
        libnacl: 1.4.3
   msgpack-pure: 0.1.3
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.21.3
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: 0.2.4

System Versions:
           dist:
        machine: AMD64
        release: 8
         system: 8 6.2.9200  Multiprocessor Free

And if I run same cmd line against a standard official windows minion built on python ver 2.7.8 I get the SSL error:

uk-it-20:
    The minion function caused an exception: Traceback (most recent call last):
      File "C:\salt\bin\lib\site-packages\salt\minion.py", line 1004, in _thread_return
        return_data = func(*args, **kwargs)
      File "C:\salt\bin\lib\site-packages\salt\modules\http.py", line 30, in query
        return salt.utils.http.query(url=url, opts=__opts__, **kwargs)
      File "C:\salt\bin\lib\site-packages\salt\utils\http.py", line 424, in query
        **req_kwargs
      File "C:\salt\bin\lib\site-packages\tornado\httpclient.py", line 102, in fetch
        self._async_client.fetch, request, **kwargs))
      File "C:\salt\bin\lib\site-packages\tornado\ioloop.py", line 445, in run_sync
        return future_cell[0].result()
      File "C:\salt\bin\lib\site-packages\tornado\concurrent.py", line 215, in result
        raise_exc_info(self._exc_info)
      File "<string>", line 3, in raise_exc_info
    SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

But @twangboy and @terminalmage ^^ why on earth is the saltutil http.query (or fileclient in this case) using any SSL at all when I pass it a plain non-secure http URL, like so:

salt-call 'uk-it-20' http://en.pdf24.org/products/pdf-creator/download/pdf24-creator-7.3.0.msi (note: the http not https URL !)

saltutil's http.query should never go anywhere near checking SSL server certificates for a plain non-secure http URL, in my opinion.

@terminalmage ^^

TheBigBear changed the title ~~winrepo - SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed - python >= 2.7.9 SNI support required~~ winrepo - SSLError - python >= 2.7.9 SNI SSL support required Sep 12, 2015

TheBigBear changed the title ~~winrepo - SSLError - python >= 2.7.9 SNI SSL support required~~ winrepo - SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed Sep 12, 2015

twangboy added severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around Windows P2 Priority 2 Platform Relates to OS, containers, platform-based utilities like FS, system based apps ZRELEASED - Beryllium labels Sep 14, 2015

twangboy added this to the Be -2 milestone Sep 14, 2015

twangboy self-assigned this Sep 14, 2015

twangboy modified the milestones: B 12, Be -2 Sep 14, 2015

twangboy added the Stretch label Sep 15, 2015

This was referenced Sep 17, 2015

2015.8.0 winrepo downloader corrupts some installers #27093

Closed

[ERROR ] Exception 'close_fds is not supported on Windows platforms if you redirect stdin/stdout/stderr' #27220

Closed

twangboy mentioned this issue Sep 22, 2015

Pass ca_bundle for windows (fixes SSL Error) #27301

Merged

twangboy closed this as completed Sep 22, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

winrepo - SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed #27081

winrepo - SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed #27081

TheBigBear commented Sep 12, 2015

TheBigBear commented Sep 12, 2015

TheBigBear commented Sep 13, 2015

krak3n commented Sep 14, 2015

twangboy commented Sep 14, 2015

TheBigBear commented Sep 14, 2015

TheBigBear commented Sep 14, 2015

krak3n commented Sep 15, 2015

TheBigBear commented Sep 15, 2015

winrepo - SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed #27081

winrepo - SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed #27081

Comments

TheBigBear commented Sep 12, 2015

TheBigBear commented Sep 12, 2015

TheBigBear commented Sep 13, 2015

krak3n commented Sep 14, 2015

twangboy commented Sep 14, 2015

TheBigBear commented Sep 14, 2015

TheBigBear commented Sep 14, 2015

krak3n commented Sep 15, 2015

TheBigBear commented Sep 15, 2015