Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Temporarily disable SG certificate verification #5432

Merged
merged 2 commits into from
May 30, 2023
Merged

Conversation

jbdietrich
Copy link
Contributor

Description

The SG parser currently fails with a certificate error:

~/code/electricitymaps-contrib [master] 馃Ж poetry run test_parser SG                                                                                                                                                                                                     
2023-05-30 08:57:15,550 DEBUG    urllib3.connectionpool         Starting new HTTPS connection (1): www.emcsg.com:443                                                                                                                                                     
Traceback (most recent call last):                                                                                                                                                                                                                                       
  File "/Users/james/code/electricitymaps-contrib/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen                                                                                                                                     
    httplib_response = self._make_request(                                                                                                                                                                                                                               
  File "/Users/james/code/electricitymaps-contrib/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request                                                                                                                               
    self._validate_conn(conn)                                                                                                                                                                                                                                            
  File "/Users/james/code/electricitymaps-contrib/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn                                                                                                                             
    conn.connect()                                                                                                                                                                                                                                                       
  File "/Users/james/code/electricitymaps-contrib/.venv/lib/python3.8/site-packages/urllib3/connection.py", line 419, in connect                                                                                                                                         
    self.sock = ssl_wrap_socket(                                                                                                                                                                                                                                         
  File "/Users/james/code/electricitymaps-contrib/.venv/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket                                                                                                                                  
    ssl_sock = _ssl_wrap_socket_impl(                                                                                                                                                                                                                                    
  File "/Users/james/code/electricitymaps-contrib/.venv/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl                                                                                                                            
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)                                                                                                                                                                                                
  File "/Users/james/.pyenv/versions/3.8.12/lib/python3.8/ssl.py", line 500, in wrap_socket                                                                                                                                                                              
    return self.sslsocket_class._create(                                                                                                                                                                                                                                 
  File "/Users/james/.pyenv/versions/3.8.12/lib/python3.8/ssl.py", line 1040, in _create                                                                                                                                                                                 
    self.do_handshake()                                                                                                                                                                                                                                                  
  File "/Users/james/.pyenv/versions/3.8.12/lib/python3.8/ssl.py", line 1309, in do_handshake                                                                                                                                                                            
    self._sslobj.do_handshake()                                                                                                                                                                                                                                          
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)                                                                                                                           
                                                                                                                                                                                                                                                                         
During handling of the above exception, another exception occurred:                                                                 
                                                                  
Traceback (most recent call last):                                                                                                                                                                                                                                       
  File "/Users/james/code/electricitymaps-contrib/.venv/lib/python3.8/site-packages/requests/adapters.py", line 439, in send                                                                                                                                             
    resp = conn.urlopen(                                                                                                                                                                                                                                                 
  File "/Users/james/code/electricitymaps-contrib/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen                                                                                                                                     
    retries = retries.increment(                                  
  File "/Users/james/code/electricitymaps-contrib/.venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment                                                                                                                                       
    raise MaxRetryError(_pool, url, error or ResponseError(cause))                                                                  
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.emcsg.com', port=443): Max retries exceeded with url: /ChartServer/blue/ticker (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get
 local issuer certificate (_ssl.c:1131)')))                       
                                                                                                                                    
During handling of the above exception, another exception occurred:                                                                 
                                                                                                                                    
Traceback (most recent call last):                                                                                                  
  File "<string>", line 1, in <module>                                                                                              
  File "/Users/james/code/electricitymaps-contrib/.venv/lib/python3.8/site-packages/click/core.py", line 1134, in __call__                                                                                                                                               
    return self.main(*args, **kwargs)                                                                                               
  File "/Users/james/code/electricitymaps-contrib/.venv/lib/python3.8/site-packages/click/core.py", line 1059, in main                                                                                                                                                   
    rv = self.invoke(ctx)                                         
  File "/Users/james/code/electricitymaps-contrib/.venv/lib/python3.8/site-packages/click/core.py", line 1401, in invoke                                                                                                                                                 
    return ctx.invoke(self.callback, **ctx.params)                                                                                  
  File "/Users/james/code/electricitymaps-contrib/.venv/lib/python3.8/site-packages/click/core.py", line 767, in invoke                                                                                                                                                  
    return __callback(*args, **kwargs)                                                                                              
  File "/Users/james/code/electricitymaps-contrib/test_parser.py", line 60, in test_parser                                                                                                                                                                               
    res = parser(*args, target_datetime=target_datetime, logger=getLogger(__name__))                                                                                                                                                                                     
  File "/Users/james/code/electricitymaps-contrib/parsers/SG.py", line 150, in fetch_production                                                                                                                                                                          
    response = requests_obj.get(TICKER_URL)                                                                                         
  File "/Users/james/code/electricitymaps-contrib/.venv/lib/python3.8/site-packages/requests/sessions.py", line 555, in get                                                                                                                                              
    return self.request('GET', url, **kwargs)                                                                                       
  File "/Users/james/code/electricitymaps-contrib/.venv/lib/python3.8/site-packages/requests/sessions.py", line 542, in request                                                                                                                                          
    resp = self.send(prep, **send_kwargs)                                                                                           
  File "/Users/james/code/electricitymaps-contrib/.venv/lib/python3.8/site-packages/requests/sessions.py", line 655, in send                                                                                                                                             
    r = adapter.send(request, **kwargs)                                                                                                                                                                                                                                  
  File "/Users/james/code/electricitymaps-contrib/.venv/lib/python3.8/site-packages/requests/adapters.py", line 514, in send                                                                                                                                             
    raise SSLError(e, request=request)                                                                                              
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.emcsg.com', port=443): Max retries exceeded with url: /ChartServer/blue/ticker (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get loc
al issuer certificate (_ssl.c:1131)')))

The root cause of the issue seems to be an missing intermediate certificate:

Screenshot 2023-05-30 at 09 05 12

The long term fix is either to get the server admin to send the intermediate certificate or to install the cert chain manually ourselves in the image that runs this parser in the cloud. For now, this will let us get the parser back up and running.

Double check

  • I have tested my parser changes locally with poetry run test_parser "zone_key"
  • I have run pnpx prettier --write . and poetry run format to format my changes.

Copy link
Member

@VIKTORVAV99 VIKTORVAV99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First PR in the contrib repo!
Congrats James! 馃帀

These changes look straight forward to me so I'll approve them right away but we need to remember to check up on this source in a while. 馃槄

@jbdietrich jbdietrich merged commit 4c446cd into master May 30, 2023
16 checks passed
@jbdietrich jbdietrich deleted the jb/patch-sg-parser branch May 30, 2023 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants