Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Command line option to skip SSL certificate validation while connecting to api.adsabs.harvard.edu #33

Closed
kirxkirx opened this issue Jan 25, 2023 · 2 comments · Fixed by #35
Closed

Command line option to skip SSL certificate validation while connecting to api.adsabs.harvard.edu #33

kirxkirx opened this issue Jan 25, 2023 · 2 comments · Fixed by #35
Assignees
@yymao
Labels
feature-request

Comments

@kirxkirx
Copy link

This is not adstex problem that could have an adstex-specific solution. The ADS servers (api.adsabs.harvard.edu) apparently got a new SSL certificate that my operating system (unlike my browser) does not recognize. It would be great to add a command line option to adstex that would tell it to skip certificate validation while connecting, for example as discussed here.

For now I'm getting the following error message:

$ adstex nher.tex

------------------------------------------------------------
Auto-identifying bibtex files...
Main bibtex source (output file): nher.bib

------------------------------------------------------------
Overwritting existing string for key: june.
Traceback (most recent call last):
  File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1147)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.adsabs.harvard.edu', port=443): Max retries exceeded with url: /v1/search/query/?q=identifier%3A%222013MNRAS.433.1991R%22&fl=id&fl=bibcode&sort=score+desc%2Cid+desc&cursorMark=%2A&rows=50 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1147)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kirx/.local/bin/adstex", line 33, in <module>
    sys.exit(load_entry_point('adstex==0.3.12', 'console_scripts', 'adstex')())
  File "/home/kirx/.local/lib/python3.8/site-packages/adstex.py", line 396, in main
    bibcode_new = entry2bibcode(bib.entries_dict[key])
  File "/home/kirx/.local/lib/python3.8/site-packages/adstex.py", line 261, in entry2bibcode
    id_this = id2bibcode(unquote(entry[field_name]), possible_id_types)
  File "/home/kirx/.local/lib/python3.8/site-packages/adstex.py", line 165, in id2bibcode
    return next(s).bibcode
  File "/home/kirx/.local/lib/python3.8/site-packages/ads/search.py", line 501, in __next__
    self.execute()
  File "/home/kirx/.local/lib/python3.8/site-packages/ads/search.py", line 533, in execute
    self.session.get(self.HTTP_ENDPOINT, params=self.query)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.adsabs.harvard.edu', port=443): Max retries exceeded with url: /v1/search/query/?q=identifier%3A%222013MNRAS.433.1991R%22&fl=id&fl=bibcode&sort=score+desc%2Cid+desc&cursorMark=%2A&rows=50 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1147)')))
@yymao yymao mentioned this issue Jan 25, 2023
Merged
@yymao yymao self-assigned this Jan 25, 2023
@yymao
Copy link
Owner

yymao commented Jan 25, 2023

Thanks for the report @kirxkirx. I can confirm this issue. Hopefully this is just temporary and will soon go away once the certificates are updated. In the meantime, I implemented an --disable-ssl-verification option in #35. If you can give it a test that'd be much appreciated (see instruction in #35).

@kirxkirx
Copy link
Author

Thanks for fixing this so quickly!

I can confirm that the fix is working:

$ adstex nher.tex 

------------------------------------------------------------
Auto-identifying bibtex files...
Main bibtex source (output file): nher.bib

------------------------------------------------------------
Overwriting existing string for key: june.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 453, in wrap_socket
    cnx.do_handshake()
  File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1915, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1647, in _raise_ssl_error
    _raise_current_error()
  File "/usr/lib/python3/dist-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 344, in connect
    ssl_context=context)
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 344, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 459, in wrap_socket
    raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.adsabs.harvard.edu', port=443): Max retries exceeded with url: /v1/search/query/?q=identifier%3A%221990ApJ...356..250L%22&fl=id&fl=bibcode&sort=score+desc%2Cid+desc&cursorMark=%2A&rows=50 (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/.local/bin/adstex", line 10, in <module>
    sys.exit(main())
  File "/home/pi/.local/lib/python3.7/site-packages/adstex.py", line 417, in main
    bibcode_new = entry2bibcode(bib.entries_dict[key])
  File "/home/pi/.local/lib/python3.7/site-packages/adstex.py", line 268, in entry2bibcode
    id_this = id2bibcode(unquote(entry[field_name]), possible_id_types)
  File "/home/pi/.local/lib/python3.7/site-packages/adstex.py", line 172, in id2bibcode
    return next(s).bibcode
  File "/home/pi/.local/lib/python3.7/site-packages/ads/search.py", line 501, in __next__
    self.execute()
  File "/home/pi/.local/lib/python3.7/site-packages/ads/search.py", line 533, in execute
    self.session.get(self.HTTP_ENDPOINT, params=self.query)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.adsabs.harvard.edu', port=443): Max retries exceeded with url: /v1/search/query/?q=identifier%3A%221990ApJ...356..250L%22&fl=id&fl=bibcode&sort=score+desc%2Cid+desc&cursorMark=%2A&rows=50 (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
pi@raspberrypi:~/work/62deb7924a8fcc48089dee34 $ adstex nher.tex --disable-ssl-verification
You have chosen to disable SSL verification. This will render your API key vulnerable. Do you want to continue? [y/N] y

------------------------------------------------------------
Auto-identifying bibtex files...
Main bibtex source (output file): nher.bib

------------------------------------------------------------
Overwriting existing string for key: june.
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:849: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2008MNRAS.383..627P: EXISTING
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:849: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
1956ApJ...123...68W: EXISTING
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:849: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2019ascl.soft05011F: EXISTING
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:849: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2018ApJ...852L...8L: EXISTING
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:849: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2022ApJ...940L..56P: EXISTING
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:849: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2014MNRAS.442..713M: EXISTING
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:849: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2021JAVSO..49..257S: EXISTING
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:849: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2006Natur.442..279O: EXISTING
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:849: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2009MNRAS.400.2050G: EXISTING
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:849: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2021ATel14705....1L: EXISTING
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:849: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

...

1989A&A...221..180D: EXISTING
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:849: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2022MNRAS.510.6110P: EXISTING
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:849: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2020ApJ...904....4F: EXISTING

------------------------------------------------------------
Building new bibtex file, please wait...
------------------------------------------------------------
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:849: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

------------------------------------------------------------
Done!
------------------------------------------------------------

and the resulting .bib file looks good.

@kirxkirx kirxkirx changed the title Command line option to skil SSL certificate validation while connecting to api.adsabs.harvard.edu Command line option to skip SSL certificate validation while connecting to api.adsabs.harvard.edu Jan 25, 2023
@yymao yymao closed this as completed in #35 Jan 25, 2023
yymao added a commit that referenced this issue Jan 25, 2023
@yymao
Merge pull request #35 from /issues/33/option-disable-ssl-verify
6993378 
Add option to disable ssl verify
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yymao yymao

Labels
feature-request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add option to disable ssl verify yymao/adstex
2 participants
@yymao @kirxkirx