Getting "Invalid JWT Signature" after upgrading to rsa==4.7

[alvaroabascar]

Environment details

• OS: Linux, Mac
• Python version: 3.7, 3.8
• pip version: 20.3.3
• google-auth version: 1.22.1

Steps to reproduce

We found this bug while using dvc and using Google Cloud Storage as a backend. Authentication with google is done via a service key file. When running dvc with rsa==4.6 everything works fine, but when upgrading to rsa==4.7, we encounter the following error:

  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/dvc/tree/gs.py", line 139, in isfile
    return blob.exists()
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 484, in exists
    _target_object=None,
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/cloud/_http.py", line 431, in api_request
    timeout=timeout,
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/cloud/_http.py", line 289, in _make_request
    method, url, headers, data, target_object, timeout=timeout
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/cloud/_http.py", line 327, in _do_request
    url=url, method=method, headers=headers, data=data, timeout=timeout
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/auth/transport/requests.py", line 460, in request
    self.credentials.before_request(auth_request, method, url, request_headers)
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/auth/credentials.py", line 133, in before_request
    self.refresh(request)
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/oauth2/service_account.py", line 361, in refresh
    access_token, expiry, _ = _client.jwt_grant(request, self._token_uri, assertion)
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/oauth2/_client.py", line 153, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/oauth2/_client.py", line 124, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/home/alvaro/.virtualenvs/myenv/lib/python3.7/site-packages/google/oauth2/_client.py", line 60, in _handle_error_response
    raise exceptions.RefreshError(error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_grant: Invalid JWT Signature.', '{"error":"invalid_grant","error_description":"Invalid JWT Signature."}')

[busunkim96]

Hi,

Thanks for the report! I've marked this as external for now since I see you also opened sybrenstuvel/python-rsa#173. Please let me know if something needs to be fixed in this library.

[jamescooke]

Hi @busunkim96 , I've added a stacktrace for what I think is causing the error above to the ticket on python-rsa here: sybrenstuvel/python-rsa#173 (comment)

EDIT: the original issue on python-rsa contains new info that shows that this library's call to initialise the key is working fine. No need to read the below.

Looking at that stack trace, the call from /google/auth/crypt/_python_rsa.py to python-rsa looks like:

  File "/usr/local/lib/python3.6/site-packages/google/auth/jwt.py", line 112, in encode
    signature = signer.sign(signing_input)
  File "/usr/local/lib/python3.6/site-packages/google/auth/crypt/_python_rsa.py", line 136, in sign
    return rsa.pkcs1.sign(message, self._key, "SHA-256")

Is there any chance that the _key might not have been initialised correctly given the changes that introduced the blindfac attr here: sybrenstuvel/python-rsa@06ec1ea ?

[lechen26]

jumping in.
we started to get this error exactly recently,.
on our log we have the error

google.auth.exceptions.RefreshError: ('invalid_grant: Invalid JWT Signature.', '{"error":"invalid_grant","error_description":"Invalid JWT Signature."}')

we are using google service account to access google cloud storage on the python service.
nothing has changed on our serviec account or his key.
also, this error is intermittent , meaning we can fetch files from storage all the time and every couple of hours we get this error.

is this the same issue?

[busunkim96]

Hi @lechen26,

The root cause is still up in the air. See sybrenstuvel/python-rsa#173. Can you try explicitly pinning to rsa==4.6?

[fcollman]

I am experiencing the same problem.. i hotfixed pip installed rsa==4.6 and the problem went away.

[lechen26]

i'm trying to pin this rsa version.
i thought its not related to my case as i dont have this package on my requirements.txt file.
but checked with pipdeptree and indeed the rsa is dependency of the google packages and it was set to 4.7
pinned it now to 4.6 and will see if issue not reproduce again.

[sybrenstuvel]

I'm the author of the RSA package. As you can see in sybrenstuvel/python-rsa#173, there is a problem where attributes on the key objects are missing:

  File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/usr/local/lib/python3.6/site-packages/luigi/contrib/gcs.py", line 258, in _forward_args_to_put
    return self.put(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/luigi/contrib/gcs.py", line 255, in put
    self._do_put(media, dest_path)
  File "/usr/local/lib/python3.6/site-packages/luigi/contrib/gcs.py", line 173, in _do_put
    status, response = request.next_chunk()
  File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line 993, in next_chunk
    headers=start_headers,
  File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line 177, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/google_auth_httplib2.py", line 190, in request
    self._request, method, uri, request_headers)
  File "/usr/local/lib/python3.6/site-packages/google/auth/credentials.py", line 133, in before_request
    self.refresh(request)
  File "/usr/local/lib/python3.6/site-packages/google/oauth2/service_account.py", line 360, in refresh
    assertion = self._make_authorization_grant_assertion()
  File "/usr/local/lib/python3.6/site-packages/google/oauth2/service_account.py", line 354, in _make_authorization_grant_assertion
    token = jwt.encode(self._signer, payload)
  File "/usr/local/lib/python3.6/site-packages/google/auth/jwt.py", line 112, in encode
    signature = signer.sign(signing_input)
  File "/usr/local/lib/python3.6/site-packages/google/auth/crypt/_python_rsa.py", line 136, in sign
    return rsa.pkcs1.sign(message, self._key, "SHA-256")
  File "/usr/local/lib/python3.6/site-packages/rsa/pkcs1.py", line 331, in sign
    return sign_hash(msg_hash, priv_key, hash_method)
  File "/usr/local/lib/python3.6/site-packages/rsa/pkcs1.py", line 306, in sign_hash
    encrypted = priv_key.blinded_encrypt(payload)
  File "/usr/local/lib/python3.6/site-packages/rsa/key.py", line 463, in blinded_encrypt
    blinded = self.blind(message)  # blind before encrypting
  File "/usr/local/lib/python3.6/site-packages/rsa/key.py", line 165, in blind
    self._update_blinding_factor()
  File "/usr/local/lib/python3.6/site-packages/rsa/key.py", line 190, in _update_blinding_factor
    if self.blindfac < 0:
AttributeError: blindfac

Is Google Auth creating the keys in some non-standard way? The self.blindfac attribute is set in the key's __init__() function, so I don't see how it could be missing.

[jamescooke]

After looking at the RSA code and @sybrenstuvel 's patch, I would like to reinstate my original comment:

Is there any chance that the _key might not have been initialised correctly given the changes that introduced the blindfac attr?

Running the updated rsa==4.7.1 gives a new error: AttributeError: mutex < both this attr and blindfac are is initialised in AbstractKey.__init__() which you can see in the release diff: sybrenstuvel/python-rsa@version-4.7...version-4.7.1

[busunkim96]

@jamescooke @sybrenstuvel Thank you both for the analyses. 🙏 I'm not very familiar with this bit of the codebase so I will have to take a closer look. I will provide an update tomorrow.

[busunkim96]

For anyone who's seeing this, could you try installing cryptography? google-auth will prefer crytography's RSA implementation when it's available.

google-auth-library-python/google/auth/crypt/rsa.py

Lines 18 to 30 in d4d7f38

	try:
	# Prefer cryptograph-based RSA implementation.
	from google.auth.crypt import _cryptography_rsa
	RSASigner = _cryptography_rsa.RSASigner
	RSAVerifier = _cryptography_rsa.RSAVerifier
	except ImportError: # pragma: NO COVER
	# Fallback to pure-python RSA implementation if cryptography is
	# unavailable.
	from google.auth.crypt import _python_rsa
	RSASigner = _python_rsa.RSASigner
	RSAVerifier = _python_rsa.RSAVerifier

I'm wondering if the library does something incorrect with keys that only manifests in a multithreaded or multiprocess environment

[jamescooke]

Hi @busunkim96 , after installing cryptography and removing rsa, I get a new error when running the same code in our work project:

waluigi/tests/contrib/gcs/wa_gcs_client/test_remove.py:26: in full_dir_path
    wa_client.put_multiple(filepaths, dir_path, num_process=20)
venv/lib/python3.6/site-packages/luigi/contrib/gcs.py:280: in put_multiple
    return p.map(self._forward_args_to_put, put_kwargs_list)
/usr/local/lib/python3.6/multiprocessing/pool.py:266: in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
/usr/local/lib/python3.6/multiprocessing/pool.py:644: in get
    raise self._value
/usr/local/lib/python3.6/multiprocessing/pool.py:424: in _handle_tasks
    put(task)
/usr/local/lib/python3.6/multiprocessing/connection.py:206: in send
    self._send_bytes(_ForkingPickler.dumps(obj))
/usr/local/lib/python3.6/multiprocessing/reduction.py:51: in dumps
    cls(buf, protocol).dump(obj)
E   TypeError: can't pickle _cffi_backend.FFI objects

Some context - waluigi is our vendored wrapper around Luigi. We're using luigi.contrib.gcs's put_multiple() to upload multiple files in parallel to GCS.

It looks like the _cffi_backend.FFI objects are from https://github.com/cffi/cffi which is required by cryptography:

$ pip install cryptography
Requirement already satisfied: cryptography in ./venv/lib/python3.6/site-packages (3.4.6)
Requirement already satisfied: cffi>=1.12 in ./venv/lib/python3.6/site-packages (from cryptography) (1.14.4)
Requirement already satisfied: pycparser in ./venv/lib/python3.6/site-packages (from cffi>=1.12->cryptography) (2.20)

Hope that's helpful.

[busunkim96]

Thanks @jamescooke! It looks like that creates some other problems we'll have to look into separately.

I haven't tracked down what the cause is, but here is how a service account credential gets initialized, starting from from_service_account_file. Please holler if anything here looks odd. :)

google-auth-library-python/google/oauth2/service_account.py

Lines 221 to 236 in 65074d3

	@classmethod
	def from_service_account_file(cls, filename, **kwargs):
	"""Creates a Credentials instance from a service account json file.
	Args:
	filename (str): The path to the service account json file.
	kwargs: Additional arguments to pass to the constructor.
	Returns:
	google.auth.service_account.Credentials: The constructed
	credentials.
	"""
	info, signer = _service_account_info.from_filename(
	filename, require=["client_email", "token_uri"]
	)
	return cls._from_signer_and_info(signer, info, **kwargs)

google-auth-library-python/google/auth/_service_account_info.py

Lines 60 to 74 in 65074d3

	def from_filename(filename, require=None):
	"""Reads a Google service account JSON file and returns its parsed info.
	Args:
	filename (str): The path to the service account .json file.
	require (Sequence[str]): List of keys required to be present in the
	info.
	Returns:
	Tuple[ Mapping[str, str], google.auth.crypt.Signer ]: The verified
	info and a signer instance.
	"""
	with io.open(filename, "r", encoding="utf-8") as json_file:
	data = json.load(json_file)
	return data, from_dict(data, require=require)

google-auth-library-python/google/auth/_service_account_info.py

Lines 25 to 57 in 65074d3

	def from_dict(data, require=None):
	"""Validates a dictionary containing Google service account data.
	Creates and returns a :class:`google.auth.crypt.Signer` instance from the
	private key specified in the data.
	Args:
	data (Mapping[str, str]): The service account data
	require (Sequence[str]): List of keys required to be present in the
	info.
	Returns:
	google.auth.crypt.Signer: A signer created from the private key in the
	service account file.
	Raises:
	ValueError: if the data was in the wrong format, or if one of the
	required keys is missing.
	"""
	keys_needed = set(require if require is not None else [])
	missing = keys_needed.difference(six.iterkeys(data))
	if missing:
	raise ValueError(
	"Service account info was not in the expected format, missing "
	"fields {}.".format(", ".join(missing))
	)
	# Create a signer.
	signer = crypt.RSASigner.from_service_account_info(data)
	return signer

google-auth-library-python/google/auth/crypt/base.py

Lines 94 to 115 in 65074d3

	def from_service_account_info(cls, info):
	"""Creates a Signer instance instance from a dictionary containing
	service account info in Google format.
	Args:
	info (Mapping[str, str]): The service account info in Google
	format.
	Returns:
	google.auth.crypt.Signer: The constructed signer.
	Raises:
	ValueError: If the info is not in the expected format.
	"""
	if _JSON_FILE_PRIVATE_KEY not in info:
	raise ValueError(
	"The private_key field was not found in the service account " "info."
	)
	return cls.from_string(
	info[_JSON_FILE_PRIVATE_KEY], info.get(_JSON_FILE_PRIVATE_KEY_ID)
	)

google-auth-library-python/google/auth/crypt/_python_rsa.py

Lines 138 to 173 in 65074d3

	@classmethod
	def from_string(cls, key, key_id=None):
	"""Construct an Signer instance from a private key in PEM format.
	Args:
	key (str): Private key in PEM format.
	key_id (str): An optional key id used to identify the private key.
	Returns:
	google.auth.crypt.Signer: The constructed signer.
	Raises:
	ValueError: If the key cannot be parsed as PKCS#1 or PKCS#8 in
	PEM format.
	"""
	key = _helpers.from_bytes(key) # PEM expects str in Python 3
	marker_id, key_bytes = pem.readPemBlocksFromFile(
	six.StringIO(key), _PKCS1_MARKER, _PKCS8_MARKER
	)
	# Key is in pkcs1 format.
	if marker_id == 0:
	private_key = rsa.key.PrivateKey.load_pkcs1(key_bytes, format="DER")
	# Key is in pkcs8.
	elif marker_id == 1:
	key_info, remaining = decoder.decode(key_bytes, asn1Spec=_PKCS8_SPEC)
	if remaining != b"":
	raise ValueError("Unused bytes", remaining)
	private_key_info = key_info.getComponentByName("privateKey")
	private_key = rsa.key.PrivateKey.load_pkcs1(
	private_key_info.asOctets(), format="DER"
	)
	else:
	raise ValueError("No key could be detected.")
	return cls(private_key, key_id=key_id)

As far as I can tell rsa.key.PrivateKey is being instantiated normally and we aren't using a different object for rsa.key.AbstractKey.

google-auth-library-python/google/auth/crypt/_python_rsa.py

Line 160 in 65074d3

private_key = rsa.key.PrivateKey.load_pkcs1(key_bytes, format="DER")

google-auth-library-python/google/auth/crypt/_python_rsa.py

Lines 167 to 168 in 65074d3

	private_key = rsa.key.PrivateKey.load_pkcs1(
	private_key_info.asOctets(), format="DER"

Continuing on to the RSA library:

• load_pkcs_1 calls PrivateKey._load_pkcs1_der
• Key created by creating instance of class: key=cls(*as_ints)
• PrivateKey.__init__ calls AbstractKey.__init__(self, n, e)
• AbstractKey.__init__ sets self.mutex <-- according to Possible bug introduced in 4.7 sybrenstuvel/python-rsa#173 (comment) this is not happening and results in an AttributeError when the key is used to sign by google.auth

Will update once I have a repro of sybrenstuvel/python-rsa#173 (comment)

[busunkim96]

PSA:

Anyone who was seeing "Invalid JWT Signature" with rsa==4.7 in a multithreaded context with one of the google-cloud-* client libraries, please upgrade to rsa==4.7.1 which fixes the issue.

---

For the multiprocessing error, I've managed to recreate the AttributeError: mutex. If I'm reading https://github.com/spotify/luigi/blob/31d98156a5e044d643184fd153677e828b2a6e3e/luigi/contrib/gcs.py#L278-L281 correctly it looks like it shares one client across multiple processes. Google Cloud libraries like google-cloud-bigquery explicitly fail when you attempt to do pass the client object to a pool with an error there is non-trivial state on the client and as a result it is not pickleable. I suspect that may also be the case for google-api-python-client, and it might be related to this error.

This is a contrived example that repeatedly calls refresh() on the client's credential.

requirements.txt

google-auth
google-api-python-client

from multiprocessing import Pool
from contextlib import closing

import google.auth
from google.auth.transport import requests
from googleapiclient import discovery


def refresh_credentials(client):
    client._http.credentials.refresh(requests.Request())

def with_multiprocessing():
    # Set GOOGLE_APPLICATION_CREDENTIALS to a service account file
    creds, _ = google.auth.default(
        scopes=["https://www.googleapis.com/auth/cloud-platform"]
    )
    
    client = discovery.build("storage", "v1", credentials=creds)
    num_process = 5

    with closing(Pool(num_process)) as p:
        return p.map(refresh_credentials, [client])

def no_multiprocessing():
    # Set GOOGLE_APPLICATION_CREDENTIALS to a service account file
    creds, _ = google.auth.default(
        scopes=["https://www.googleapis.com/auth/cloud-platform"]
    )
    client = discovery.build("storage", "v1", credentials=creds)

    refresh_credentials(client)    

if __name__ == "__main__":
    no_multiprocessing()
    with_multiprocessing()

(env) busunkim@busunkim:~/github/google-auth-library-python$ python3 repro.py 
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/google/home/busunkim/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/google/home/busunkim/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "repro.py", line 11, in refresh_credentials
    client._http.credentials.refresh(requests.Request())
  File "/usr/local/google/home/busunkim/github/google-auth-library-python/google/oauth2/service_account.py", line 375, in refresh
    assertion = self._make_authorization_grant_assertion()
  File "/usr/local/google/home/busunkim/github/google-auth-library-python/google/oauth2/service_account.py", line 364, in _make_authorization_grant_assertion
    token = jwt.encode(self._signer, payload)
  File "/usr/local/google/home/busunkim/github/google-auth-library-python/google/auth/jwt.py", line 112, in encode
    signature = signer.sign(signing_input)
  File "/usr/local/google/home/busunkim/github/google-auth-library-python/google/auth/crypt/_python_rsa.py", line 136, in sign
    return rsa.pkcs1.sign(message, self._key, "SHA-256")
  File "/usr/local/google/home/busunkim/github/google-auth-library-python/env/lib/python3.8/site-packages/rsa/pkcs1.py", line 331, in sign
    return sign_hash(msg_hash, priv_key, hash_method)
  File "/usr/local/google/home/busunkim/github/google-auth-library-python/env/lib/python3.8/site-packages/rsa/pkcs1.py", line 306, in sign_hash
    encrypted = priv_key.blinded_encrypt(payload)
  File "/usr/local/google/home/busunkim/github/google-auth-library-python/env/lib/python3.8/site-packages/rsa/key.py", line 477, in blinded_encrypt
    blinded, blindfac_inverse = self.blind(message)
  File "/usr/local/google/home/busunkim/github/google-auth-library-python/env/lib/python3.8/site-packages/rsa/key.py", line 167, in blind
    blindfac, blindfac_inverse = self._update_blinding_factor()
  File "/usr/local/google/home/busunkim/github/google-auth-library-python/env/lib/python3.8/site-packages/rsa/key.py", line 203, in _update_blinding_factor
    with self.mutex:
AttributeError: mutex
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "repro.py", line 36, in <module>
    with_multiprocessing()
  File "repro.py", line 23, in with_multiprocessing
    return p.map(refresh_credentials, [client])
  File "/usr/local/google/home/busunkim/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/local/google/home/busunkim/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
AttributeError: mutex

[busunkim96]

PrivateKeyseems to be missing self.mutex after it's unpickled. Calling AbstractKey.__init() in PrivateKey.__setstate__() seems to fix it for my example.

    def __setstate__(self, state: typing.Tuple[int, int, int, int, int, int, int, int]) -> None:
        """Sets the key from tuple."""
        self.n, self.e, self.d, self.p, self.q, self.exp1, self.exp2, self.coef = state
        AbstractKey.__init__(self, self.n, self.e)

[busunkim96]

Opened sybrenstuvel/python-rsa#178

[jamescooke]

@busunkim96 Thanks for the PR above. rsa==4.7.2 has been released by @sybrenstuvel and is working in our project with multiprocessing and luigi 👍🏻

[busunkim96]

Hooray!

It looks like this is resolved now. Anyone who was pinning to an older version, please upgrade to rsa==4.7.2. :)

[sybrenstuvel]

Thanks for collaborating on this, people! 🥳 🎈