KeyError: 'endpoint_resolver'

[CosmicAnalogue465]

Hi,

I sometimes get that error trying to call a lambda function
boto3==1.3.1

def lambda_execute(payload):
    import boto3
    client = boto3.client('lambda', aws_access_key_id=KEY, aws_secret_access_key=SECRET region_name=REGION)
    client.invoke(**payload)

payload is in this format:

{'FunctionName': fct, 'InvocationType': 'Event', 'LogType': 'None', 'Payload': simplejson.dumps(payload, default=encode_model)}

error seems to be coming from get_component in botocore/session.py

Can you help ?

[jamesls]

Is this function being called in multiple threads? The only way I see that happening is if boto3.client is being invoked in multiple threads (client creation is not thread safe).

[CosmicAnalogue465]

Hi James,
Yes it is - weird it hasn't been raised before.
would you recommend this as a solution? https://geekpete.com/blog/multithreading-boto3/

[JordonPhillips]

@vinczente Yes, each thread should have its own session.

[cpury]

I'm getting this in one case out of tens of thousands, but consistently in this case. I'm already creating a resource per thread... My code inside the thread is:

awslambda = boto3.client('lambda')

This happens inside a lambda function (it invokes other lambda functions). The exact error is:

Error in thread: Traceback (most recent call last):
  File "/var/task/service.py", line 78, in call_recursively
    awslambda = boto3.client('lambda')
  File "/var/runtime/boto3/__init__.py", line 83, in client
    return _get_default_session().client(*args, **kwargs)
  File "/var/runtime/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/var/runtime/botocore/session.py", line 826, in create_client
    endpoint_resolver = self.get_component('endpoint_resolver')
  File "/var/runtime/botocore/session.py", line 701, in get_component
    return self._components.get_component(name)
  File "/var/runtime/botocore/session.py", line 901, in get_component
    del self._deferred[name]
KeyError: 'endpoint_resolver'

I tried replacing it with

session = boto3.session.Session()
awslambda = session.resource('lambda')

But this causes:

Error in thread (p. 10-18): Traceback (most recent call last):
  File "/var/task/service.py", line 79, in call_recursively
    awslambda = session.resource('lambda')
  File "/var/runtime/boto3/session.py", line 347, in resource
    has_low_level_client)
ResourceNotExistsError: The 'lambda' resource does not exist.
The available resources are:
   - cloudformation
   - cloudwatch
   - dynamodb
   - ec2
   - glacier
   - iam
   - opsworks
   - s3
   - sns
   - sqs

Consider using a boto3.client('lambda') instead of a resource for 'lambda'

Any other options to try?

[cpury]

So it seems you can only create lambda clients using the default session? I'm not allowed to get it from a custom session, and boto3.client(...) always uses the default session. Why?

[CosmicAnalogue465]

@cpury Looking back at my code, I cannot remember why I didn't go with the session solution (perf? maybe) but what I have now is (simplified):

lambda_client = boto3.client('lambda', aws_access_key_id='xxx', aws_secret_access_key='xxx', region_name='eu-west-1')

def execute_async(fct, payload):
        threading.Thread(target=lambda_execute({'FunctionName': fct, 'InvocationType': 'Event', 'LogType': 'None', 'Payload': simplejson.dumps(payload, default=encode_model)}))

def lambda_execute(payload):
    lambda_client.invoke(**payload)

then the code is calling execute_async
Hope that helps ?

[cpury]

@vinczente thanks for your code! So you're using a shared client for all threads?

My code is similar, but more complex. I'm using a ThreadPool, though. And I create the client in each thread.

I should have emphasized more clearly: My code works in 99.9999% of all cases! The lambda function (that calls other lambda functions) has been run on tens of thousands of files as inputs, and never failed. There is exactly one file where this error consistently appears.

I assume there are weird edge cases where this can happen, and you might run into it yourself at some point :(

So it would be great if at least there was a way to create a lambda client that's not using the default session, as that would likely fix this.

[cpury]

Wow, super weird! Inspired by your code, I changed mine to use a global lambda client object that's shared among all threads. AKA, do exactly the opposite of what's recommended here in this ticket. And that fixes it!

So for anyone running into the same issue: Just use a global lambda object. And if anyone of the boto3 team wants to explain how this makes sense, please do, I'm curious :)

[khuevu]

@cpury , I observed the same issue too. It gave the exception when I use separate client per thread and only works if I use one global client.

[junghoon-picwell]

@cpury: I ran into the same issue as well. I wasn't comfortable creating a global client due to this documentation, so I tried until I get a client, e.g.

    client = None
    while not client:
        try:
            client = boto3.client('lambda')
        except:
            client = None

This seems to work.

[cpury]

@junghoon-picwell that looks scary, but great that it works! I've changed jobs so can't confirm if that fixes our case.

[junghoon-picwell]

Yes, it does make me uncomfortable. I introduced retry limits now, but I don't think it is still good. Any thoughts?

[claweyenuk]

Is this issue fixed after #73?

I hit this exception, and thought threading might be the issue, but it might have been a false alarm. It turned out the proxy server I was using went down. When I switched to a working proxy server, the KeyError: 'endpoint_resolver' went away.

I was using threads, and I haven't yet had a chance to dig into whether pointing to a bad proxy server while not using threading triggers this.

[rkiyanchuk]

I observe similar behavior on v1.9.16, but the error is KeyError: 'credential_provider' and it depends on the number of workers in a ThreadPool. For me the error seems to happen only when the number of threads is more than half (n/2+1) of tasks.

Making client global also "fixes" the error.

#1592 seems to be relevant.

[dineshmc2]

Ideal fix would be

session = boto3.session.Session()
awslambda = session.client('lambda')

(note that its not using sesion.resource() which is only applicable for small number of resources)

All the clients created using boto3.client()/boto3.resource() share default global session and thus aren't thread safe.

boto3/boto3/__init__.py

Lines 85 to 100 in c939956

	def client(*args, **kwargs):
	"""
	Create a low-level service client by name using the default session.
	See :py:meth:`boto3.session.Session.client`.
	"""
	return _get_default_session().client(*args, **kwargs)
	def resource(*args, **kwargs):
	"""
	Create a resource service client by name using the default session.
	See :py:meth:`boto3.session.Session.resource`.
	"""
	return _get_default_session().resource(*args, **kwargs)

[matse004]

The problem with the "global fix" for me is that with the shared lambda client I get about 50% more invocations than I have workers/threads.

Can't explain why that's happening, but a client per session fixes that. When it works :) And it often doesn't.

[farrellit]

We've run into this recently with python 2.7 and bot 1.10.16.

We think we likely need to work around it with uwsgi lazy loading setting toggle. but still looking through it.

[golharam]

I'm running into this now. I have a Flask application sitting behind Apache. When multiple clients are accessing a resource in the Flask app, that request triggers a boto3.client('s3').list_objects call, where this error comes up.
My app is using "boto3":"1.7.4" and "botocore":"1.10.6".

[cmcmillon]

Try using concurrent.futures

[zplizzi]

I'm still encountering this issue on boto3==1.17.41.

The

session = boto3.session.Session()
s3 = session.resource('s3')

fix seems to work, though.

[marcelo321]

What if I need to pass the key and secret in each call because there are different accounts? How do I do this? I can't put the boto3.client( before the loop since I will be calling just one account?

Is there any solution that works to my problem?

session.client('s3', aws_access_key_id=account, aws_secret_access_key=key, region_name=region)

I need to declare the id/key each time so can't do it globally as explained before

[rmzi]

We've run into this recently with python 2.7 and bot 1.10.16.

We think we likely need to work around it with uwsgi lazy loading setting toggle. but still looking through it.

I'm using uWSGI lazy-apps=true and still getting this issue. Running single process, multiple threads enabled, but not explicitly invoked in Django 2.2, Python 3.8.

[ahmedsadman]

My solution was creating a session for each thread and reusing that session in a particular thread:

# boto.py
from threading import local

thread_local_storage = local()

def get_s3_resource():
    if not hasattr(thread_local_storage, 'boto_session'):
        thread_local_storage.boto_session = boto3.Session()
    return thread_local_storage.boto_session.resource('s3')

# consumer.py
s3 = get_s3_resource()
# rest of the code