Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add retries when creating boto3 client #33

Merged
merged 1 commit into from
May 3, 2020

Conversation

donnlee
Copy link
Contributor

@donnlee donnlee commented May 2, 2020

Fixes #12
This diff adds a retry loop when we attempt to create the boto3 client client.
Also creates a new amazon_rekognition config option boto_retries which is optional and defaults to 5 retries.

I'm currently using this patch in my home setup.
Here's the dev testing that was done:

  • Tested with boto_retries: 0 (equivalent to how things worked before this diff).

With boto_retries set to 0, restarted HA until boto3.client() failed and exception was thrown. Output of Dev Tools > Logs is what we expect:

Error while setting up amazon_rekognition platform for image_processing
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 178, in _async_setup_platform
    await asyncio.wait_for(asyncio.shield(task), SLOW_SETUP_MAX_WAIT)
  File "/usr/local/lib/python3.7/asyncio/tasks.py", line 442, in wait_for
    return fut.result()
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/amazon_rekognition/image_processing.py", line 124, in setup_platform
    "the boto_retries setting. Retry counter was {}".format(retries))
Exception: Failed to create boto3 client. Maybe try increasing the boto_retries setting. Retry counter was 1
  • Tested with boto_retries omitted from the amazon_rekognition config: Logging at level debug shows boto_retries is 5 (the default).

Restarted HA until a boto3.client() failure was encountered and retry was successful (boto_retries=5):

bash-5.0# cat home-assistant.log | grep rek
2020-05-02 16:14:32 WARNING (MainThread) [homeassistant.loader] You are using a custom integration for amazon_rekognition which has not been tested by Home Assistant. This
component might cause stability problems, be sure to disable it if you experience issues with Home Assistant.
2020-05-02 16:14:32 INFO (MainThread) [homeassistant.components.image_processing] Setting up image_processing.amazon_rekognition
2020-05-02 16:14:32 INFO (MainThread) [homeassistant.components.image_processing] Setting up image_processing.amazon_rekognition
2020-05-02 16:14:32 INFO (MainThread) [homeassistant.components.image_processing] Setting up image_processing.amazon_rekognition
2020-05-02 16:14:32 INFO (SyncWorker_13) [custom_components.amazon_rekognition.image_processing] boto3 client failed, retries=0
2020-05-02 16:14:32 INFO (SyncWorker_33) [custom_components.amazon_rekognition.image_processing] boto3 client failed, retries=0

Same test but with logging level set to debug. One camera was configured with boto_retries: 4, rest did not have a boto_retries yaml line:

bash-5.0# cat home-assistant.log | grep rek
2020-05-02 16:17:00 WARNING (MainThread) [homeassistant.loader] You are using a custom integration for amazon_rekognition which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant.
2020-05-02 16:17:00 INFO (MainThread) [homeassistant.components.image_processing] Setting up image_processing.amazon_rekognition
2020-05-02 16:17:00 INFO (MainThread) [homeassistant.components.image_processing] Setting up image_processing.amazon_rekognition
2020-05-02 16:17:00 INFO (MainThread) [homeassistant.components.image_processing] Setting up image_processing.amazon_rekognition
2020-05-02 16:17:00 DEBUG (SyncWorker_0) [custom_components.amazon_rekognition.image_processing] boto_retries setting is 4
2020-05-02 16:17:00 DEBUG (SyncWorker_4) [custom_components.amazon_rekognition.image_processing] boto_retries setting is 5
2020-05-02 16:17:00 DEBUG (SyncWorker_33) [custom_components.amazon_rekognition.image_processing] boto_retries setting is 5
2020-05-02 16:17:00 INFO (SyncWorker_33) [custom_components.amazon_rekognition.image_processing] boto3 client failed, retries=0
bash-5.0#

@robmarkcole
Copy link
Owner

many thanks for this! Lets leave it undocument in the readme for now, as it probably should be for internal use by the integration but may be useful as a config option for debugging

@robmarkcole robmarkcole merged commit 4d12033 into robmarkcole:master May 3, 2020
@donnlee donnlee deleted the add_boto_client_retries branch May 3, 2020 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

KeyError: ‘endpoint_resolver’ when using multiple cameras
2 participants