-
Notifications
You must be signed in to change notification settings - Fork 20
-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Is Error Handling example upto date? #170
Comments
Thanks @ankurchavda, can you confirm which docs URL you're following, and which SDK/Server versions you're on, and we'll take a look! |
@osfameron The one on the official website - https://docs.couchbase.com/python-sdk/current/howtos/error-handling.html. And the github code for this is docs-sdk-python/modules/howtos/examples/error-handling.py Lines 154 to 198 in a1bd25c
|
OK, that page definitely needs some update, thanks for spotting. I've opened https://issues.couchbase.com/browse/DOC-9281 to track this internally, and have added to this sprint. In the meantime:
So it looks like you're good to go! Let me know if you have any other questions. |
Thanks for the quick action @osfameron. I am trying to add proper error handling to my code. Can you please check if the below code looks good? def couchbase_connection(username, password, host, bucket='bucketname', max_retries=5, back_off=0.1):
"""
Create couchbase connection
Parameters:
username: user id
password: password
host: connection string
bucket (string): Name of the couchbase bucket
max_retries (int): An integer
back_off (int): An integer
Returns:
collection (couchbase.collections): A couchbase collection object
"""
try:
log.info("Creating couchbase connection.")
# Setting timeout options for cluster object. kv_timeout is for key-value operations
# like get() and other mutation commands. Simlarly query_timeout is for N1QL queries.
options = ClusterTimeoutOptions(kv_timeout=timedelta(
seconds=60), query_timeout=timedelta(seconds=120))
cluster = Cluster('couchbase://' + host, ClusterOptions(
PasswordAuthenticator(username, password), timeout_options=options))
bucket = cluster.bucket('matrixDB')
collection = bucket.default_collection()
log.info("Connection created.")
return collection
# Transient errors are something that resolve automatically. is_retyable is provided
# for such errors. So have added a retry mechanism if the error is retryable.
except couchbase_exceptions.CouchbaseTransientException as err:
log.error(f"Encountered a transient error: {err}")
if max_retries > 0:
log.info(f"Retries left: {max_retries}")
log.info(f"Backing Off: {back_off} seconds")
sleep(back_off)
couchbase_connection(username, password, bucket,
max_retries=max_retries-1, back_off=back_off*2)
else:
log.error(
f"Too many attempts to create connection with couchbase: {err}")
raise
# catch generic couchbase exception
except couchbase_exceptions.CouchbaseException as err:
log.error(f"Unexpected error {err}") So I have written the above function to create a couchbase connection. I catch transient errors and retry couchbase creation there. So should I also add is_transient check under |
I'll speak to our Python SDK specialist soon hopefully and get them to look over it (which will also help me improve and update the documentation!) |
Sure that would be great, thank you! |
Hi @ankurchavda - The is_retryable and is_transient properties are not actually available within the 3.x SDK. The doc strings need to be updated in order to reflect what is actually available (I will look to create a ticket to track these changes accordingly). When it comes to retrying, definitely can retry on the Something I would note with your current solution is that maybe you want to make a retry function that can behave as a decorator, that way you can reuse the retry mechanism for other operations. Also, the couchbase_connection() call in the exception path does not return anything. Having the connection wrapped via a decorator would also help to solve this problem. I will try to provide a sample of what I mean by tomorrow. |
Hi @thejcfactor - The Authentication exception came in despite the credentials being correct. Also, it was resolved on a retry, but I have to specifically catch via Also, it would great if you can share the sample of what you are trying to suggest. |
Hmmm, there seems to be more going on in terms of the overall cluster setup if you are seeing intermittent connection issues. I would suggest trying out SDK doctor to help diagnose the connection. Below is some sample code that demonstrates what I am talking about. You can apply the allow_retries decorator to any method that you would like to have retries. Also, you can also change the decorator signature to have more functionality (for instance pass in a dict that has exception types mapped to a function on how you want to handle the exception type...lots of possibilities :)). import functools
import time
from datetime import timedelta
from typing import Optional, Tuple, Callable
from couchbase.cluster import Cluster, ClusterOptions, ClusterTimeoutOptions
from couchbase.auth import PasswordAuthenticator
from couchbase.exceptions import AuthenticationException, CouchbaseTransientException
def allow_retries(retry_limit=3, # type: int
backoff=1.0, # type: float
exponential_backoff=False, # type: bool
allowed_exceptions=None # type: Optional[Tuple]
) -> Callable:
def handle_retries(func):
@functools.wraps(func)
def func_wrapper(*args, **kwargs):
for i in range(retry_limit):
try:
return func(*args, **kwargs)
except Exception as ex:
if allowed_exceptions is None or not isinstance(ex, allowed_exceptions):
raise
if (retry_limit-1)-i == 0:
raise
delay = backoff
if exponential_backoff is True:
delay *= i
print(f"Retries left: {(retry_limit-1) - i}")
print(f"Backing Off: {delay} seconds")
time.sleep(delay)
return func_wrapper
return handle_retries
@allow_retries(retry_limit=5,
backoff=0.5,
allowed_exceptions=(CouchbaseTransientException, AuthenticationException))
def couchbase_connect(host, # type: str
username, # type: str
pw, # type: str
bucket_name # type: str
) -> Tuple:
options = ClusterTimeoutOptions(kv_timeout=timedelta(seconds=60),
query_timeout=timedelta(seconds=120))
cluster = Cluster(host, ClusterOptions(
PasswordAuthenticator(username, pw), timeout_options=options))
bucket = cluster.bucket(bucket_name)
collection = bucket.default_collection()
return cluster, bucket, collection
def run_sample_code():
try:
_, _, collection = couchbase_connect("couchbase://localhost",
"Administrator", "password", "default")
_ = collection.upsert("testKey", {"info": "is a test doc"})
res = collection.get("testKey")
print(f"Got: {res.content_as[dict]}")
except Exception as ex:
import traceback
traceback.print_exc()
if __name__ == '__main__':
run_sample_code() |
Hi @thejcfactor, Sadly can't use SDK Doctor for the time being. But the code that you shared is very helpful, have added this decorator in my code for retrying after exceptions. Will let you know how the retry robustness changes after using this but the code definitely has become cleaner. Thanks. |
Hi @thejcfactor - I used the base exception
The I was expecting that since |
Hi @ankurchavda - I apologize for the delayed response (something is up w/ my notifications). I imagine you have figured out how to alter the helper method already. Just to close the loop, you would either need to pass in the specific exceptions if you are wanting to allow when using isinstance(), otherwise you could use issubclass(). I would lean on the side of being more explicit and passing in a tuple of all the errors you want to allow. IMHO, that would allow the code to be more clear. |
Hi @thejcfactor, no problem at all. Yes, I did figure it out eventually. Just as you suggested, I passed the errors explicitly that I wanted to handle. The program is working fine now. Thanks for all the help :D. Btw any updates on when the official documentation is going to be updated? |
Hi @ankurchavda - Good to hear! I am planning to make updates to the docs within the coming weeks (prior to the US holidays). |
Hi @thejcfactor , Hope you are having a nice time :D Apologies for asking a different question under this issue, I don't see an option of opening an issue in the Python SDK repo. I wanted to understand a little more about the I set the option as 120 seconds in the
Why the whichever is lower? Shouldn't the preference ideally be given to the lowest level of setting i.e. timeout settings in that query options? In this way, I might never be able to specify different timeouts for different queries if it exceeds the global timeout. Also, I set I am using Also, do let me know if there is a better place to raise this issue, will be happy to do it :D |
Initial error is closed and merged, should appear on our docs site at next build. @ankurchavda sorry just noted your last question - is that still an open issue? If you still have an issue, you could open a new ticket, or as this looks like perhaps more of a discussion point, then https://forums.couchbase.com/ might be appropriate. |
is_retryable does not seem to be available for usage. I can see is_transient. Should we use that or am I missing something?
Even the exception caught is
coucbase.exceptions.CouchbaseError
, hasn't this changed tocoucbase.exceptions.CouchbaseException
?The text was updated successfully, but these errors were encountered: