make a decorator for lambda functions #191

rokob · 2017-09-05T21:15:23Z

This implements a decorator lambda_function to be used like

import os
import rollbar

token = os.getenv('ROLLBAR_KEY', 'missing_api_key')
rollbar.init(token, 'production')

@rollbar.lambda_function
def lambda_handler(event, context):
    raise Exception('Hello threaded wrapped Rollbar!')
    return 'Hello from Lambda'

If we comment out the raise Exception line then everything will just run normally and the rollbar code will pass things through as necessary. If we do have an exception then Rollbar will handle sending it to the API and waiting for any processing threads as necessary. If the handler happened to be set to another value like 'blocking' then the wait is superfluous but otherwise the same outcome will be observed.

rokob · 2017-09-06T21:40:43Z

Fixes #192

rokob · 2017-09-06T21:41:39Z

rollbar/__init__.py

@@ -332,9 +339,28 @@ def init(access_token, environment='production', **kw):
                                   **SETTINGS['locals']['sizes'])
    _transforms.append(shortener)

+    _threads = queue.Queue()


Queue is thread safe

rokob · 2017-09-06T21:42:06Z

rollbar/__init__.py

@@ -415,6 +441,7 @@ def send_payload(payload, access_token):
    else:
        # default to 'thread'
        thread = threading.Thread(target=_send_payload, args=(payload, access_token))
+        _threads.put(thread)


Add the thread to the queue when it is created, before it starts running

Good idea. I think this is actually going to be a great addition to the non-lambda use cases. Is there an easy way of using a thread pool here instead of creating a new thread for each call?

rokob · 2017-09-06T21:42:42Z

rollbar/__init__.py

@@ -445,6 +472,12 @@ def search_items(title, return_fields=None, access_token=None, endpoint=None, **
                    **search_fields)


+def wait(f=None):
+    _threads.join()


This join will return when all task_done is called on the queue the same number of times that put has been called. It will block the calling thread, which cannot be any of the threads in the queue that need to call task_done because we control all those threads and they only execute _send_payload

If we are not using threads, e.g. using the blocking handler, then _threads will be empty and join will return right away.

rokob · 2017-09-06T21:44:08Z

rollbar/__init__.py

+def wait(f=None):
+    _threads.join()
+    if f is not None:
+        return f()


This allows wait to return a value which we use to return the value at the end of the lambda function

rokob · 2017-09-06T21:46:17Z

rollbar/__init__.py

@@ -1133,6 +1166,11 @@ def _send_payload(payload, access_token):
        _post_api('item/', payload, access_token=access_token)
    except Exception as e:
        log.exception('Exception while posting item %r', e)
+    try:


If we are using threads this will be called on one of the threads created for sending a payload. get_nowait will remove a thread from the queue, it doesn't really matter which one it removes as long as each thread calls this exactly once because we don't do anything with the returned value. Again, since we are on one of the threads, calling task_done here can signal to join to return.

If we are not using threads, i.e. any other handler, then _threads will just always be empty and get_nowait will throw queue.Empty and we just move on.

I still don't understand the need for removing one of the threads here. What would happen if this code were not here?

…t if possible

rokob · 2017-09-08T18:47:49Z

If you use the decorator, you now also get data added the custom field from the lambda context. I also did in this for Javascript, it is an easy thing to add, and it probably adds some value.

coryvirok · 2017-09-08T18:45:36Z

rollbar/__init__.py

@@ -415,6 +441,7 @@ def send_payload(payload, access_token):
    else:
        # default to 'thread'
        thread = threading.Thread(target=_send_payload, args=(payload, access_token))
+        _threads.put(thread)


Good idea. I think this is actually going to be a great addition to the non-lambda use cases. Is there an easy way of using a thread pool here instead of creating a new thread for each call?

coryvirok · 2017-09-08T18:48:17Z

rollbar/__init__.py

@@ -1133,6 +1166,11 @@ def _send_payload(payload, access_token):
        _post_api('item/', payload, access_token=access_token)
    except Exception as e:
        log.exception('Exception while posting item %r', e)
+    try:


I still don't understand the need for removing one of the threads here. What would happen if this code were not here?

coryvirok · 2017-09-08T18:49:01Z

rollbar/__init__.py

+    @functools.wraps(f)
+    def wrapper(event, context):
+        global _CURRENT_LAMBDA_CONTEXT
+        _CURRENT_LAMBDA_CONTEXT = context


If lambda can run Python threads then this global is going to need to be a thread local.

rokob added 2 commits September 5, 2017 14:14

make a decorator for lambda functions

088d638

implement thread tracking and a wait function to join on them

32874d2

rokob commented Sep 6, 2017

View reviewed changes

rokob requested review from coryvirok and ezarowny September 6, 2017 22:22

rokob added 3 commits September 8, 2017 11:27

attempt to enrich log/error messages with data from the lambda contex…

19fb654

…t if possible

python is not ruby

c9f718d

the function is called get_remaining..., not just remaining...

713ff46

coryvirok reviewed Sep 8, 2017

View reviewed changes

rokob merged commit b4ff85d into master Sep 8, 2017

This was referenced Sep 13, 2017

Update rollbar to 0.13.13 FlowFX/unkenmathe.de#214

Merged

Update rollbar to 0.13.13 michaelmior/pylinks#119

Merged

Update rollbar to 0.13.13 psykzz/flask-rollbar#24

Merged

Scheduled weekly dependency update for week 38 quanted/qed#103

Closed

pyup-bot mentioned this pull request Sep 25, 2017

Scheduled weekly dependency update for week 39 quanted/qed#104

Merged

rokob deleted the lambda-support branch October 25, 2017 21:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make a decorator for lambda functions #191

make a decorator for lambda functions #191

rokob commented Sep 5, 2017 •

edited

Loading

rokob commented Sep 6, 2017

rokob Sep 6, 2017

rokob Sep 6, 2017

coryvirok Sep 8, 2017

rokob Sep 6, 2017 •

edited

Loading

rokob Sep 6, 2017

rokob Sep 6, 2017

coryvirok Sep 8, 2017

rokob commented Sep 8, 2017 •

edited

Loading

coryvirok Sep 8, 2017

coryvirok Sep 8, 2017

coryvirok Sep 8, 2017

make a decorator for lambda functions #191

make a decorator for lambda functions #191

Conversation

rokob commented Sep 5, 2017 • edited Loading

rokob commented Sep 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rokob Sep 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rokob commented Sep 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rokob commented Sep 5, 2017 •

edited

Loading

rokob Sep 6, 2017 •

edited

Loading

rokob commented Sep 8, 2017 •

edited

Loading