Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serverless framework support #2

Closed
datar-ai opened this issue Oct 26, 2017 · 12 comments
Closed

Serverless framework support #2

datar-ai opened this issue Oct 26, 2017 · 12 comments
Assignees

Comments

@datar-ai
Copy link

When using aws-xray-sdk 0.92 on Lambda to trace a Flask application, adding the Flask middleware results in this log and traceback.

[1509032914734] Traceback (most recent call last):
[1509032914734] File "/var/task/flask/app.py", line 1982, in wsgi_app
[1509032914734] response = self.full_dispatch_request()
[1509032914734] File "/var/task/flask/app.py", line 1615, in full_dispatch_request
[1509032914734] return self.finalize_request(rv)
[1509032914734] File "/var/task/flask/app.py", line 1632, in finalize_request
[1509032914734] response = self.process_response(response)
[1509032914734] File "/var/task/flask/app.py", line 1856, in process_response
[1509032914734] response = handler(response)
[1509032914734] File "/var/task/aws_xray_sdk/ext/flask/middleware.py", line 59, in _after_request
[1509032914734] segment.put_http_meta(http.STATUS, response.status_code)
[1509032914734] File "/var/task/aws_xray_sdk/core/models/facade_segment.py", line 42, in put_http_meta
[1509032914734] raise FacadeSegmentMutationException(MUTATION_UNSUPPORTED_MESSAGE)
[1509032914734] aws_xray_sdk.core.exceptions.exceptions.FacadeSegmentMutationException: FacadeSegments cannot be mutated.
[1509032914734] FacadeSegments cannot be mutated.

Please check the forums for the detail info :
https://forums.aws.amazon.com/thread.jspa?messageID=803898&tstart=0

@haotianw465
Copy link
Contributor

Hi, which serverless framework are you using?

@datar-ai
Copy link
Author

I use zappa , and python flask framework.

@haotianw465
Copy link
Contributor

The long term solution is to have the middleware directly add all http metadata to the root segment generated by Lambda worker. The SDK currently doesn't have access to this segment right now and we don't have an ETA yet.

The workaround is to have middleware to only generate subsegments when running on Lambda. This causes a problem in the service graph. The color of the Lambda function node in the service graph is decided by status code in lambda segments. And this status code only indicates if the lambda function runs or not. It doesn't respect the actual status code returned by your web application running within that lambda function. If you have correct IAM role setup and no syntax error on your application code, most likely you will always see an all green service graph node despite a lot of 4xx returned by your web app.

This workaround makes it difficult to discover issues and we cannot call this workaround "support for serverless". But we are open to here any suggestions. If such workaround is acceptable we can make it an opt-in so that at least FacadeSegmentMutationException is avoided.

@haotianw465 haotianw465 changed the title Python, Lambda and Flask: FacadeSegments cannot be mutated Serverless framework support Nov 7, 2017
@Sniedes722
Copy link

patching the libraries seemed to solve this issue for me, also using lambda, flask, & zappa.

from aws_xray_sdk.core import xray_recorder, patch_all
from aws_xray_sdk.core.context import Context
from aws_xray_sdk.ext.flask.middleware import XRayMiddleware

patch_all()

and then this was how i wrapped it inside the flask application:

xray_recorder.configure(service='Your Service Name', sampling=False, context=Context())
XRayMiddleware(app, xray_recorder)

@Sniedes722
Copy link

also enabled in zappa config checkout: https://github.com/Miserlou/Zappa#aws-x-ray

@iandees
Copy link

iandees commented Jan 28, 2018

For others running into this issue, @Sniedes722's combination of patching and enabling x-ray in Zappa worked for me. I'm not sure why, but it didn't work until I added the sampling=False, context=Context() stuff.

@Sniedes722
Copy link

@iandees I can also confirm that @haotianw465 is completely correct about getting green OK status codes in x-ray when the API is returning 400 statuses with this workaround

@xplorld
Copy link
Contributor

xplorld commented Jun 8, 2018

@haotianw465 What are the consequences if we use our own Context() instead of using SDK-provided Lambda context?

@haotianw465
Copy link
Contributor

@xplorld the recorder class intentionally exposes an API to allow users to configure their own context management https://github.com/aws/aws-xray-sdk-python/blob/master/aws_xray_sdk/core/recorder.py#L75.

You can have your own implementation of the context() used in Lambda container which creates another type of facadesegment which can be modified freely and replicate everything it has to a child subsegment.

Please let me know if you need any help on implementing a customized context management that specifically fits your use case.

@haotianw465
Copy link
Contributor

Hi all, API Gateway with active X-Ray integration is launched. You can see more details here: https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-xray.html.

Feel free to leave any feedback on the new customer experience and we are happy to discuss a workaround based on those new feedback. We will also be working with Lambda team based on those new feedback to better improve the overall experience. Thank you for your patience.

@ghost
Copy link

ghost commented Oct 18, 2018

I see the a dangling subsegment even when calling configure before patching third party libraries.

from`aws_xray_sdk.core import xray_recorder, patch_all
from aws_xray_sdk.core.context import Context
from aws_xray_sdk.ext.flask.middleware import XRayMiddleware

app = connexion.App(__name__, specification_dir='./swagger/')
app.app.json_encoder = encoder.JSONEncoder
app.add_api('swagger.yaml', arguments={'title': 'Swagger Petstore'})

xray_recorder.configure(service=' python-flask-se-development', sampling=False, context=Context())
XRayMiddleware(app.app, xray_recorder)

patch_all()

chanchiem added a commit to chanchiem/aws-xray-sdk-python that referenced this issue Feb 8, 2019
    * Serverless architecture in this case includes one that utilizes
      Lambda and API Gateway.
    * A new "Serverless" context is created to give the abstraction of
      Segments being the toplevel entities but is then converted to a
      subsegment upon transmission to the data plane.
      These segments are called MimicSegments.  All generated
      segments have a parent segment that is the FacadeSegment.
    * Currently supports Flask and Django as middlewares; this has been
      confirmed to be natively working with Zappa if the application is
      running under Flask/Django.
chanchiem added a commit to chanchiem/aws-xray-sdk-python that referenced this issue Feb 8, 2019
    * Serverless architecture in this case includes one that utilizes
      Lambda and API Gateway.
    * A new "Serverless" context is created to give the abstraction of
      Segments being the toplevel entities but is then converted to a
      subsegment upon transmission to the data plane.
      These segments are called MimicSegments.  All generated
      segments have a parent segment that is the FacadeSegment.
    * Currently supports Flask and Django as middlewares; this has been
      confirmed to be natively working with Zappa if the application is
      running under Flask/Django.
chanchiem added a commit to chanchiem/aws-xray-sdk-python that referenced this issue Feb 14, 2019
    * Serverless architecture in this case includes one that utilizes
      Lambda and API Gateway.
    * A new "Serverless" context is created to give the abstraction of
      Segments being the toplevel entities but is then converted to a
      subsegment upon transmission to the data plane.
      These segments are called MimicSegments.  All generated
      segments have a parent segment that is the FacadeSegment.
    * Currently supports Flask and Django as middlewares; this has been
      confirmed to be natively working with Zappa if the application is
      running under Flask/Django.
chanchiem added a commit to chanchiem/aws-xray-sdk-python that referenced this issue Feb 20, 2019
    * Serverless architecture in this case includes one that utilizes
      Lambda and API Gateway.
    * A new "Serverless" context is created to give the abstraction of
      Segments being the toplevel entities but is then converted to a
      subsegment upon transmission to the data plane.
      These segments are called MimicSegments.  All generated
      segments have a parent segment that is the FacadeSegment.
    * Currently supports Flask and Django as middlewares; this has been
      confirmed to be natively working with Zappa if the application is
      running under Flask/Django.
@chanchiem
Copy link
Contributor

Good news! Serverless support has been merged into the master branch!
#127

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants