patch() throws error for `pymysql` in aws lambda

[aniljaiswal]

I have updated the requirements.txt with the latest version of the SDK i.e. v2.5.0 and also enabled tracing for pymysql, the support for which was added in the latest release (#214).

My x-ray config:

# Patch All third party libraries for AWS X-Ray
global_sdk_config.set_sdk_enabled(env.X_RAY_SDK_ENABLED)
xray_recorder.configure(service=lambda_name)
libs_to_patch = ("boto3", "requests", "pymysql")
patch(libs_to_patch)

I package all the libraries using a Lambda Layer and I have verified the contents of the layer by downloading it and looking at the source code. In the CloudWatch logs, I see the below:

[ERROR] Exception: modules pymysql are currently not supported for patching
Traceback (most recent call last):
  File "/var/task/serverless_sdk/__init__.py", line 130, in wrapped_handler
    return user_handler(event, context)
  File "/var/task/s_receiveJob.py", line 23, in error_handler
    raise e
  File "/var/task/s_receiveJob.py", line 18, in <module>
    user_handler = serverless_sdk.get_user_handler('datadog_handlers/receiveJob.handler')
  File "/var/task/serverless_sdk/__init__.py", line 54, in get_user_handler
    user_module = import_module(user_module_name)
  File "/var/lang/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "datadog_handlers/receiveJob.py", line 2, in <module>
    from src.handlers.job import handler as handler_impl
  File "/var/task/src/handlers/job.py", line 18, in <module>
    start_lambda_services("eventProcessor")
  File "/var/task/src/lib/lambda_initializer.py", line 15, in start_lambda_services
    patch(libs_to_patch)
  File "/opt/python/lib/python3.8/site-packages/aws_xray_sdk/core/patcher.py", line 92, in patch
    raise Exception('modules %s are currently not supported for patching'

It is clearly visible from the error logs that the Lambda is somehow using the older version of the SDK (v2.4.3) as the raise Exception line has been moved from 92 to 94 in v2.5.0

Details:

Python: 3.8
Xray SDK: 2.5.0

[aniljaiswal]

@srprash I'm facing this now in Lambda

[srprash]

Hi @anil-grexit ,
Sorry to hear about the issue. I did a quick test with a lambda and it seems to working fine in patching the pymysql library.
I also verified downloading the aws-xray-sdk v2.5.0 with pip and the code is fine too.

As per the lambda, can you verify your deployment package has picked up the latest release? I cannot think of a reason for lambda to not pick up v2.5.0.

[aniljaiswal]

Hi @srprash, As I mentioned in the OP, I have downloaded the lambda layer, unzipped and verified the SDK package, it does in fact contains the latest v2.5.0 code. I'm not sure how and where from the lambda environment is picking up the older version. Could you point me in the right direction as in what to look for next? Should I raise a support ticket with AWS?

[srprash]

Hi @anil-grexit
I put together a lambda layer with the new v2.5.0 and was able to patch pymysql fine. I am not really sure what could be wrong with your layer, but it might be worth checking that you are using the latest version of your layer in your function. If the issue still persists, feel free to open a ticket with AWS.

[aniljaiswal]

Hey @srprash as you can see above, I referenced a PR from Datadog sdk. The problem is that we also use Datadog's lambda layer which contains the v2.4.3 of Xray SDK and its overriding our own package. The referenced PR once merged and released will resolve this. I'm closing this issue. Thanks for putting your efforts into it. Really appreciate it.

[sunithaajit]

We are seeing the same issue with pymysql using patch. No errors with patch_all. What's the fix? pymysql version: 0.9.3