API Gateway - Error during return : Object of type datetime is not JSON serializable

[filol]

What were you trying to accomplish?

I'm trying to return the list of quicksight dashboards via an api call.

Expected Behavior

Return the data

Current Behavior

Trace :

[ERROR] TypeError: Object of type datetime is not JSON serializable
Traceback (most recent call last):
  File "/var/task/aws_lambda_powertools/logging/logger.py", line 354, in decorate
    return lambda_handler(event, context)
  File "/var/task/app.py", line 15, in lambda_handler
    return app.resolve(event, context)
  File "/var/task/aws_lambda_powertools/event_handler/api_gateway.py", line 498, in resolve
    return self._resolve().build(self.current_event, self._cors)
  File "/var/task/aws_lambda_powertools/event_handler/api_gateway.py", line 557, in _resolve
    return self._call_route(route, match_results.groupdict())  # pass fn args
  File "/var/task/aws_lambda_powertools/event_handler/api_gateway.py", line 611, in _call_route
    return ResponseBuilder(self._to_response(route.func(**args)), route)
  File "/var/task/aws_lambda_powertools/event_handler/api_gateway.py", line 684, in _to_response
    body=self._json_dump(result),
  File "/var/task/aws_lambda_powertools/event_handler/api_gateway.py", line 688, in _json_dump
    return self._serializer(obj)
  File "/var/lang/lib/python3.9/json/__init__.py", line 234, in dumps
    return cls(
  File "/var/lang/lib/python3.9/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/var/lang/lib/python3.9/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/var/task/aws_lambda_powertools/shared/json_encoder.py", line 16, in default
    return super().default(obj)
  File "/var/lang/lib/python3.9/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '

Possible Solution

Steps to Reproduce (for bugs)

Object that i try to return (result from : https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/quicksight.html#QuickSight.Client.list_dashboards)

{
   "ResponseMetadata":{
      "RequestId":"xxxxxxx",
      "HTTPStatusCode":200,
      "HTTPHeaders":{
         "date":"Wed, 30 Mar 2022 07:35:02 GMT",
         "content-type":"application/json",
         "content-length":"12119",
         "connection":"keep-alive",
         "x-amzn-requestid":"xxxx"
      },
      "RetryAttempts":0
   },
   "Status":200,
   "DashboardSummaryList":[
      {
         "Arn":"arn:aws:quicksight:eu-west-1:xxxxxx:dashboard/xxxxxx",
         "DashboardId":"079dfe29-8857-43b0-859c-4f1130ccab68",
         "Name":"xxxxxx",
         "CreatedTime":"2022-02-01 18:51:13.217000+00:00",
         "LastUpdatedTime":"2022-02-03 15:00:33.301000+00:00",
         "PublishedVersionNumber":4,
         "LastPublishedTime":"2022-02-01 18:51:13.217000+00:00"
      },
      {
         "Arn":"arn:aws:quicksight:eu-west-1:xxxxxx:dashboard/xxxxx",
         "DashboardId":"xxxx",
         "Name":"xxxx",
         "CreatedTime":"2022-02-02 13:33:32.003000+00:00",
         "LastUpdatedTime":"2022-02-02 13:41:15.952000+00:00",
         "PublishedVersionNumber":2,
         "LastPublishedTime":"2022-02-02 13:33:32.003000+00:00"
      },
      {
         "Arn":"arn:aws:quicksight:eu-west-1:xxxxx:dashboard/xxxxx",
         "DashboardId":"xxxxx",
         "Name":"xxxxxxx",
         "CreatedTime":"2021-04-20 15:30:05.956000+00:00",
         "LastUpdatedTime":"2021-04-20 16:08:14.600000+00:00",
         "PublishedVersionNumber":2,
         "LastPublishedTime":"2021-04-20 15:30:05.956000+00:00"
      }
   ],
   "RequestId":"b9474cf1-f20a-4e2e-8381-7c910c0405c5"
}

from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.event_handler.api_gateway import ApiGatewayResolver, CORSConfig

from quicksight import list_dashboards, get_dashboard_embed_url

logger = Logger()
cors_config = CORSConfig(allow_origin="*", max_age=600, allow_credentials=True)
app = ApiGatewayResolver(cors=cors_config)


@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
def lambda_handler(event, context):
    logger.info(event)
    return app.resolve(event, context)


@app.get("/test-bug")
def get_list_dashboard():
    res = list_dashboards() 
    logger.info(res) # <----- Object just before
    return res

Environment

• Powertools version used: 1.25.5
• Packaging format (Layers, PyPi):
• AWS Lambda function runtime: 3.9 (Arm)
• Debugging logs

How to enable debug mode**

# paste logs here

[michaelbrewer]

@filol what happens if you try to do a pure json.dumps on res?

[michaelbrewer]

I would say it is the nested datetime fields that can't be serialized so you will need to pass in a custom serializer to overcome this, as this will probably against your example:

json.dumps(res, default=str)

Read the docs for more about how to setup a custom serializer:
https://awslabs.github.io/aws-lambda-powertools-python/latest/core/event_handler/api_gateway/#custom-serializer

[filol]

Hi @michaelbrewer, My temporary fix is :

    res = list_dashboards()
    logger.info(res)
    dashboards = res["DashboardSummaryList"]
    dashboards_fmt = json.loads(json.dumps(dashboards, default=json_serial)) 

    return dashboards_fmt

With

def json_serial(obj):
    """JSON serializer for objects not serializable by default json code"""

    if isinstance(obj, (datetime, date)):
        return obj.isoformat()
    raise TypeError ("Type %s not serializable" % type(obj))

And it's working but, for my point of view, this lib should handle this natively as it's a common type and it's returned by the aws api itself, what do you think ?

[michaelbrewer]

@filol you could pass in your custom serializer to the api gateway resolver.

ie:

import json

from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.event_handler.api_gateway import ApiGatewayResolver, CORSConfig

from quicksight import list_dashboards, get_dashboard_embed_url

def json_serial(obj):
    if isinstance(obj, (datetime, date)):
        return obj.isoformat()
    raise TypeError ("Type %s not serializable" % type(obj))


def custom_serializer(obj) -> str:
    return json.dumps(obj, default=json_serial)


logger = Logger()
cors_config = CORSConfig(allow_origin="*", max_age=600, allow_credentials=True)
app = APIGatewayRestResolver(serializer=custom_serializer, cors=cors_config)


@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
def lambda_handler(event, context):
    logger.info(event)
    return app.resolve(event, context)


@app.get("/test-bug")
def get_list_dashboard():
    res = list_dashboards() 
    ...

There is no end for all of the data types json.dumps should / could support.But the default formatting is not always clear.

[heitorlessa]

Hi @filol - Thank you for opening an issue with us. @michaelbrewer is correct - this is a type serialization issue that only you would be best positioned to addressed it (e.g., should it be as string, as ISO string, etc.).

This is also similar to any non-JSON serializable values. We document them here on how you can bring your own serializer, like Michael suggested above.

Closing.

Thank you!