MapAttribute serialisation issues

[njsnx]

Looking to get some help - I am trying to use this fork to implement a single table design in my python-based API.

Overall, works fine but when I store a Map Attribute field, the result in Dynamo seems like it's been serialised multiple times and looks incorrect (although valid in dynamo)

Comparison:

Created via falcano

new = Item(hash_key=f'testItem#{id}', range_key="testRange#3749189f-3d4c-4937-b31a-924bc84b366c")
 new.data = dict(hello="message", nested={"hello": "message"}, enabled=False, number=0)

Results in

"data": {
    "M": {
      "number": {
        "M": {
          "N": {
            "N": "0"
          }
        }
      },
      "hello": {
        "M": {
          "S": {
            "S": "message"
          }
        }
      },
      "nested": {
        "M": {
          "M": {
            "M": {
              "hello": {
                "M": {
                  "S": {
                    "S": "message"
                  }
                }
              }
            }
          }
        }
      },
      "enabled": {
        "M": {
          "BOOL": {
            "BOOL": false
          }
        }
      }
    }
  },

Creating the attribute manually via the console as Map attribute, this is what it looks like and what I'd expect the python code to do too.

"console": {
    "M": {
      "string": {
        "S": "sdasda"
      },
      "number": {
        "N": "0"
      },
      "map": {
        "M": {
          "boolean": {
            "BOOL": false
          }
        }
      }
    }
  }

What am I doing wrong?

[njsnx]

From further digging, it seems like the data is being serilized twice before upload.

I checked this by re-running the code above, then also running a print
of

n = Item().data.serialize(Item().data.serialize(dict(hello="message", nested={"hello": "message"}, enabled=False, number=0)))
print(n)

>>

{'hello': {'M': {'S': {'S': 'message'}}}, 'nested': {'M': {'M': {'M': {'hello': {'M': {'S': {'S': 'message'}}}}}}}, 'enabled': {'M': {'BOOL': {'BOOL': False}}}, 'number': {'M': {'N': {'N': 0}}}}

Looking at the tests for map attribute, it's all based on mock data but has is there any tests of data that is stored in Dynamo then retrieved again?

will keep digging

[njsnx]

And a bit more digging -

Looking at the api call to put_item, even though serialize is defaulted to false, it looks like Map Attributes are still serialized

If I print out kwargs that are passed to put_item under the Model class, i see this:

new = Item(hash_key=f'testItem#{id}', range_key="testRange#3749189f-3d4c-4937-b31a-924bc84b366c",
        data=dict(hello="message", nested={"hello": "message"}, enabled=False, number=0)
    )
print(new.save())

>>>

{'TableName': 'my-table', 'Item': {'Type': 'testItem', 'data': {'hello': {'S': 'message'}, 'nested': {'M': {'hello': {'S': 'message'}}}, 'enabled': {'BOOL': False}, 'number': {'N': 0}}, 'pk': 'testItem#<built-in function id>', 'sk': 'testRange#3749189f-3d4c-4937-b31a-924bc84b366c'}, 'ReturnValues': 'NONE'}

The issue seems to be that the put_item API for the Table Resource doesn't require any serialization and does it automatically.
I tested this by running the API calls directly with my raw input and it stores as expected in Dynamo

 import boto3

table = boto3.resource('dynamodb')
table = table.Table('my-table')

table.put_item(Item={'Type': 'testItem', 'data': dict(hello="message", nested={"hello": "message"}, enabled=False, number=0), 'pk': 'testItem#cca06093-e0c6-49a6-93d5-266bed23cad1', 'sk': 'testRange#3749189f-3d4c-4937-b31a-924bc84b366c'})

The result is a successful API call and the Dynamo console shows the expected outcome:

{
  "pk": {
    "S": "testItem#cca06093-e0c6-49a6-93d5-266bed23cad1"
  },
  "sk": {
    "S": "testRange#3749189f-3d4c-4937-b31a-924bc84b366c"
  },
  "Type": {
    "S": "testItem"
  },
  "data": {
    "M": {
      "number": {
        "N": "0"
      },
      "hello": {
        "S": "message"
      },
      "nested": {
        "M": {
          "hello": {
            "S": "message"
          }
        }
      },
      "enabled": {
        "BOOL": false
      }
    }
  }
}

Fix appears to stop whatever serilizes the MapAttribute before put_item is called as this is an unnecessary step for the current API choice - Either that or change the API from the Resource Table.put_item to the DynamoDB Client put_item, which does need serialization - however this would mean all values need to be serialized, not just Map.

I'll see if I can fix and submit a PR