DynamoDB: 'id' key is S-typed but coerced as bytes

[ikonst]

The resulting stored data looks like:

{
    "Count": 1,
    "Items": [
        {
            "timestamp": {
                "N": "1554962898.2549338"
            },
            "result": {
                "B": "..."
            },
            "id": {
                "S": "b'celery-task-meta-3b0be007-8cf5-490b-a75f-c6d05d3c81ad'"
            }
        }
    ],
    "ScannedCount": 1,
    "ConsumedCapacity": null
}

The DynamoDB 'S' type expects a Unicode string. Note the b'' repr, presumably an effect of BaseKeyValueStoreBackend.key_t that defaults to ensure_bytes.

[auvipy]

got any working solution to fix it?

[ikonst]

from celery.backends.dynamodb import DynamoDBBackend
DynamoDBBackend.key_t = str

fixes it, but this breaks compatibility with the existing IDs

[georgepsarakis]

@ikonst thanks for the bug report. Would it make sense then to handle both formats simultaneously? Just an observation that the code is currently performing additional transformations on the key (for Python 2/3 compatibility reasons), both on put and get operations.

I think the best way to make the transition would be to use BatchGetItem API and attempt to retrieve both Task ID formats.

[ikonst]

I think the best way to make the transition would be to use BatchGetItem API and attempt to retrieve both Task ID formats.

You probably wouldn't want this behavior by default, since it 2x cost even for new instances.

[georgepsarakis]

@ikonst I see your point, you are right, we need to examine the impact on cost, since it could require more read capacity units. However, with what I am proposing the read request will always return a single item (depending on the Celery version that created the result record), so I think read capacity unit consumption will be the same. How does that sound?

[ikonst]

If you perform a read operation on an item that does not exist, DynamoDB still consumes provisioned read throughput

DynamoDB processes each item in the batch as an individual GetItem request

So a BatchGetItem for two keys will consume two units, or am I reading it wrong?

[georgepsarakis]

@ikonst indeed, this seems to be the case, so my assumption in the previous comment is invalid.

[georgepsarakis]

We can add an option, so that users can enable the key format change, along with a documentation note and a deprecation warning in the code, that will prompt for upgrade. Eventually the setting will be removed and the new key format will become the default. If the additional unit consumption is not a problem, users can enable the backwards compatible query for a short migration period, and then use the new format.

How does that sound?

[ikonst]

So there's effectively 3 modes:

• old id
• new id
• both ids for migration period

Sounds good to me.