Dict args with integer keys get renamed as strings

[j6k4m8]

Hey @william-silversmith :) It's 10pm on a work night, which means it's time for a weird bug(feature?) report from Jordan!

I've put together a minimum viable example to illustrate this, I promise I can do this math in my head usually :)

Check out the code below. In short, I have a dictionary with a key of 1 (int) and a value of some number (here, 100), which I decrement by 1, re-insert into the queue, and then process the next job:

Min-reproducible:

from functools import partial
from taskqueue import queueable, TaskQueue

QUEUE_URI = "fq://demoqueue"
Q = TaskQueue(QUEUE_URI)


@queueable
def subtract_one(my_data: dict[int, int]):
    if my_data[1] > 0:
        new_data = my_data.copy()
        new_data[1] -= 1
        Q.insert([partial(subtract_one, new_data)])
    else:
        print("Done")


if __name__ == "__main__":
    subtract_one({1: 100})
    Q.poll()

So on the first execution, my payload should be {1: 100}, then {1: 99}, etc.

Error

but when I run this, I get the following error:

Traceback (most recent call last):
  File "mvp-integer-keys.py", line 25, in <module>
    Q.poll()
  File ".../python3.9/site-packages/taskqueue/taskqueue.py", line 375, in poll
    task.execute(*execute_args, **execute_kwargs)
  File ".../python3.9/site-packages/taskqueue/queueablefns.py", line 78, in execute
    self(*args, **kwargs)
  File ".../python3.9/site-packages/taskqueue/queueablefns.py", line 87, in __call__
    return self.tofunc()()
  File "mvp-integer-keys.py", line 14, in subtract_one
    if my_data[1] > 0:
KeyError: 1

Checking the queue files, the my_data arg in subtract_one gets populated as {"1": 99} instead:

From <ts>-<uuid>.json in demoqueue/queue/:

{"payload": [["__main__", "subtract_one"], [{"1": 99}], {}, -1], "queueName": "demoqueue", "id": "fcb21030-f8f8-4721-a9df-3001cb5c2c79"}

Funky! I'm guessing this is because orjson needs string keys? But it's opaque to the end-user here (I'm still not totally sure that that's what's going wrong!)

Fix

If I just assume that my keys are all str, this same code runs perfectly:

...

@queueable
def subtract_one(my_data: dict[str, int]):
    if my_data["1"] > 0:                                 # 👀
        new_data = my_data.copy()
        new_data["1"] -= 1                               # 👀
        Q.insert([partial(subtract_one, new_data)])
    else:
        print("Done")


if __name__ == "__main__":
    subtract_one({"1": 100})                             # 👀
    Q.poll()

So not a huge deal! But thought I'd report it in case others encounter the same thing!

[william-silversmith]

Ack! The reason for this is probably the JSON encoder that requires dict keys to be strings. I could probably add some type information and then force the keys to be integers.

[william-silversmith]

On the other hand, Python allows you to mix integers and strings... that could get hairy.

[j6k4m8]

It might be easier to just do a deep equality check before and after serial/deserialization and refuse to insert (or insert with warning?) This isn't mission-critical for me, but thought it was interesting!

[william-silversmith]

I wonder if there are other formats out there as easy to use as JSON that would support valid integer keys. I think the "good" options here are (a) be explicit that JSON is the medium of transmission with the caveats that go with that (b) use a different format that doesn't have this issue (c) reject or warn about conversions which will cost extra computation.

[william-silversmith]

This alternative format could be a viable replacement for JSON: https://msgpack.org/ However, because it's a binary format, it doesn't have the advantage of being able to edit the queue files easily using bash tools. That saved my bacon once when it took hours to generate some tasks (lots of computation) and then I got one parameter wrong... I was able to edit all the tasks really quickly using sed.