performance issues

[hugoliv]

Hi there,

Loading a 1Go JSON file may take a while. Using the standard JSON API with the same JSON file may take approximatively 35 seconds.

with jsons: time~ 20min

    with open(json_path, "r") as json_file:
        content = json_file.readlines()
        json_data = jsons.loads(content[0], CustomObject)

The readlines function take 2 or 3 seconds.

with json: time~ 35s

    with open(json_path, "r") as json_file:
        content = json.load(json_file)
        json_data = CustomObject(**content)

What I'm doing wrong ?

[ramonhagenaars]

Hi hugoliv,

That's a long time indeed. Could you show (an example of) a part of the content you are reading?

[hugoliv]

Hi,

Hum, I can't give you the data, but the JSON is nothing else than classical JSON (afaik):

{"version":1,"last_update":"dd/mm/yyhh:mm:ss","data":[{"foo_id":"an_id","bar_id":"an_id","baz_id":"an_id","qux":[{"corge":true,"grault":"an_id","garply":"25","waldo":{"x_1":0,"y_1":0,"x_2":0,"y_2":0},"fred":{"x_1":1,"y_1":1,"x_2":1,"y_2":1},"d":[{"plugh":18,"xyzzy":false},{"plugh":17,"xyzzy":false},{"plugh":16,"xyzzy":true},{"plugh":15,"xyzzy":true},{"plugh":14,"xyzzy":true},{"plugh":13,"xyzzy":true},{"plugh":12,"xyzzy":false},{"plugh":11,"xyzzy":true},{"plugh":21,"xyzzy":true},{"plugh":22,"xyzzy":true},{"plugh":23,"xyzzy":true},{"plugh":24,"xyzzy":false},{"plugh":25,"xyzzy":false},{"plugh":26,"xyzzy":false},{"plugh":27,"xyzzy":false},{"plugh":28,"xyzzy":false},{"plugh":48,"xyzzy":false},{"plugh":47,"xyzzy":false},{"plugh":46,"xyzzy":true},{"plugh":45,"xyzzy":true},{"plugh":44,"xyzzy":true},{"plugh":43,"xyzzy":true},{"plugh":42,"xyzzy":true},{"plugh":41,"xyzzy":true},{"plugh":31,"xyzzy":true},{"plugh":32,"xyzzy":true},{"plugh":33,"xyzzy":true},{"plugh":34,"xyzzy":false},{"plugh":35,"xyzzy":false},{"plugh":36,"xyzzy":false},{"plugh":37,"xyzzy":false},{"plugh":38,"xyzzy":false}],"thud_1":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1],"thud_2":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1],"thud_3":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1],"thud_4":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]}]}]}

[ramonhagenaars]

There are some optimizations that can (and will) be done. The aim is to bring the performance of jsons closer to that of json, but since the former depends on the latter for string deserialization, it will never surpass it.

The next 0.9.0 release will have some improvements on this point.

[ramonhagenaars]

I'm closing this issue, because there are no real acceptation criteria for this issue and there have been performance improvements in the former releases.