pyc file with a large tuple is ~5x bigger in 3.11+ than in 3.9

[fried]

Bug description:

The code to produce the regression is large see: https://gist.github.com/fried/75fc3e3477634927444693329444648c for the best example I could come up with that I could share.

Basically we have this old code generator for thrift that generates these things like DataClasses for use in serialization and rpc. Well there is a C level accelerator that helps to serialize and desalinize the structures so it generates a runtime available form of the rpc/field specs that we can pass into the accelerator. The form is an array where the array position is the field id in thrift. If a field id is skipped in the thrift specification you put a None in the empty positions in the array. This works well but there are some edge cases where structs skip large number of field ids, say going from field id 5 to field id 9000. You do this enough in a thrift file and you have many large tuples. To make a form of this I could share I just have one tuple with 20k entries.

Looking at the pyc sizes you can see there is a serious issue in 3.10

-rw-r--r-- 1 carljm users 6425726 Sep  6 16:08 bigpyc.cpython-310.pyc
-rw-r--r-- 1 carljm users  397049 Sep  6 16:14 bigpyc.cpython-311.pyc
-rw-r--r-- 1 carljm users  396930 Sep  6 16:12 bigpyc.cpython-312.pyc
-rw-r--r-- 1 carljm users  396930 Sep  6 16:08 bigpyc.cpython-313.pyc
-rw-r--r-- 1 carljm users   81114 Sep  6 16:19 bigpyc.cpython-38.pyc
-rw-r--r-- 1 carljm users   81114 Sep  6 16:15 bigpyc.cpython-39.pyc

This is directly translating to Memory usage of the imported pyc. I believe because in 3.10 the bytecode creates the tuples as a list then convert them to a tuple at import with LIST_TO_TUPLE. So we need the memory for a list and a tuple to exist at the same time so we see around a 2.8x memory regression in real world examples.

CPython versions tested on:

3.8, 3.9, 3.10, 3.11, 3.12, CPython main branch

Operating systems tested on:

Linux, macOS

[fried]

@carljm for generating the list of pyc sizes for versions I didn't have installed.

[carljm]

Looks like one factor here is a compiler change from 3.9 to 3.10 that limits stack size used and thus switches away from a giant BUILD_TUPLE to a series of LIST_APPEND and then LIST_TO_TUPLE. But that doesn't fully explain the massive bytecode size in 3.10, or the remaining 5x regression in 3.11+ relative to 3.9. I suspect maybe the precise error locations, but haven't confirmed that yet.

[DinoV]

It does look like co_lnotab is 6345620 on 3.10 and 40368 on 3.8.

[fried]

I was able to mitigate this issue at Meta by making the generated tuple sparse in the .py and at import time expanding it with the None's so the c extensions are still happy.

[carljm]

Since the massive regression in 3.10 is already fixed, and the remaining regression here in 3.11+ looks to be explained by precise error locations (which were accepted knowing they would increase pyc file size), and there's no specific proposal here to further reduce pyc file sizes while maintaining functionality, I'm closing this issue.