[mypyc] feat: cache len for container creation from expressions with length known at compile time #19503
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
BobTheBuidler
changed the title
|
@ilevkivskyi this one is also ready for review if and when you get a chance, though its definitely a bit more involved than #19497 |
BobTheBuidler
force-pushed
the
for-loop-len-cache
branch
from
07f40fc to
622f38f
Compare
| @@ -1147,3 +1187,33 @@ def gen_step(self) -> None: | |||
| def gen_cleanup(self) -> None: | |||
| for gen in self.gens: | |||
| gen.gen_cleanup() | |||
|
|
|||
|
|
|||
| def get_expr_length(expr: Expression) -> int | None: | |||
There was a problem hiding this comment.
These 2 helper functions can be extended to cover more cases and used for other length-based optimizations I have in mind
BobTheBuidler
changed the title
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
BobTheBuidler
force-pushed
the
for-loop-len-cache
branch
from
904945c to
537c8af
Compare
|
This PR shows a very meager improvement (~0.3%) on self-check, but I'm not sure if self-check is a reliable benchmark for this particular set of changes. It's a pretty niche-case optimization |
|
Would it make this one easier to review if I split it up? I thought keeping all the length stuff together made sense but I could plausibly do a separate PR just for the immutable type iteration stuff. |
Yes, this would benefit from being split into at least 2 PRs. Right now it's a bit difficult to see which changes are related to each other. |
| abc: Final = "abc" | ||
|
|
||
| def test() -> None: | ||
| a = [str(x) for x in [*abc, *"def", *b"ghi", ("j", "k"), *("l", "m", "n")]] |
There was a problem hiding this comment.
I think this would be better as a run test. It's pretty hard to review the IR for correctness.
There was a problem hiding this comment.
The idea behind this IR test is the line r19 = PyList_New(13) which demonstrates mypyc successfully parsed the length of all star inputs and was able to predetermine the length of the resulting list object, which allows mypyc to use the list creation fastpath
| abc: Final = "abc" | ||
|
|
||
| def test() -> None: | ||
| a = tuple(str(x) for x in [*abc, *"def", *b"ghi", ("j", "k"), *("l", "m", "n")]) |
There was a problem hiding this comment.
Similarly this seems to work better as a run test.
There was a problem hiding this comment.
Like in the other case, the idea behind this IR test is the line r19 = PyTuple_New(13) which demonstrates mypyc successfully parsed the length of all star inputs and was able to predetermine the length of the resulting tuple object
There was a problem hiding this comment.
I'll need to look into why the list is still being created with variable length, but that can be a separate PR
BobTheBuidler
changed the title
Currently, if a user uses an immutable type as the sequence input for a for loop, the length is checked once at each iteration which, while necessary for some container types such as list and dictionaries, is not necessary for iterating over immutable types tuple, str, and bytes.
This PR modifies the codebase such that the length is only checked at the first iteration, and reused from there.
Also, in cases where a simple genexp is the input argument for a tuple, the length is currently checked one additional time before entering the iteration (this is done to determine how to size the new tuple). In those cases, we don't even need a length check at the first iteration step, and can reuse the result of that first
lencall (or compile-time determined constant) instead.Lastly, in cases where a tuple is created from a genexp and the length of the genexp is knowable at compile time, this PR replaces PyList_AsTuple with the tuple constructor fast-path.