Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Unable to predict for dict and set while optimizing "any" built-in #349
As discussed in #246 : Unable to predict
python -m nuitka --version
Thanks for reporting this, great find.
A few things I want to highlight @bksahu about optimization in Nuitka. First of all, I have adopted a method, where I give each level of optimization its own spin, and your finding is evidence of it being useful. So, what I would do, is first:
a) Run all world test without the optimization (done already)
On each stage, there is a tendency to find bugs. Right now, you can't really run the "all world test", but it has to be done on my infrastructure, as it needs really a lot of CPU and time. If you have a machine that might heat up and become not as responsive, you can try running the CPython test suite for one version at least. But I think I have to make an arrangement for you. It's a shame we don't have public CPU in large quantity yet. I will try and push for donated resources soon though. Nothing like Travis will allow us to use much CPU by default.
So, this is the general approach, and as you can see, had you headed my additional request to specialize
This testing approach deserves to be formalized with a table I guess. I am really happy about the opportunity to remember this, and why I have done it like above normally.
Also, another experience with optimization is, that when we add new one, we increase the burden on it being correct. Nuitka has historically been incorrect on many things with these kind of things, and then when new optimization gets added, old errors get exposed, because now we depend on it being correct more deeply.
The compile whole world test is going to crash in places you don't expect, and this could well have happened to us, had you not sought good enough test coverage yourself. For example, it could have gone unnoticed, and a for loop over constant values in the CPython test suite, would not hit the code now. But the day we start to unroll such loops, then it would crash, unrelated to the optimization being done there.
So there is a tendency to uncover existing bugs that is far more widespread in Nuitka than you would normally encounter, because easily everything can affect everything. Every new optimization adds new requirements, or makes new test cases. For example, previously nothing was known about
As for your bug, I would suggest you would question this code in
Being iterable at all, doesn't mean you can predict the values, and definitely not this way:
So, this could be done with
Then I think, random access, and sequential access, could be differentiated. Right now your
If you search for
@kayhayen I have a doubt. While debugging I have noticed that for example
I don't what exactly understand what do you mean by the handle and how it is going to return an expression instead of index. Could you give me some pointers?
Do you mean something like this:
@bksahu I was thinking that a node, like range, constant value, etc. should be able to return a new kind of object, one with methods, that will be used instead of itself. So instead of calling
This would then offer ways do get the
The problem of your method is that the iterator gets lost. That is something I would expect the handle to store, and re-use.
This is a rought sketch up:
Obviously, this needs to have its way of saying
For some constants, or ranges, or list multiplications, we will have better knowledge, and random access will be possible.
There are remnants of very weak iterator tracing in Nuitka. These handles, I would assume, could become value traces of the
Basically you need to think of these objects as intermediates. They address the need of generic algorithms to have more or less random access, as well as the difficulty of the producing side, to have its own state. We cannot well track a
From this, pretty immediately the need for an object between the node and the using algorithm comes into play, and that is pretty much immediately the only way to really do it.