-
-
Notifications
You must be signed in to change notification settings - Fork 31.7k
ceval: Optimize list[int] (subscript) operation similarly to CPython 2.7 #70468
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
See also issue bpo-21955 |
Yury, |
Would be nice to collect statistics about arguments types during running the testsuite. May be list is not the most popular type of collection. |
Zach, first I was going to collect some stats on this (as Serhiy also suggests). It would be interesting to collect some stats on how many times BINARY_SUBSCR receives lists, tuples, dicts, and other types. I'd instrument the code to collect those stats and run Python tests suite and benchmarks. I'm pretty sure that optimizing lists (and tuples?) is a great idea. It would also be nice to optimize [-1] lookup. I'd also measure if we should add a fast path for dicts (PyDict_CheckExact + PyDict_GetItem). If you have time to work on this issue, then by all means go ahead. We'll be glad to assist you and review the patch. |
Zach, BTW, you can see how I instrumented ceval for stats collection in the patch for issue bpo-26219. That's only for the research purposes, we won't commit that... or maybe we will, but in a separate issue :). |
I think it's a good idea indeed.
How is that different from the above? :) |
Ok, I've started on the instrumenting, thanks for that head start, that would have taken me a while to figure out where to call the stats dump function from. Fun fact: BINARY_SUBSCR is called 717 starting python. |
I'll put together something comprehensive in a bit, but here's a quick preview: $ ./python
Python 3.6.0a0 (default, Feb 4 2016, 20:08:03)
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
Total BINARY_SUBSCR calls: 726
List BINARY_SUBSCR calls: 36
Tuple BINARY_SUBSCR calls: 103
Dict BINARY_SUBSCR calls: 227
Unicode BINARY_SUBSCR calls: 288
Bytes BINARY_SUBSCR calls: 68
[-1] BINARY_SUBSCR calls: 0 $ python bm_elementtree.py -n 100 --timer perf_counter
...[snip]...
Total BINARY_SUBSCR calls: 1078533
List BINARY_SUBSCR calls: 513
Tuple BINARY_SUBSCR calls: 1322
Dict BINARY_SUBSCR calls: 1063075
Unicode BINARY_SUBSCR calls: 13150
Bytes BINARY_SUBSCR calls: 248
[-1] BINARY_SUBSCR calls: 0 Lib/test$ ../../python -m unittest discover So dict seems to be the winner here |
Looks like we want to specialize it for lists, tuples, and dicts; as expected. Not so sure about [-1, but I suggest to benchmark it anyways ;) |
One thing I forgot to do was count slices. |
Looks as statistics varies from test to test too much. Could you please collect the statistics for running all tests? |
The test suite is not an appropriate workload to run benchmarks or statistics. Can you run with the benchmarks suite instead? |
I also suggest counting the number of BINARY_SUBSCR calls that are *not* one of the builtin types under consideration. Also, it would be good to distinguish slicing from integer indexing, for lists and tuples. |
I'm attaching output from a selection of the benchmarks, I'm counting non-builtins and slices, but for everything, not just lists and tuples. Quick observation: math workloads seem list heavy, text workloads seem dict heavy, and tuples are usually somewhere in the middle. |
Oh, I just created a duplicate with a patch for list[int]: issue bpo-26301. |
Here's a patch that looks likes Victor's from the duplicate, but with tuples covered as well. I ran some straight forward micro benchmarks but haven't bothered running the benchmark suite yet. Unsurprisingly, optimized paths are faster, and the others take a penalty. [0]byrnez@byrnez-laptop: [0]byrnez@byrnez-laptop: [0]byrnez@byrnez-laptop: [0]byrnez@byrnez-laptop: |
I suggest to try to inline PyList_GetItem: use PyList_GET_ITEM and raise I'm not sure that it's ok to add PyLong_AsSize_t() to the slow path. Copy |
Is it worth handling the exception, or just let it take the slow path and get caught by PyObject_GetItem()? We're still making sure the index is in bounds. Also, where would be an appropriate place to put a macro for adjusting negative indices? |
The new patch "subscr2" removes the tuple block, and addresses Victor's comments. This one looks a little faster, down to 0.0215 usec for the same test. |
This looks like a case of specialization. |
Very much so. Irit, do you want to give it a try? (Note there are helpful instructions now in Python/adaptive.md.) |
Sure, I'll have a look. |
Here is my previous attempt at this, for reference: |
Way to go Irit!-- |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: