Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Faster ElementTree iterating #70061

Closed
serhiy-storchaka opened this issue Dec 15, 2015 · 6 comments
Closed

Faster ElementTree iterating #70061

serhiy-storchaka opened this issue Dec 15, 2015 · 6 comments
Assignees
Labels
expert-XML extension-modules C modules in the Modules dir performance Performance or resource usage

Comments

@serhiy-storchaka
Copy link
Member

serhiy-storchaka commented Dec 15, 2015

BPO 25873
Nosy @brettcannon, @scoder, @vadmium, @serhiy-storchaka
Files
  • etree_iter.patch
  • etree_iter2.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
    closed_at = <Date 2015-12-21.11:51:17.800>
    created_at = <Date 2015-12-15.19:17:03.640>
    labels = ['extension-modules', 'expert-XML', 'performance']
    title = 'Faster ElementTree iterating'
    updated_at = <Date 2015-12-21.11:51:17.799>
    user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2015-12-21.11:51:17.799>
    actor = 'serhiy.storchaka'
    assignee = 'serhiy.storchaka'
    closed = True
    closed_date = <Date 2015-12-21.11:51:17.800>
    closer = 'serhiy.storchaka'
    components = ['Extension Modules', 'XML']
    creation = <Date 2015-12-15.19:17:03.640>
    creator = 'serhiy.storchaka'
    dependencies = []
    files = ['41318', '41324']
    hgrepos = []
    issue_num = 25873
    keywords = ['patch']
    message_count = 6.0
    messages = ['256472', '256495', '256502', '256507', '256674', '256795']
    nosy_count = 5.0
    nosy_names = ['brett.cannon', 'scoder', 'python-dev', 'martin.panter', 'serhiy.storchaka']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'performance'
    url = 'https://bugs.python.org/issue25873'
    versions = ['Python 3.6']

    @serhiy-storchaka
    Copy link
    Member Author

    serhiy-storchaka commented Dec 15, 2015

    Proposed patch refactors and optimize the code for iterating ElementTree. Refactoring and minor optimization makes the code about 30 lines smaller and 2-3% faster. Using a continuous array instead of linked list makes the code up to 40% faster (lxml benchmark was used to measurement).

    In addition the patch fixes possible issue with owning references.

    @serhiy-storchaka serhiy-storchaka added extension-modules C modules in the Modules dir performance Performance or resource usage labels Dec 15, 2015
    @vadmium
    Copy link
    Member

    vadmium commented Dec 16, 2015

    Looks technically correct as far as my knowledge of the malloc routines goes. What was the problem with references that you fixed? Maybe with parent_stack_push_new() failure?

    The main reference counting bug that sticks out to me is with the text and tail elements, because element_get_text() etc return a borrowed reference. It might be good to fix this while you are at it:

    >>> element = Element("tag")
    >>> class Text:
    ...     def __bool__(self):
    ...         element.text = "changed"
    ...         return True
    ... 
    >>> element.text = Text()
    >>> i = element.itertext()
    >>> next(i)
    Segmentation fault (core dumped)
    [Exit 139]

    @serhiy-storchaka
    Copy link
    Member Author

    serhiy-storchaka commented Dec 16, 2015

    Good catch Martin. I missed this case. Following patch fixes it.

    Other bugs look similar. They presumably can be triggered when remove the element itself during executing the __next__() method. I have no a reproducer though.

    I don't know wherever it is worth to fix these bugs in maintained releases.

    @vadmium
    Copy link
    Member

    vadmium commented Dec 16, 2015

    The new changes look good.

    I see these kind of reference counting bugs as fairly low priority. I suspect there are plenty more in this module, at least bpo-24103 and bpo-24104.

    @serhiy-storchaka
    Copy link
    Member Author

    serhiy-storchaka commented Dec 18, 2015

    Thank you for your review Martin. I have opened a separate bpo-25902 for refcount bugs in iteration. Its patch is purposed for all versions, while this issue is for developer version only.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Dec 21, 2015

    New changeset 5a5d5268afd5 by Serhiy Storchaka in branch 'default':
    Issue bpo-25873: Optimized iterating ElementTree.
    https://hg.python.org/cpython/rev/5a5d5268afd5

    @serhiy-storchaka serhiy-storchaka self-assigned this Dec 21, 2015
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    expert-XML extension-modules C modules in the Modules dir performance Performance or resource usage
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants