Skip to content

concurrent.futures.ProcessPoolExecutor.map() doesn't batch function arguments by chunks #55480

Closed
@tbrink

Description

@tbrink
mannequin
BPO 11271
Nosy @brianquinlan, @pitrou, @MojoVampire
Files
  • map_comparison.py: Demonstration and workaround
  • new_processpoolexecutor.py: Improved the workaround
  • map_chunksize.patch: Adds chunksize parameter to ProcessPoolExecutor.map.
  • map_chunksize_with_test.patch
  • test_mult.py: benchmark script
  • map_chunksize_with_docs.patch: Updated patch with doc changes.
  • map_chunksize_docs_update.patch: More tests, small change to how we "yield from" the result chunks
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2014-10-04.18:28:06.450>
    created_at = <Date 2011-02-21.15:04:48.909>
    labels = ['library', 'performance']
    title = "concurrent.futures.ProcessPoolExecutor.map() doesn't batch function arguments by chunks"
    updated_at = <Date 2014-10-05.00:28:50.713>
    user = 'https://bugs.python.org/tbrink'

    bugs.python.org fields:

    activity = <Date 2014-10-05.00:28:50.713>
    actor = 'dan.oreilly'
    assignee = 'none'
    closed = True
    closed_date = <Date 2014-10-04.18:28:06.450>
    closer = 'pitrou'
    components = ['Library (Lib)']
    creation = <Date 2011-02-21.15:04:48.909>
    creator = 'tbrink'
    dependencies = []
    files = ['20825', '20826', '36067', '36184', '36185', '36306', '36352']
    hgrepos = []
    issue_num = 11271
    keywords = ['patch']
    message_count = 19.0
    messages = ['128963', '128970', '137351', '137359', '137692', '155114', '223802', '223803', '223855', '223966', '224369', '224372', '224451', '225045', '225143', '225214', '228472', '228473', '228514']
    nosy_count = 8.0
    nosy_names = ['bquinlan', 'pitrou', 'neologix', 'tbrink', 'python-dev', 'sbt', 'josh.r', 'dan.oreilly']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'performance'
    url = 'https://bugs.python.org/issue11271'
    versions = ['Python 3.5']

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      performancePerformance or resource usagestdlibPython modules in the Lib dir

      Projects

      No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions