Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

serialize individual args/kwargs rather than the containers #2246

Merged
merged 2 commits into from Aug 9, 2012

Conversation

Projects
None yet
3 participants
Owner

minrk commented Aug 3, 2012

This allows specials to be handled inside containers as args/kwargs

It is less efficient than before (still more efficient than 0.13) in cases where there is a very large number of simple arguments, but I think that generally the number of arguments is modest, and simple cases of small containers of arrays/References are likely (as most readily demonstrated by use in map).

The relevant behaviors are tested.

closes #2239

By the way, here's a plot showing why the new serialization had to be done:

figure

So about 100x faster for returning a 10k element list

It is the output from:

f = lambda n: range(n)
view = rc[-1]

for p in range(20):
    time.sleep(1)
    n = 2**p
    print "%i elements" % n
    %timeit view.apply_sync(f, n)

minrk added some commits Aug 3, 2012

@minrk minrk test canning for first-level containers as args/kwargs to apply and map ca8c553
@minrk minrk serialize elements of args/kwargs in pack_apply message
rather than the containers themselves.

This means arguments that are containers will be inspected for canning, but only at the highest level.  Containers of containers are not checked.
2fc7d23
Owner

minrk commented Aug 3, 2012

Test results for commit 2fc7d23 merged into master
Platform: darwin

  • python2.6: OK (libraries not available: cython matplotlib oct2py pygments pymongo qt rpy2 tornado wx wx.aui)
  • python2.7: OK (libraries not available: oct2py rpy2 wx wx.aui)
  • python3.2: OK (libraries not available: cython matplotlib oct2py pygments pymongo qt rpy2 wx wx.aui)

Not available for testing:

Owner

takluyver commented Aug 4, 2012

Test results for commit 2fc7d23 merged into master
Platform: linux2

  • python2.7: OK (libraries not available: oct2py rpy2)
  • python3.2: OK (libraries not available: cython oct2py pymongo rpy2 wx wx.aui)

Extra args: ['--all']
Not available for testing: python2.6

Owner

takluyver commented Aug 4, 2012

A convenient guinea-pig. Note that it lists the extra args, so a reviewer can check that a PR like this has had a --all test run.

Owner

ellisonbg commented Aug 9, 2012

This is a spectacular speed up, looking through the code now.

Owner

ellisonbg commented Aug 9, 2012

Code looks really solid, tests pass, I would say it is ready for merging. I will let you pull the trigger.

@minrk minrk added a commit that referenced this pull request Aug 9, 2012

@minrk minrk Merge pull request #2246 from minrk/canning_depth
serialize individual args/kwargs rather than the containers

This allows specials to be handled inside containers as args/kwargs

It is less efficient than before (still more efficient than 0.13) in cases where there is a very large number of simple arguments, but I think that generally the number of arguments is modest, and simple cases of small containers of arrays/References are likely (as most readily demonstrated by use in map).

The relevant behaviors are tested.

closes #2239
75c66c8

@minrk minrk merged commit 75c66c8 into ipython:master Aug 9, 2012

@minrk minrk deleted the minrk:canning_depth branch Mar 31, 2014

@mattvonrocketstein mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this pull request Nov 3, 2014

@minrk minrk Merge pull request #2246 from minrk/canning_depth
serialize individual args/kwargs rather than the containers

This allows specials to be handled inside containers as args/kwargs

It is less efficient than before (still more efficient than 0.13) in cases where there is a very large number of simple arguments, but I think that generally the number of arguments is modest, and simple cases of small containers of arrays/References are likely (as most readily demonstrated by use in map).

The relevant behaviors are tested.

closes #2239
5284a48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment