Limiting data copy in xmlrpclib

[kilobug]

BPO	1747670
Nosy	@loewis, @pitrou, @serhiy-storchaka
Files	xmlrpclib.patch xmlrpc_less_copy.diff xmlrpc_less_copy-1.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2007-07-04.09:54:37.000>
labels = ['library', 'performance']
title = 'Limiting data copy in xmlrpclib'
updated_at = <Date 2016-01-20.09:57:29.986>
user = 'https://bugs.python.org/kilobug'

bugs.python.org fields:

activity = <Date 2016-01-20.09:57:29.986>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2007-07-04.09:54:37.000>
creator = 'kilobug'
dependencies = []
files = ['8070', '29519', '29579']
hgrepos = []
issue_num = 1747670
keywords = ['patch']
message_count = 12.0
messages = ['52808', '110892', '110900', '110920', '184825', '184949', '184950', '184961', '185264', '189963', '190134', '190136']
nosy_count = 5.0
nosy_names = ['loewis', 'pitrou', 'kilobug', 'serhiy.storchaka', 'Thomas Fenzl']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'pending'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue1747670'
versions = ['Python 3.4']

[kilobug]

Data going through the xmlrpclib are copied many times in Python. This creates problem when submitting blob of data in XML-RPC (I know the protocol is not aimed for that, but sometimes it's useful).

This little patch try to limit memory usage of xmlrpclib by adding a few optimizations.

I did the patch on Python 2.4 because it's what our customer is using, but it should work on Python 2.5 or more recent too.

[BreamoreBoy]

The code in the patch has been removed from py3k. This can only go forward if a unit test is provided. Change assigned to as per maintainers list.

[skrah]

Martin has recently removed his name from py3k/Misc/maintainers.rst.
Also, he has stated that assigning bugs to him might mean that they
get less attention (due to the large number of issues that he's involved
in).

[BreamoreBoy]

Could somebody update here please http://svn.python.org/view/*checkout*/python/branches/py3k/Misc/maintainers.rst

[ThomasFenzl]

Adapted the patch to python3.3
It may be useful with large amounts of transfered data avoiding a copy and freeing memory earlier.
The functionality is tested with the existing unit tests.

[pitrou]

Well, str.join() already optimizes this case to avoid copies:

>>> s = "x" * 50
>>> id(s)
139712615414000
>>> id(''.join([s]))
139712615414000

[pitrou]

I would close this as out of date, unless you have other suggestions to improve memory consumption.

[kilobug]

Great that join does the optimisation by itself now, but the last issue of the patch (cleaning the _data array before calling f() so the memory of _data can be collected earlier) still seems meaningful today.

[ThomasFenzl]

I removed the unnecessary check on single-element arrays.
No strong opinion on usefulness, as I don't use xmlrpc a lot...

[serhiy-storchaka]

Do you have any benchmarks?

[kilobug]

It's not something that can be easily benched because it depends a lot of the use case. If some conditions are present (big amount of data sent to XML-RPC, the XML-RPC server taking a long time to answer, end either Python giving back the memory to the OS or another thread reusing the garbage collected memory) the gain in memory can be very significant, in most other cases, it won't change anything.

Do you me to craft a simple example where the difference can be seen ?

[serhiy-storchaka]

Yes, it would be nice if you provided an example.

[iritkatriel]

Apart from this being abandoned for 9 years, the change proposed here is only valid if end() is only called once. Is this guaranteed? I can't find anything about this in the doc.

I suggest we close the issue.