pickletools.optimize doesn't reframe correctly

[pitrou]

BPO	19754
Nosy	@tim-one, @pitrou, @avassalotti, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/avassalotti'
closed_at = <Date 2013-12-02.00:28:56.557>
created_at = <Date 2013-11-24.13:29:04.493>
labels = ['performance']
title = "pickletools.optimize doesn't reframe correctly"
updated_at = <Date 2013-12-02.00:32:00.050>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2013-12-02.00:32:00.050>
actor = 'python-dev'
assignee = 'alexandre.vassalotti'
closed = True
closed_date = <Date 2013-12-02.00:28:56.557>
closer = 'alexandre.vassalotti'
components = []
creation = <Date 2013-11-24.13:29:04.493>
creator = 'pitrou'
dependencies = []
files = []
hgrepos = []
issue_num = 19754
keywords = []
message_count = 5.0
messages = ['204213', '204235', '204981', '204982', '204983']
nosy_count = 5.0
nosy_names = ['tim.peters', 'pitrou', 'alexandre.vassalotti', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'low'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue19754'
versions = ['Python 3.4']

[pitrou]

pickletools.optimize() can output arbitrarily large frames (much larger than then 64 KB heuristic). This may be annoying for memory use when pickling -- on the other hand if you keep the whole pickle in memory for optimize() perhaps it's not a problem keeping it in memory for load() :-)

$ ./python -c "import pickle, pickletools; d = pickle.dumps(list(range(100000)), 4); pickletools.dis(d)"  | head
    0: \x80 PROTO      4
    2: \x95 FRAME      65537
   11: ]    EMPTY_LIST
   12: \x94 MEMOIZE
   13: (    MARK
   14: K        BININT1    0
   16: K        BININT1    1
   18: K        BININT1    2
   20: K        BININT1    3
   22: K        BININT1    4

$ ./python -c "import pickle, pickletools; d = pickle.dumps(list(range(100000)), 4); pickletools.dis(pickletools.optimize(d))"  | head
    0: \x80 PROTO      4
    2: \x95 FRAME      368875
   11: ]    EMPTY_LIST
   12: \x94 MEMOIZE
   13: (    MARK
   14: K        BININT1    0
   16: K        BININT1    1
   18: K        BININT1    2
   20: K        BININT1    3
   22: K        BININT1    4

[serhiy-storchaka]

on the other hand if you keep the whole pickle in memory for optimize() perhaps it's not a problem keeping it in memory for load() :-)

This is not always true. You run optimize() on one computer or environment and load() on other.

[python-dev]

New changeset bb71baa28f1b by Alexandre Vassalotti in branch 'default':
Issue bpo-19754: Make pickletools.optimize respect the frame size target.
http://hg.python.org/cpython/rev/bb71baa28f1b

[avassalotti]

Now, pickletools.optimize doesn't do anything on protocol 4. :)

[python-dev]

New changeset 9feada79a411 by Alexandre Vassalotti in branch 'default':
Issue bpo-19754: Fix typo.
http://hg.python.org/cpython/rev/9feada79a411